Service Affecting increase in disk io after installing Cloud Linux

Portal Home > Knowledgebase > Articles Database > Service Affecting increase in disk io after installing Cloud Linux

Posted by Zenon, 03-27-2014, 12:33 PM
Hey Folks, Looking for some advice, thoughts or opinions. I'm running a WHM/cPanel server with about a 180GB of data in /home I installed Cloud Linux a few days ago. My backups used to run in about 6hrs now they are taking 12-15hrs. I have determined the cause to be a significant increase in disk io (see the attached image) You can see that starting on the 24th the disk io demand changed significantly which is when Cloud Linux was installed. No changes were made to the backup configuration so its clearly a side effect of CL but does anyone have an idea's why? I tried increasing the KB chunk to 512 last night thinking if it did compression in memory and it was to compress larger chunks it would have less frequent io, but I think that it had the opposite effect by the looks of today's stats. I plan to try reducing it down to the minimum 128 tonight and maybe reduce the number of pigz processes to 6 and see if that alleviates the demand, but I'd still like opinions and comments. TECHNICAL HARDWARE DETAILS: sda is a 1TB RAID10 array on a 3ware 9650SE-4LP sdc is a 2TB drive on the motherboard Server is a Dual E5410 8GB RAM BACKUP CONFIGURATION: Compressed Backups using the new cPanel implementation (not legacy) COMPRESSIONS SETTINGS: use pigz - yes gzip compression level - minimum (1) Number of pigz processes - 8 Number of kilobyte chunks per compression work unit - 512 Thanks Attached Thumbnails
Posted by iseletsk, 03-27-2014, 10:03 PM
Could you check that raid still in tact? We have seen few cases where raid would start re-syncing after reboot. Also, is it KVM or Xen or real hardware?
Posted by Zenon, 03-27-2014, 10:56 PM
Hi Igor, thanks for jumping in I just checked and can confirmed that the RAID is verified and fully in-tact. This is 100% real hardware nothing virtualized. Furthermore I've been looking at stats and thinking, at first I thought maybe CL was placing a considerably higher demand on sda for general activity which left less disk resource available for the backup, BUT when the backups not running the disk io is the same as before the switch so it doesn't appear to be a base increase in disk-io. So that leaves the question what about the backups under CL is different then not. Thanks
Posted by iseletsk, 03-27-2014, 11:03 PM
This makes more sense, as single threads get lower IO priority under CL --> maybe that is what causing the slower back ups. What do you use for backup?
Posted by Zenon, 03-27-2014, 11:05 PM
Im using the native whm backup. (the new one they implemented a few months ago, the legacy backup system is disabled.)
Posted by iseletsk, 03-27-2014, 11:10 PM
My guess it has to do with process IO priorities. Ping our support if it causes a problem. It shouldn't create extra load on IO --> it should just take longer.
Posted by Zenon, 03-28-2014, 12:20 AM
I assume by pinging you mean submit a ticket? Its definitely causing a problem LOL just look at the 6hrs of 100% utilization.
Posted by prashant1979, 03-28-2014, 02:28 AM
Can you confirm if you have kept the backup schedule and the stats updation schedule to different time? In the recent cPanel updates, the Analog stats is enabled by default and it takes a very long time to update the stats and also the i/o increases. Make sure the backup and stats updation does not run at the same time and also disable Analog stats.
Posted by Zenon, 03-28-2014, 02:57 AM
I'm processing log every 12 hrs and bandwidth every 6hrs. I have checked the box to "Prevent cpanellogd (Log Processing) and cpbackup (Backups) from running at same time" I have NOT selected times from "Statistics Processing Times" grid where it tells me "Check all hours that you would like log analysis NOT to be performed" Am I right that so long as the "Prevent cpanellogd (Log Processing) and cpbackup (Backups) from running at same time" box is selected that also prevents stats from running during backups or is cpanellogd separate from that? Analog stats is disabled by default, AWstats and Webalizer is active. :::NOTE::: I appreciate your suggesting and am very welcoming to configuration tweaks to alleviate the symptoms, however I'd like to underscore the other primary focus of this thread is to discuss why with all things being equal the disk activity during backups is so much greater under CL vs a native Centos environment. yet there is little to no change in the disk activity during normal server functions.
Posted by EthernetServers, 03-28-2014, 04:28 AM
In this case, the word 'ping' does indeed mean to contact them. http://www.audioenglish.org/dictionary/ping.htm For reference, they can be contacted here: https://helpdesk.cloudlinux.com/inde...Tickets/Submit
Posted by Zenon, 03-28-2014, 01:05 PM
I went ahead and submitted a ticket last night so we shall see what we find out. Coincidentally I lowered the KB per compression chunk down to 128 and the backup completed in about 12 hours with only 6hrs spent at 100% disk utilization. So that did improve it, but its still significantly more disk activity then previously to the CL. PS So it doens't get lost in the ticket, I would still like someone to confirm/clarify my understanding of the stats processing scheduling options I described above.
Posted by mpascal, 04-12-2014, 06:20 PM
Any updates on this? I was looking into cloud linux. As for the WHM backups, mine were taking too long also and I disabled compression. My content is 95% jpgs and gz compression has no effect on them anyway. But it does take a lot of cpu to create the compressed file.
Posted by Zenon, 04-13-2014, 12:10 AM
I believe we solved the issue yesterday, but one backup run is not enough data for me to be sure, I will reply back either tomorrow or Monday after I can re-introduce some of my tweaks and evaluate the results thereof. Again I'll post more details of an eval of cloud linux later, but in short so far I am very, very happy with the results and support assistance I have received. (hopefully my satisfaction is not pre-mature)