High IoWAIT / Constant Reboot Needed

Portal Home > Knowledgebase > Articles Database > High IoWAIT / Constant Reboot Needed

Posted by The Calling, 11-07-2010, 10:28 AM
Can someone help me determine the *cause* of the high iowait status on my server? Thanks See attached. Results of iostat, sar, hdparm Attached Thumbnails
Posted by drspliff, 11-07-2010, 06:23 PM
What's stored on sdb3? Have you considered running `iotop` to see what the offending processes are?
Posted by The Universes, 11-07-2010, 07:18 PM
Agreed, theres definitely a good amount of activity on sdb. Run iotop or dstat. (dstat is available from DAG/RPMForge, you can run "dstat -d --top-bio" to see which process is using the most i/o).
Posted by kaniini, 11-07-2010, 08:12 PM
it looks like the iowait is from filesystem journal updates... to take a guess: are you running a mysql server on sdb3 using innodb? if so, you should tune your innodb database and it should go away. the problem here is that you're doing tons of FS transactions a second (i'm pretty sure it's actually the transactions here killing you), so the kernel is updating the journal of the filesystem and thus has the filesystem locked, causing iowait to shoot up. other than the high iowait, the server behaves normally yes? (e.g. accesses to files not on sdb3 are fine?)
Posted by The Calling, 11-07-2010, 11:20 PM
See attached. My my.cnf file Server specs: Dual Quad Core Xeon 4GB Memory 2x300GB SCSI Drives Attached Thumbnails
Posted by sOliver, 11-08-2010, 01:24 AM
TheCalling, I had some problems with high disk IO too. After disabling a lot of services I was able to fix the issue. You can use "top" to see what processes are consuming a lot of resources. Also "One cause of excess I/O on a SQL Server is page splitting" Try to optimize your tables and reduce slow SQL queries..
Posted by guru4hosting, 11-08-2010, 01:54 AM
Try disabling atime
Posted by Hillockhosting, 11-08-2010, 02:25 AM
your i/o does not seems to bad Value of 15-20 is fine .
Posted by The Calling, 11-08-2010, 07:54 AM
What services should I disable? What about this atime? How do I disable it and is it important for a shared hosting environment (Cpanel)?
Posted by drspliff, 11-08-2010, 08:58 AM
It updates the access time on files whenever they're accessed. For high-traffic sites this could lead to lots of small updates which could lead to journal activity like you're seeing. To turn it off add the 'noatime' flag to the appropriate entry in your `fstab`. Turning off atime logging shouldn't have any major side-effects.
Posted by OLM | DavidG, 11-08-2010, 09:33 AM
Modifying /etc/fstab to add the "noatime" flag will not have any effect until the server has been rebooted. You can make it take effect immediately by carefully using the "mount" command with the "-o remount,rw,noatime" options. You should also make sure to include any other mount options the filesystem is using (such as "usrquota" for instance). This can be checked by executing the "mount" command with no arguments.
Posted by The Calling, 11-08-2010, 10:00 AM
This is what I have now How do I add the noatime flag?
Posted by OLM | DavidG, 11-08-2010, 10:02 AM
You should change the first line as follows: LABEL=/ / ext3 defaults,usrquota,noatime 1 1