Virtuozzo Node Load increasing

Portal Home > Knowledgebase > Articles Database > Virtuozzo Node Load increasing

Posted by javii, 09-15-2010, 03:28 AM
Hi, I have Virtuozzo hardware node with 8 VEs inside. These are my machine specs: cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 30 model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping : 5 cpu MHz : 2400.072 cache size : 8192 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm bogomips : 4800.14 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [8] processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 30 model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping : 5 cpu MHz : 2400.072 cache size : 8192 KB physical id : 0 siblings : 4 core id : 1 cpu cores : 4 apicid : 2 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm bogomips : 4799.85 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [8] processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 30 model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping : 5 cpu MHz : 2400.072 cache size : 8192 KB physical id : 0 siblings : 4 core id : 2 cpu cores : 4 apicid : 4 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm bogomips : 4799.94 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [8] processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 30 model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping : 5 cpu MHz : 2400.072 cache size : 8192 KB physical id : 0 siblings : 4 core id : 3 cpu cores : 4 apicid : 6 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm bogomips : 4799.85 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: [8] cat /proc/meminfo MemTotal: 16393020 kB MemFree: 9328224 kB Buffers: 351372 kB Cached: 3284648 kB SwapCached: 0 kB Active: 4720564 kB Inactive: 1584648 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 16393020 kB LowFree: 9328224 kB SwapTotal: 16386292 kB SwapFree: 16386292 kB Dirty: 4032 kB Writeback: 0 kB AnonPages: 2669184 kB Mapped: 394704 kB Slab: 526140 kB PageTables: 126228 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 24582800 kB Committed_AS: 8352888 kB VmallocTotal: 34359738364 kB VmallocUsed: 320228 kB VmallocChunk: 34359418008 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB And two SAS disks in RAID 1 mode. I have noticed my load average is not too low as expected. But I'm experiencing something cuorius, if a limit in Virtuozzo the VE that is using most load to use less CPU, the VE now increases its load (ok with this) but the hardware node increases its load too... I though hardware node load should be lower if one VE is using less CPU. I tweaked these values of the VE in Virtuozzo : cpulimit percent Allowed CPU power burst_cpulimit percent Container CPU power limit burst_cpu_avg_usage percent CPU usage limit This is my top right now: top - 09:26:51 up 1:10, 1 user, load average: 3.08, 2.04, 1.70 Tasks: 767 total, 3 running, 764 sleeping, 0 stopped, 0 zombie Cpu(s): 12.9%us, 5.1%sy, 0.0%ni, 70.0%id, 10.8%wa, 0.2%hi, 1.0%si, 0.0%st Mem: 16393020k total, 7753468k used, 8639552k free, 384192k buffers Swap: 16386292k total, 0k used, 16386292k free, 3768732k cached any help would be appreciated
Posted by PCS-Chris, 09-15-2010, 03:37 AM
There is nothing wrong with that load at all. The acceptable load levels are completely different between for example a cPanel server and an OpenVZ node.
Posted by GOT, 09-15-2010, 12:40 PM
Instead of running top, you should run vzstat. That will give you a per VE breakdown of usage and can help you figure out where the load is coming from.
Posted by eth00, 09-15-2010, 12:49 PM
It may also be an IO issue, that snapshot only shows 10% iowait which is fine but if you watch it for a few it might be higher. Give vztop a shot instead of regular top, it along with vzstat will give you a better idea of where the resources are going.
Posted by javii, 09-15-2010, 04:50 PM
Thank you for your help: vztop gives me the same result as top but I can see container id of each process. This is really useful. Thanks! vzstat result: 10:43pm, up 14:27, 1 user, load average: 1.75, 2.05, 2.42 CTNum 9, procs 1014: R 10, S 1003, D 0, Z 1, T 0, X 0 CPU [ OK ]: CTs 100%, CT0 0%, user 27%, sys 8%, idle 65%, lat(ms) 42/0 Mem [ OK ]: total 16008MB, free 1028MB/0MB (low/high), lat(ms) 8/2 ZONE0 (DMA): size 9MB, act 0MB, inact 0MB, free 10MB (0/0/0) ZONE1 (DMA32): size 2983MB, act 946MB, inact 728MB, free 996MB (2/3/4) ZONE2 (Normal): size 13104MB, act 8873MB, inact 2721MB, free 16MB (12/16/19) Mem lat (ms): A0 0, K0 1, U0 0, K1 8, U1 0 Slab pages: 1466MB/1465MB (ino 777MB, de 306MB, bh 98MB, pb 65MB) Swap [ OK ]: tot 16002MB, free 16002MB, in 0.000MB/s, out 0.000MB/s Net [ OK ]: tot: in 0.058MB/s 202pkt/s, out 0.172MB/s 229pkt/s lo: in 0.000MB/s 0pkt/s, out 0.000MB/s 0pkt/s eth0: in 0.058MB/s 202pkt/s, out 0.172MB/s 229pkt/s eth1: in 0.000MB/s 0pkt/s, out 0.000MB/s 0pkt/s sit0: in 0.000MB/s 0pkt/s, out 0.000MB/s 0pkt/s Disks [ OK ]: in 0.015MB/s, out 0.523MB/s CTID ST %VM %KM PROC CPU SOCK FCNT MLAT IP 1 OK 0.8/- 0.0/- 0/77/256 0.02/0.9 27/1256 0 0 192.168.66.193 100 OK 2.8/- 0.1/- 0/70/32567 5.59/0.9 164/MAX 0 0 xxx.xxx.11.25 101 OK 1.8/- 0.1/- 0/60/32567 0.03/0.9 44/MAX 0 0 xxx.xxx.11.34 105 OK 1.1/- 0.0/- 0/25/32567 0.03/0.9 11/MAX 0 0 xxx.xxx.11.22 106 OK 5.6/- 0.4/- 3/223/32567 17.4/85 401/MAX 0 42 xxx.xxx.11.46 107 OK 4.4/- 0.3/- 0/171/32567 10.2/8.5 134/MAX 0 1 xxx.xxx.11.30 108 OK 2.9/- 0.1/- 0/92/32567 1.42/0.9 91/MAX 0 1 xxx.xxx.11.52 109 OK 1.5/- 0.1/- 0/53/32567 0.01/0.9 53/MAX 0 1 xxx.xxx.11.61 110 OK 0.7/- 0.1/- 0/52/32567 3.06/0.9 51/MAX 0 2 xxx.xxx.11.18 The CPU is not always 100%, it is changing 25% 45% 85% ....
Posted by javii, 09-17-2010, 06:55 AM
how could i determine if it is a IO problem? thank you for your help
Posted by care4server, 09-17-2010, 01:44 PM
try command sar 1 121
Posted by javii, 10-27-2010, 04:36 AM
root@xxxxx ~]# sar 1 121 Linux 2.6.18-028stab070.4 (xxxxxxxxx) 27/10/10 10:35:42 CPU %user %nice %system %iowait %steal %idle 10:35:43 all 19,05 0,75 8,52 15,29 0,00 56,39 10:35:44 all 28,25 0,00 13,75 45,00 0,00 13,00 10:35:45 all 27,36 0,25 13,18 44,03 0,00 15,17 10:35:46 all 35,41 0,25 16,46 3,24 0,00 44,64 10:35:47 all 19,10 0,00 9,05 12,56 0,00 59,30 10:35:48 all 32,92 0,50 15,46 2,74 0,00 48,38 10:35:49 all 20,20 0,00 7,98 15,96 0,00 55,86 10:35:50 all 13,53 0,00 12,03 0,50 0,00 73,93 10:35:51 all 15,46 0,25 7,23 4,49 0,00 72,57 10:35:52 all 25,25 0,00 9,75 0,75 0,00 64,25 10:35:53 all 30,85 0,00 24,63 1,99 0,00 42,54 10:35:54 all 51,37 0,00 32,17 3,49 0,00 12,97 10:35:55 all 42,25 0,50 13,50 18,25 0,00 25,50 10:35:56 all 44,97 0,00 23,37 8,04 0,00 23,62 10:35:57 all 47,76 0,25 14,93 5,47 0,00 31,59 10:35:58 all 18,75 0,00 4,50 2,00 0,00 74,75 10:35:59 all 14,25 0,00 6,00 1,75 0,00 78,00 10:36:00 all 29,82 0,25 18,55 3,26 0,00 48,12 10:36:01 all 22,69 0,25 12,97 1,00 0,00 63,09 10:36:02 all 39,70 0,00 34,74 1,99 0,00 23,57 10:36:03 all 47,63 0,00 37,16 2,00 0,00 13,22 10:36:04 all 39,45 0,00 24,12 1,51 0,00 34,92 10:36:05 all 32,67 0,00 27,18 2,49 0,00 37,66 I think, it is a CPU problem, right?