Oh boy... I need some MAJOR help - ! Server Problem - Knowledgebase

Portal Home > Knowledgebase > Articles Database > Oh boy... I need some MAJOR help - ! Server Problem

Oh boy... I need some MAJOR help - ! Server Problem

Posted by solecollector, 12-25-2009, 05:34 PM
I have a great hosting company in Iowa and they are pretty sharp - but I have a problem right now that must be fairly complex because the server has been having problems for two days now - and it hurts...! Here is the quick run-down: I have a forum with 160,000 members on it and it runs fine - optimized for the server and gets a little slow at peak times - but runs solid. I also have 5 wordpress blogs (2.9) on the server - they all work fine. And earlier this week we put our first iPhone application up (it was approved by Apple on Sunday night) and this is where the fun begins. To do push notifications we needed openSSL to be enabled. My hosting provider upgraded PHP to 5.2.12 and then enabled openSSL. The server started to get super high load averages - 700+. They reverted back to the original setup - but we are still having this problem. When we restart httpd the server runs fine for awhie - sometimes 20 minutes - sometimes 3 hours - but most of the time 20 to 30 minutes. All the sudden there is a huge spike in the server load and we have to shut things down and bring them back up again. The iPhone app guys worked on the server all night but really didn't get anywhere. The problem my hosting provider and my tech guy thought was going on was the iPhone app leaving connections open (we did an e-mail blast out to 100,000 people so we had a spike in usage pretty quickly - like 500 people or so) but now I don't see any waits. The server is cPanel Ver. 11.25.0-RELEASE Main domain is solecollector.com Can anyone help me out on this? What direction should I go?
Posted by funkywizard, 12-25-2009, 05:44 PM
I would move the iphone app to another server as it's the main thing that's changed since things were running well, especially since the site is running a bit slow during peak already. The iphone app might be getting a lot of traffic or it might have some horrible bugs. Either way it should probably be off the main server
Posted by solecollector, 12-25-2009, 05:47 PM
The problem is that it is the content the iPhone app calls is on the server - and that is built into the iPhone app and is on the iPhone. So even when we took away the iPhone directory on the server - all those calls where still going onto the server looking for that specific directory. So what we did was created an xml page for the iPhone users to see - and that solved the problem. We can update the iPhone app software - but have to wait until the 28th.... Apple is closed right now. If I had to do this over - I would have built out a server just for this app as we have two more on the way.... but... I honestly don't know if this is the problem we are having now. All of the waits are in control and we are still getting the hammer on the server.
Posted by solecollector, 12-25-2009, 05:50 PM
Is there an area on this forum that showcases consultants that can help in this area? I need someone pretty sharp as the people who have worked on this problem are no slouches. Well... I am - but they aren't. I don't mind trying to fix it through this thread - and if I could... someone would get a nice bottle of something great -!!!!!!!!!!
Posted by funkywizard, 12-25-2009, 05:56 PM
You could use squid or nginx or something to direct requests from the forum to one server and requests for the iphone to a different server. Also both of those front ends can handle a lot more connections than apache without bloating ram, so if the iphone app keeps connections open when it shouldn't you won't have as many problems with that
Posted by OpenInternet-Vince, 12-25-2009, 06:10 PM
Are the connections to the HTTP stay connected? If so, try cutting down the connection timeout.
Posted by solecollector, 12-25-2009, 06:11 PM
When I asked our hosting guy yesterday about this (I think this is what you are saying) this is what he said: It’s a port 80/http call to the custom iPhone CMS, it doesn’t run on a unique port unfortunately. Here’s a sample: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 1 0 ::ffff:147.202.23.200:80 ::ffff:173.79.220.43:56147 CLOSE_WAIT 29493/httpd tcp 1 0 ::ffff:147.202.23.200:80 ::ffff:74.72.39.213:2129 CLOSE_WAIT 29709/httpd tcp 0 0 ::ffff:147.202.23.200:80 ::ffff:99.254.109.74:53810 CLOSE_WAIT 29485/httpd tcp 1 0 ::ffff:147.202.23.200:80 ::ffff:99.41.50.92:51116 CLOSE_WAIT 29497/httpd tcp 0 0 ::ffff:147.202.23.200:80 ::ffff:99.254.109.74:53811 CLOSE_WAIT 30025/httpd tcp 1 0 ::ffff:147.202.23.200:80 ::ffff:99.41.50.92:51117 CLOSE_WAIT 29618/httpd
Posted by solecollector, 12-25-2009, 06:15 PM
From last night - [root@web ~]# netstat -nap | grep CLOSE_WAIT | wc -l 0 [root@web ~]# w 07:39:10 up 1 day, 18:45, 1 user, load average: 0.29, 0.28, 0.20 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT solecoll pts/0 122.164.213.53 06:52 0.00s 0.10s 0.02s sshd: solecoll [priv] [root@web ~]# date Fri Dec 25 07:39:17 CST 2009
Posted by funkywizard, 12-25-2009, 06:16 PM
Yes but if you put nginx in front it can accept all connections and proxy back iphone requests to one server and other requests to another server
Posted by net, 12-25-2009, 06:49 PM
Moved > Hosting Security and Technology .
Posted by Steven, 12-26-2009, 12:52 AM
I suggest running your application through xdebug and profile it to make sure theres no slow downs in the code that bring problems when there are many concurrent connections. It's not uncommon to see an application die after a small number of concurrent conenctions due to simple code issues. One customer of mine about a year ago launched a recoded version of their site, a line of code was on the wrong side of a for loop and it caused severe cpu and ram utilization with just 5 concurrent connections, because it would loop 100-200 times per page load. Xdebug: http://xdebug.org/ Profiler: http://xdebug.org/docs/profiler Since the iphone app was the only thing added besides the recompile (which shouldn't do this, also you reverted and it remains), I am going to have to say its probably a problem with the application itself. Last edited by Steven; 12-26-2009 at 12:56 AM.