Apache Request Maxed

Portal Home > Knowledgebase > Articles Database > Apache Request Maxed

Posted by nickCR, 12-13-2011, 01:07 PM
Hello All, We have been having very infrequent yet completely random freezes on our server. We are hosted on a decent server and barely use 5% of the CPU capacity so I'm not sure where the issue is and I have come here in hopes that you guys can help me figure out what is going on. My host is telling me that the problem seems to be that the requests are overloading and causing the server to freeze momentarily (no more than 5 min and it comes back to life like nothing happened with no noticed from httpd or anything else). There are only 3 sites that run on the server and the main one is a daily deal site. So when the emails go out there is a TON of request for images. We increased the requests to 400 yesterday so I don't know if this solved the problem or not. I would greatly appreciate any help you guys can provide. Thanks in advance!
Posted by SolidJoe, 12-13-2011, 01:32 PM
Apache is poop for lots of simultaneous requests. You could investigate trying out Lighttpd or any of the other c10k compliant servers.
Posted by nickCR, 12-13-2011, 02:42 PM
I just got a reply back from the host that they are not sure that it was Apache but based on the request hovering around 160-200 when our emails go out I wouldn't be surprised if that's what is going on. It also all seems to be requests for images and such. So maybe going to a CDN might be the right way to go. I do see that about 5% of the request show "..reading..", not sure what that's all about or if I should be concerned about them. The server should be able to handle quite a few request as it has the following specs: Intel Xeon 5430 8 x 2.66 GHz/2 x 12M 6 GB RAM
Posted by ddrager, 12-13-2011, 03:32 PM
You didn't say what the specs of your discs are. If your apache says "Reading..." it is probably waiting to read the files to disk. You would likely benefit from memcached - since you said a lot of it is images from a single mailing, you are getting requests for the same files. Storing that in memory instead of reading it off the disk will help greatly. As the others have said, since it is static content, you may be better off storing it on a CDN. Apache also has maxes, so make sure they are high enough in your config. In contrast to SolidJoe, I do not think Apache is poop for simultaneous requests but it does need to be configured properly.
Posted by nickCR, 12-13-2011, 04:20 PM
It's just a 300GB SATA II drive. Can the caching be achieved via .htaccess or do I need to install a mod to get that to work? It really seems like the CDN is the easiest to implement though. Especially based on the volume of requests that we are getting.
Posted by timstephenson, 12-13-2011, 04:31 PM
A single SATA disk can be an IO bottleneck if it needs to deliver many concurrent reads. One option here would be to setup Varnish as a caching reverse proxy ahead of Apache. This would be able to handle requests for images & other static content, offloading repeat requests from Apache. A CDN would be another option if you have the ability to change the URLs you serve images from, but in the short term you may get more bang-for-buck from using Varnish. Without getting into a debate of Apache vs Nginx vs something else, I would also recommend removing Apache and replacing it with Nginx - performance out of the box with high numbers of concurrent requests tends to be significantly better than Apache.
Posted by nickCR, 12-14-2011, 02:25 PM
I don't see a problem with adding software, however I think it's going to be better and easier to just move everything over to a CDN as it will help out with our bandwidth which is sucked up like crazy right now.
Posted by Kernelosh, 12-14-2011, 03:45 PM
You can use Nginx for static content only(images, css, js and etc), If you want to save all the bottlenecks, use can setup nginx to reverse proxy from Apache and cache the files to /dev/shm (ram virtual disk).
Posted by twikamltd, 12-14-2011, 03:51 PM
If you're still within your BW limits though, Varnish in front of Apache would probably be the best option and easiest to setup, and you'll be amazed at the difference.
Posted by nickCR, 12-14-2011, 05:25 PM
Currently we're hitting about 50% of our bandwidth per month. Still lots of room but we are launching a new product next year and I know we'll have a huge spike in traffic triple if not quadruple. Due to the fact that the load and uptime is extremely important I will likely be moving the images to a CDN. I will ask about the Nginx to see if that's a viable option for my host to implement. Maybe a mid way solution for now.
Posted by ddrager, 12-14-2011, 05:31 PM
Moving to a CDN is pretty easy these days, this would be a simple solution to your problem. Offsetting the majority of the hits would definitely help with your server load.
Posted by nickCR, 12-14-2011, 05:37 PM
The problem is when we send out our emails and all the images from the email starts loading. We have a very large database and a significant number of those actually open the email so the images are the majority of the request. Amazon seems to have the least downtime right now. Would an S3 bucket be enough or would it be wise to also jump on with CloudFront?
Posted by ddrager, 12-14-2011, 05:40 PM
The S3 bucket is mainly the storage source, CloudFront is the actual CDN which replicates the download across the globe. Meaning, you could also use CloudFront while using your server as the origin. It's dirt cheap - I would highly recommend it. Used it as a CDN for a site which was techcrunched and biebered with no issues whatsoever. My monthly bill was only about $2-5.
Posted by timstephenson, 12-14-2011, 05:43 PM
S3 would be a good option to offload image storage with a minimum of effort. CloudFront is designed to work with many sources, but is optimised to draw content from S3 - so if you're setting up a fresh solution using both would be a good option. A simpler option to help get you going might be to consider using CloudFlare; simple implementation & you'll get the benefits of offloading bandwidth for your static objects (without much in the way of prices!).
Posted by nickCR, 12-15-2011, 01:58 PM
Thanks guys I've setup the S3 and going to add CloudFront. I can't quite figure out how CloudFront (like how to get the images to load from it) but I am working on that. I'm sure once I link my S3 account with cloudfront it will become a little more clear!
Posted by timstephenson, 12-15-2011, 04:44 PM
Good luck!
Posted by nickCR, 12-29-2011, 02:38 AM
I got everything setup and I even have a DNS CNAME for the CDN. I need to deploy the development version in the live environment to really get an idea for it will hold up. I'm pretty sure we'll be happy especially with CloudFront speeding up delivery. I'll report back.