What is causing high server loads?

[subnet_rx]

I have server load spikes from time to time that pretty much disable my site. I'd like to be able to track down what is causing these loads, but I'm not that great at server administration. I realize that top can show me what's using processor time, but just telling me the apache process or mysql process does not help me to track down what exactly is behind this. How can I find out what's causing these loads specifically?

[JLeuze]

Hi Subnet, I'm by no means a pro at managing my server either, but I have had to try and rein in my resource consumption too.

For me, with a lower-end VPS package, running low on RAM has been my biggest issue. Steadfast has some documentation helped me tune down my RAM consumption a bit. Do a search for "Reducing Memory Usage" in the support section.

I have made a habit of reading my logs, and it has helped me isolate some resource hogs by looking at what was going on on the system at the time of the slowdown or crash.

Another thing that I have found helpful is keeping track of my stats. I have all of my domains set up with Google Analytics so that I can see which ones are getting the most traffic at certain times. I figure that the sites that are getting the most traffic need the most fine tuning, regardless of which ones are actually configured the best.

[derwert]

High server load can be caused by may different things.
Most common issues would be:

* Excessive number of connections to the server, each connection uses CPU and memory. With Keep-Alive each user should only use around 4-7 connections(educated guess based on past experience).
- If you see IP address(es) with an excessive amount of connections it could be an attempt to DoS or DDoS your site/server.
- You can view current connections by running "netstat -n" (the -n is so it doesn't try to do a DNS lookup for each address)
- You can also watch your logs for your HTTP daemon(Apache/lighttpd/etc). If you run "tail -f filename", that'll display your log in almost realtime, so you can watch for excessive requests. (press ctrl+c to terminate it)
- If it is a DoS then you should set limits for the number of connects an IP address can open to the daemon.
- If it is a DDoS then depending on the size there may be nothing you can do. If it's a flood then you could try to block in the firewall based on a pattern such as a specific request/user-agent. You'd have to parse your logs and basically setup something automated such as a script to do the work. This would still use CPU due to the firewall but not nearly as much as if the connections were allowed to continue.

* SPAM - Spam is distributed in a couple different ways: hacked servers, infected personal computers and deliberate servers setup for the purpose of spamming and relaying through open and also valid mail servers(*cough* gmail *cough*).
- I've experienced such a large amount of spam being sent to a server that it would bring the server to its knees(even though the specs for the server were fairly high).
- Virus scanning and filters can contribute toward higher CPU usage during a spam attack.
- You can also watch your mail daemon logs with "tail -f filename"
- To battle against mail from non-mail servers(i.e. infected computers) you can use grey listing.
- Having mail distributed between multiple mail servers is one way to ease the load.


* Site programming is also a cause of high CPU usage.
- Image generation can easily use up a lot of the CPU. (This isn't limited to just images)
- Database queries can also contribute to this. Look at the queries to your database and try to improve them, make sure you're using indexes correctly. If there are any repetitive intensive queries, try to cache the results so the queries won't have to occur as much.
- If your site is using PHP then you could try opcode caching, which saves some CPU so the scripts don't have to be repeatedly converted to opcodes each time they are requested.

If the above aren't the case then it may be time for you to look at upgrading to a better server as it sounds like you may have out grown your current one.