I guess growing pains are good, if you’re growing like this:
You can see the dip in traffic on Christmas, and the little dip right before the skyrocketing from 20-25,000 hits/day to 40,000 hits/day was when we were having server problems.
The problem is that http://www.writingup.com grew so fast that we had major server problems. The week between christmas and new years was really a struggle for us. The server went down for hours at a time, basically time after time after time.
We were getting so many hits that our little server couldn’t handle it…
Or so we thought.
It turned out that mysql just needed some tweaking. I guess mysql by default has caching disabled so we turned on caching by adding this line to the /etc/my.cnf file:
query-cache-size = 20M
As soon as mysql started caching queries it immediatly started working again.
Before we added that our server load had slowly gone from .2 up to about 30 before it would just die. Mysql was getting so many queries (hundreds per second) that it couldn’t do them all, so it would start getting slow queries (queries that took over 2 seconds) and those take up a lot of memory. The system would take up all the memory, and then start using the swap space until there wasn’t any swap left. Then it would just die.
After turning on caching to mysql it never gets overloaded. It never gets behind. I’ve seen it doing 565 queries per second, which is more than it can handle for a sustained period of time, but it handled it just fine.
We also added this:
key_buffer_size = 64M
Which helped a little, but not as much as turning on caching (I don’t really remember what this one did, but I think it allows mysql to use more memory when searching through tables that have keys in them).
Anyway, the server has been up now for over 7 days without any problems. We’re still going to get a bigger machine for it, so that it can grow more.