The Digg of Death

Digg.com is a fantastic site, it has an amazing influence on making stuff popular. But, it is also a constant burden on website owners. If you get "dugged" you got to have a very good hosting plan in order to stay online.

I have been dugged about 5 times within the last 6 month or so. Most of the times has not been very spectacular with only a few amount of digs. But, 2 times the traffic increase so much and so fast that my website crashed.

The spikes

1: The first time was in August, just after I had released WEB2DNA. Once that got on Digg.com the website was literally under siege. It crashed 4 times because of the high amount of traffic, and I eventually had to redesign how each DNA was made in order to handle the high traffic loads.

Eventually, my "15 minutes of fame" was over, and the traffic returned to its normal level.

2: The second time was two days ago, when an old article about the usability of package tracking appeared on the front page of Digg.com. As you can see, that created a amazing spike in traffic. In the morning (European time) I had between 2,000 - 4,000 visitors per hour. After the initial barrage things seemed to slow down, as most people in America went to bed.

But, they only slept for about 8 hours, and when they woke up they came back in force. The number of people coming in was so high that the server crashed and stayed down for about 2 hours. A few was lucky enough to get in, but it took ages to load the page - and the images would did not load.

Don't get me wrong, I like getting dugged - it is fun! But, my server does not seem to agree with my point of view.

How to avoid Digg Death?

There are a number of things you can do to prevent overloading the server.

• Reduce server handling. This is by far the most important one. Many sites uses server side scripting, and that requires a lot from the CPU when you got flood of people coming in. Limit the amount of things your server need to do, reduce your server side scripting, database requests, data conversions etc.
• Reduce the client data. If you site is 100kb in size, reducing that to 50kb will effectively allow you to serve twice as many people.Â
• Reduce external files. Instead of having 5 external CSS files, use one. Reduce the amount ads you serve from Google Adwords or other suppliers. Reduce the amount of images on each page. It all helps.

And if none of these work, you need to look at your hosting solution.

• Get another host, I would choose one of the bigger ones - even if they slightly more expensive (I did this after the first Digg "flood").
• Consider using a dedicated server. This way you avoid having to share your server load with other websites.
• Set a limit to how many visit you can handle. A web server is usually configured to handle everything, with the result that when too many people come in, it will start to slow down and eventually fail (like mine did). Setting a limit will instead instruct the server to only handle a certain amount of people and everyone else will get an error page saying something like:
  
  "Sorry, things are getting crowded, please wait outside for your turn.
  While you wait try figuring out what this means: 'lesu i//n fif'"

You can also consider to simply ignore the problem. After all, getting dugged does not happen very often.

For my part, I have already made some reductions. I have found a new host (which did a much better job than the old one, but still could not handle it) and I have some other things I might change in the future.

More from Baekdal

How scared should we be of AIs taking our jobs?

30 pages