Site slowdowns: causes and solutions
December 2, 2015
At first, we wanted to name this news item 'Slaying dragons' because that's a perfectly wonderful way to put it! But we figured it might not be very clear. So let's hear it for Matt, as he speaks clearly and plainly about the recent slowness of our site,
what caused it and what is being done about it. This is a compilation of
his forum postings on the matter.
Take it away, Matt:
"BookCrossing has been attacked by bots, in what seems to be causing somewhat of a DoS (Denial of Service attack). These bots are automated robots (in other words, computer programs, not people) that sign up for accounts and then spam membership on sites. Normally we get about 200-300 new members a day, some are spam, most are not. In the last 6 weeks it has grown to 600, then 1000, and lately 5000-6000 a day. I ran that report last night and then installed a new Google captcha on the join page, to stop the flow of new bogus members. That has worked, we have 178 members so far today ( = December 1st). But the bots are still coming and hitting the signup page, so far about 40,000 times today. I've installed monitoring software to try to catch them by IP so we can block them. That's a long answer for: we are working on it, and trying to get the site back under control."
"Ardik and I just finished working on the site tonight. We made some code changes to the pages that were being hit, and blocked the major offenders as best we could. The site came back to life once Ardik entered in a list of the worst of them. That said, we are brainstorming on some long term fixes to be able to identify and block these users automatically instead of having to pour through log files and other methods. More later, have a good night/day."
"It's now December 2nd. Ardik kept diligent watch through the night on my side of the globe and performance is still good! Some of the next steps we will be taking:
- The first one will seem odd, but over the years there has been a buildup of member accounts created by bots. We have identified many of them through email verification. Sometime this week we will be purging the database of these fake members, who do nothing but take resources from actual BookCrossing members.
- We will be adding the reCaptcha "I am not a robot" check box to the page where you can send private messages, for non-wings members. This one is easy, pretty non-intrusive, and prevents members from getting spam from automated bots that create accounts, and then login and send messages (these exist). I don't think many spammers are members paying for wings.
- We will be moving some files to a cloud server so that during peak times the web server doesn't have to send out images and other files. It makes a big difference.
- Ardik and I are coming up with a program solution to block IP addresses that abuse the site. This one will take longer, because it needs to be tested to make sure it doesn't block actual active users, but it is the long term goal."
Thank you, Matt, for these updates and explanations, and thank you both, Matt and Ardik, for slaying these dragons for us! We owe each of you a well-deserved pint of the best beer that money can buy.
Take it away, Matt:
"BookCrossing has been attacked by bots, in what seems to be causing somewhat of a DoS (Denial of Service attack). These bots are automated robots (in other words, computer programs, not people) that sign up for accounts and then spam membership on sites. Normally we get about 200-300 new members a day, some are spam, most are not. In the last 6 weeks it has grown to 600, then 1000, and lately 5000-6000 a day. I ran that report last night and then installed a new Google captcha on the join page, to stop the flow of new bogus members. That has worked, we have 178 members so far today ( = December 1st). But the bots are still coming and hitting the signup page, so far about 40,000 times today. I've installed monitoring software to try to catch them by IP so we can block them. That's a long answer for: we are working on it, and trying to get the site back under control."
"Ardik and I just finished working on the site tonight. We made some code changes to the pages that were being hit, and blocked the major offenders as best we could. The site came back to life once Ardik entered in a list of the worst of them. That said, we are brainstorming on some long term fixes to be able to identify and block these users automatically instead of having to pour through log files and other methods. More later, have a good night/day."
"It's now December 2nd. Ardik kept diligent watch through the night on my side of the globe and performance is still good! Some of the next steps we will be taking:
- The first one will seem odd, but over the years there has been a buildup of member accounts created by bots. We have identified many of them through email verification. Sometime this week we will be purging the database of these fake members, who do nothing but take resources from actual BookCrossing members.
- We will be adding the reCaptcha "I am not a robot" check box to the page where you can send private messages, for non-wings members. This one is easy, pretty non-intrusive, and prevents members from getting spam from automated bots that create accounts, and then login and send messages (these exist). I don't think many spammers are members paying for wings.
- We will be moving some files to a cloud server so that during peak times the web server doesn't have to send out images and other files. It makes a big difference.
- Ardik and I are coming up with a program solution to block IP addresses that abuse the site. This one will take longer, because it needs to be tested to make sure it doesn't block actual active users, but it is the long term goal."
Thank you, Matt, for these updates and explanations, and thank you both, Matt and Ardik, for slaying these dragons for us! We owe each of you a well-deserved pint of the best beer that money can buy.
