SEQanswers

SEQanswers (http://seqanswers.com/forums/index.php)
-   Site Announcements (http://seqanswers.com/forums/forumdisplay.php?f=2)
-   -   Updates on traffic/errors... (http://seqanswers.com/forums/showthread.php?t=26850)

ECO 01-25-2013 11:00 AM

Updates on traffic/errors...
 
So if you've been here for the last three days you've probably noticed some sporadic performance...apologies for that.

I'm continuing to monitor it (and please continue emailing me if you run into problems), but it seems like the cause was that Baidu (search engine) has been indexing the wiki and forums with a crazy amount of robotic power. Meaning 6-8 requests per second, from ~30 different IPs, of different complex wiki pages that bring the database server to it's knees.

I am now blocking their IPs, we'll see if it holds.

Grrrrrrrrrr.

seb567 02-04-2013 08:13 PM

I understand that Baiduspider does not respect the robots.txt !

http://seqanswers.com/robots.txt

User-agent: *
Disallow: /w/
Disallow: /wiki/Special:


User-agent: Baiduspider
Disallow: /

getinhours12 03-30-2020 12:38 AM

Thanks for sharing useful information.


All times are GMT -8. The time now is 07:36 AM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.