View Single Post
Old 02-04-2013, 08:13 PM   #2
seb567
Senior Member
 
Location: Québec, Canada

Join Date: Jul 2008
Posts: 260
Default

I understand that Baiduspider does not respect the robots.txt !

http://seqanswers.com/robots.txt

User-agent: *
Disallow: /w/
Disallow: /wiki/Special:


User-agent: Baiduspider
Disallow: /
seb567 is offline   Reply With Quote