| Not logged in - | |
| Not logged in - | |
Come on over and join us at the new place- www.ultrabb.net/forum
|
| Moderated by: Jim |
|
||||||||||||||
| Yahoo Slurp - Software Assistance - UltraBB - UltraBB Forums | |||||||||||||||
| Author | Post | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||
|
zzkenoman Epicurean Rider 
|
I was getting enormous web crawlers to my forum and wanted to let you know Yahoo had launched a new indexing program April 14th. I found instructions to lessen the problem, I think. http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html
|
||||||||||||||
| |||||||||||||||
| |||||||||||||||
|
Jim Administrator 
|
It's actually a good thing, well for your sites popularity anyway. I can put a file in there if you want or you can, but basically even the delay will slow them down meaning you will not get crawled as much. Maybe Yahoo follows the rules but I have seen with my own eyes bots that ignore the robots.txt file and crawl everything anyway. Whatever you want to do is fine but I would only exclude directories outside the main one.
|
||||||||||||||
| |||||||||||||||
| |||||||||||||||
|
John Floyd License Holder 
|
I don't care if they ever crawl my site, my Membership is limited and private, Could you apply this fix for me.
|
||||||||||||||
| |||||||||||||||
| Current time is 12:31 pm | |
| UltraBB Forums > UltraBB > Software Assistance > Yahoo Slurp | |
