Robots.txt file.

[Fred]

Can some kind person remind me what text/code I should place in the
Robots.txt file page so that S/E spiders will crawl and index the site. I
currently use this but unsure if this is correct.
User-agent: *
Disallow: /cgi-bin/

Fred

 

[Stefan B Rusynko]

See http://www.searchengineworld.com/robots/robots_tutorial.htm




| Can some kind person remind me what text/code I should place in the
| Robots.txt file page so that S/E spiders will crawl and index the site. I
| currently use this but unsure if this is correct.
| User-agent: *
| Disallow: /cgi-bin/
|
| Fred
|
|

 

[David Baxter]

You don't actually need a robots.txt file to tell spiders to crawl your
site - you only need it to tell spider what NOT to crawl/index.

What you have here is correct, assuming you want everything indexed
EXCEPT the cgi-bin directory (which makes sense).