The need for robot.txt file on website

[Tommy Desperate]

I subjected my website to one of those "spider tests" that SEOs give free
access to. My site (no surprise) did not pass the test. I was wondering about
one specific comment in the test results:

" The robots.txt file should live in the topmost (root) directory of your
website. It tells spiders what parts of the site they cannot read (e.g. your
admin area.) They always check for this file so if you do not have one it
will cause error messages in the logs of your site. If you want a file that
lets the spiders read all of your pages but stops the errors, create a new
text file called robots.txt and cut and past the following two lines in to
it: User-agent: *Disallow:"

I have no pages on my website that I want kept secret from the spiders. Do I
still need the robot.txt file on the root web?

 

[Thomas A. Rowe]

No, if you don't mind the "file not found" message in the server log files

--
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
==============================================
If you feel your current issue is a results of installing
a Service Pack or security update, please contact
Microsoft Product Support Services:
http://support.microsoft.com
If the problem can be shown to have been caused by a
security update, then there is usually no charge for the call.
==============================================

 

[Kevin Spencer]

To clarify: Most spiders today (e.g. Google) are smart enough not to publish
links that are to folders containing, for example, FrontPage server
extensions. However, they will make some requests that will show up as 404
errors in your server logs.


--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
Everybody picks their nose,
But some people are better at hiding it.

 

[David Baxter]

You don't need it, no. But it doesn't hurt to have it and most major
search engine spiders do look for it.

Create robots.txt as a text (ASCII) file with the following:

User-agent: *
Disallow:

Upload it to your root directory. Now everyone will be happy.

 

[Kevin Spencer]

Upload it to your root directory. Now everyone will be happy.

I would hate to be the only one that is!

--
;-),

Kevin Spencer
Microsoft MVP
..Net Developer
Everybody picks their nose,
But some people are better at hiding it.