site does NOT provide the /robots.txt file

[Dan]

After doing a SE ready check at Scrub the Web I received this message: This
site does NOT provide the /robots.txt file or the /robots.txt file is not
constructed properly. What are they talking about here? How would this tag
lay out?

 

[Kathleen Anderson [MVP - FP]]

This site should help: The Web Robots FAQ... http://www.robotstxt.org/wc/faq.html

--

~ Kathleen Anderson
Microsoft MVP - FrontPage
Spider Web Woman Designs
http://www.spiderwebwoman.com/resources/

 

[Mark Fitzpatrick]

A robots.txt file is basically a file that tells a search engine what files
to include/exclude kind of thing. This isn't something that FP or other
editors create for you, it's basically something you create for yourself.
Check out: http://www.robotstxt.org/wc/robots.html for some info on robots
exclusion.

Hope this helps,
Mark Fitzpatrick
Microsoft MVP - FrontPage

 

[Dan]

Does the meta tags do the same thing? If so why would you need the
robots.txt file at all?
<meta name="robots" content="Index,Follow">
or
<meta name="robots" content="All">

 

[Thomas A. Rowe]

The Robot.txt file is only for excluding content from being indexed, and
unless you have your own domain, you can use the robot.txt file.

You have more control when using the robot meta tags. As you can specific,
NoIndex,Follow which tells the search engine to not index the page, but
follow any links on the page.

But keep in mind that not all search engine read or honor the robot.txt file
or the robot meta tags.

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================