Robots/web crawlers

[Mike]

Is there a code that I can put on a particular HTML
document so that robots/web crawlers will not go through
it? I only want this on one document, but still want the
robots/crawlers to scan the other documents. Thanks,
Mike.

p.s. I am using Frontpage 2002.

 

[Steve Easton]

For those search bots that honor the meta tag,

<meta NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

between the head tags will do it.

 

[Thomas A. Rowe]

To have the robots not index the page but follow any links on the page use:

<meta name="robots" content="NOINDEX,FOLLOW">


To have the robots not index the page or follow any links on the page use:

<meta name="robots" content="NOINDEX,NOFOLLOW">


To have the robots index the page but not follow any links on the page use:

<meta name="robots" content="INDEX,NOFOLLOW">

--

==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
WEBMASTER Resources(tm)

FrontPage Resources, Forums, WebCircle,
MS KB Quick Links, etc.
==============================================

 

[JDR]

Mike,

Here is a meta tag you can use:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

This is not full proof as some search engines don't
support the tag.

The better solution is to write a Robots.tx file. Here
is a link that describes the file and how to create one:
http://www.robotstxt.org/wc/exclusion-admin.html

Hope it helps,
JDR

 

[Steve Easton]

The ones that ignore the meta tags also ignore the
robots.txt file