robots.txt

[Sitara Lal]

Is it a good idea to have a 'robots.txt' in the root directory? I presume
FP2003 does not install one automatically and I was wondering whether I
should add it manually?

If yes, what would be the typical content of such a file?

Thanks

 

[Tom [Pepper] Willett]

Having a robots.txt file is up to you, based upon your judgement that you
want/need one.

You must create it manually. There are many tutorials available, including:
http://www.searchengineworld.com/robots/robots_tutorial.htm

 

[Stefan B Rusynko]

IMHO
- always at least create an empty file named robots.txt to eliminate all the file not found errors (from web crawlers and search
engines) in your log files
--

_____________________________________________
SBR @ ENJOY (-: [ Microsoft MVP - FrontPage ]
"Warning - Using the F1 Key will not break anything!" (-;
To find the best Newsgroup for FrontPage support see:
http://www.frontpagemvps.com/FrontPageNewsGroups/tabid/53/Default.aspx
_____________________________________________


| Having a robots.txt file is up to you, based upon your judgement that you
| want/need one.
|
| You must create it manually. There are many tutorials available, including:
| http://www.searchengineworld.com/robots/robots_tutorial.htm
| --
| Tom [Pepper] Willett
| Microsoft MVP - FrontPage
| FrontPage Support: http://www.frontpagemvps.com/
| ----------
| | > Is it a good idea to have a 'robots.txt' in the root directory? I presume
| > FP2003 does not install one automatically and I was wondering whether I
| > should add it manually?
| >
| > If yes, what would be the typical content of such a file?
| >
| > Thanks
| >
|
|

 

[steve]

IMHO
- always at least create an empty file named robots.txt to eliminate all the file not found errors (from web crawlers and search
engines) in your log files

That's, more or less, what I do. Occasionally I add a Disallow: for
specific temporary files.

 

[Steve Easton]

The only problem with adding a Disallow is that the robots text file can be
read by anyone.
So having a Disallow simply shows them what folders you don't want browsed.

--
Steve Easton
Microsoft MVP FrontPage
95isalive
This site is best viewed............
........................with a computer

 

[steve]

The only problem with adding a Disallow is that the robots text file can be
read by anyone.
So having a Disallow simply shows them what folders you don't want browsed.

I've nothing to hide, I don't care who browses them, they would be
asleep in seconds. My only reasons for Disallow is to stop hundreds of
errors in my web stats, avoid simple redirection pages and WIP pages
getting indexed.

 

[Thomas A. Rowe]

FYI: Only the "good" spiders will honor the content of the robots.txt file, the "bad" spiders will
still index any accessible content.

--
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
==============================================
If you feel your current issue is a results of installing
a Service Pack or security update, please contact
Microsoft Product Support Services:
http://support.microsoft.com
If the problem can be shown to have been caused by a
security update, then there is usually no charge for the call.
==============================================

 

[MDBJ]

no.. the 'BAD' spiders will FOCUS on the disallowed content.

 

[Thomas A. Rowe]

The bad spiders in most cases don't even request robots.txt file.

--
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
==============================================
If you feel your current issue is a results of installing
a Service Pack or security update, please contact
Microsoft Product Support Services:
http://support.microsoft.com
If the problem can be shown to have been caused by a
security update, then there is usually no charge for the call.
==============================================

 

[Thomas A. Rowe]

John, thanks, but I won't really be back until after May 1st.

--
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
==============================================
If you feel your current issue is a results of installing
a Service Pack or security update, please contact
Microsoft Product Support Services:
http://support.microsoft.com
If the problem can be shown to have been caused by a
security update, then there is usually no charge for the call.
==============================================

 

[John Malone]

Welcome Back Thomas!
--
John Malone
==============
| The bad spiders in most cases don't even request robots.txt file.
|
| --
| ==============================================
| Thomas A. Rowe (Microsoft MVP - FrontPage)
| ==============================================
| If you feel your current issue is a results of installing
| a Service Pack or security update, please contact
| Microsoft Product Support Services:
| http://support.microsoft.com
| If the problem can be shown to have been caused by a
| security update, then there is usually no charge for the call.
| ==============================================
|
| > no.. the 'BAD' spiders will FOCUS on the disallowed content.
| >
| >> FYI: Only the "good" spiders will honor the content of the robots.txt
file, the "bad" spiders
| >> will still index any accessible content.
| >>
| >> --
| >> ==============================================
| >> Thomas A. Rowe (Microsoft MVP - FrontPage)
| >> ==============================================
| >> If you feel your current issue is a results of installing
| >> a Service Pack or security update, please contact
| >> Microsoft Product Support Services:
| >> http://support.microsoft.com
| >> If the problem can be shown to have been caused by a
| >> security update, then there is usually no charge for the call.
| >> ==============================================
| >>
| >>> On Sun, 19 Feb 2006 10:53:22 -0500, "Steve Easton"
| >>>
| >>>>The only problem with adding a Disallow is that the robots text file
can be
| >>>>read by anyone.
| >>>>So having a Disallow simply shows them what folders you don't want
browsed.
| >>>
| >>> I've nothing to hide, I don't care who browses them, they would be
| >>> asleep in seconds. My only reasons for Disallow is to stop hundreds of
| >>> errors in my web stats, avoid simple redirection pages and WIP pages
| >>> getting indexed.
| >>>
| >>> --
| >>> Steve Wolstenholme Neural Planner Software
| >>>
| >>> EasyNN-plus. The easy way to build neural networks.
| >>> http://www.easynn.com
| >>
| >>
| >
| >
|
|

 

[Joe Rohn]

Good to see you back as well Tom!

Sometime after May 1st when you get settled..give me a call!

--
Joe

Microsoft MVP FrontPage

FrontPage Portal and Users Forums:
http://www.timeforweb.com/frontpage

 

[Thomas A. Rowe]

Ok.

--
==============================================
Thomas A. Rowe (Microsoft MVP - FrontPage)
==============================================
If you feel your current issue is a results of installing
a Service Pack or security update, please contact
Microsoft Product Support Services:
http://support.microsoft.com
If the problem can be shown to have been caused by a
security update, then there is usually no charge for the call.
==============================================

 

[MDBJ]

You must be a generally good person.

"bad guy spiders" aren't in it for 'legitimate' purposes, and likely don't
have the funds/resources
to spider the entire internet.. but rather just the slices of info that
compel them to be "bad guys"

... think like a bad guy for a minute..
does not the content of the "disallow" robots.txt seem more appealing than
having to go through the whole site?

now think like a good citizen for a minute

http://www.whitehouse.gov/robots.txt

That's EVERYTHING the US Govt wants to make sure is NOT available to its
citizens via third parties like google...