Looking for a newsgroup about steering web crawlers

S

Stan Hilliard

I looked at the log files of my Frontpage web. I am not expert in
reading log files but it is obvious that robots continuously search my
site. They go where I don't think they need to go. Like hit counter
files, obsolete folders, gif files, _vti folders, etc.

I need to learn what to put initially in a robots.txt file to provide
instruction to the robots. Is this the right place to discuss that or
is there another newsgroup for it?

Advice will be appreciated,
Stan Hilliard
 
T

Tom Willett

This should get you started:
http://www.robotstxt.org/wc/robots.html
--
===
Tom Willett
Microsoft MVP - FrontPage
---
FrontPage Support:
http://www.frontpagemvps.com/
===
|I looked at the log files of my Frontpage web. I am not expert in
| reading log files but it is obvious that robots continuously search my
| site. They go where I don't think they need to go. Like hit counter
| files, obsolete folders, gif files, _vti folders, etc.
|
| I need to learn what to put initially in a robots.txt file to provide
| instruction to the robots. Is this the right place to discuss that or
| is there another newsgroup for it?
|
| Advice will be appreciated,
| Stan Hilliard
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Robots/web crawlers 4
FP discussion web -- no submission form but new posts appear. 5
Exe inside a Zip -- will some firewalls reject it. 3
Problem deleting message files from discussion subweb. 6
_vti_cnf being made in non FP Web folders off Root! 4
Solved -- the incompatibility between FP discussions and FP extensions. 2
No login for Web Folder 19
Additional files & folders for disk based web 1

Top