What goes in a robots.txt file for more protection?

[robots]

If there is a robots.txt file in the root directory, how can spiders still
list all my files even in the MEMBERS-ONLY section

 

[Dennis]

If there is a robots.txt file in the root directory, how can spiders still
list all my files even in the MEMBERS-ONLY section

Spiders aren't obligated to obey robots.txt.

 

[Wayne-I-M]

ooops - forgot to put the second link in -


Hi

Steep learning curve for me.

I am just start to create a CSS web and have created a simple drop down menu
http://www.ukoutdoorpursuits.org/Thisisatest1.html

Which "seems to work OK when I add in another couple of DIVs below (content
and Footer - note the content DIV will hold the editable region). The Header
DIV, Nav DIV and Footer DIV will only be editable on the DWT.

But as you can see – the “Hidden†is not working like it does in frontpage.
Although you can’t see the drop downs they are still there – and clickable.
Drag you mouse over the Contact DIV (white bit) and the drop downs appear -
???? (2nd LINK BELOW)
http://www.ukoutdoorpursuits.org/Thisisatest1.html
Note - the transparency of the drop downs will be sort (this is just a test


But the main problem is this anything on the Content DIV is always on top -
so the drop downs can’t be seen
http://www.ukoutdoorpursuits.org/Thisisatest3.html

Am obviously doing something very basic which is wrong - any ideas would be
really helpful.

Designed in Win7

Thank you

 

[Tom Willett]

Why did you hijack someone else's unrelated thread?
--
Tom [Pepper] Willett
Microsoft MVP - FrontPage Since 1997
---------------------------
: ooops - forgot to put the second link in -
:
:
: Hi
:
: Steep learning curve for me.
:
: I am just start to create a CSS web and have created a simple drop down
menu
: http://www.ukoutdoorpursuits.org/Thisisatest1.html
:
: Which "seems to work OK when I add in another couple of DIVs below
(content
: and Footer - note the content DIV will hold the editable region). The
Header
: DIV, Nav DIV and Footer DIV will only be editable on the DWT.
:
: But as you can see - the "Hidden" is not working like it does in
frontpage.
: Although you can't see the drop downs they are still there - and
clickable.
: Drag you mouse over the Contact DIV (white bit) and the drop downs
appear -
: ???? (2nd LINK BELOW)
: http://www.ukoutdoorpursuits.org/Thisisatest1.html
: Note - the transparency of the drop downs will be sort (this is just a
test
:
:
: But the main problem is this anything on the Content DIV is always on
top -
: so the drop downs can't be seen
: http://www.ukoutdoorpursuits.org/Thisisatest3.html
:
: Am obviously doing something very basic which is wrong - any ideas would
be
: really helpful.
:
: Designed in Win7
:
: Thank you
:

 

[Hot-text]

#
# From HTML-Kit
# This is an empty robots.txt file that can be used to indicate to
# WWW robots which parts of a web site should not be indexed.
#
# Following page has details on how to setup a robots.txt file and
# related security issues:
#
# http://www.chami.com/tips/internet/010198I.html
#


# 1. Uncomment following two lines by removing "#" at the beginning to
# indicate that all robots should stay away from the /cgi-bin/ folder

#User-agent: *
#Disallow: /cgi-bin/


# 2. Following two lines indicate that robots named "WWW_Robot_Name"
# should stay away from this site.

#User-agent: WWW_Robot_Name
#Disallow: /


# 3. Following two lines indicate that robots named "WWW_Wanderer_Name"
# should not index files in the /private/ folder.

#User-agent: WWW_Wanderer_Name
#Disallow: /private/

 

[Tom Willett]

And, any visitor can easily open the robots.txt file and look at it.
--
Tom [Pepper] Willett
Microsoft MVP - FrontPage Since 1997
---------------------------
: #
: # From HTML-Kit
: # This is an empty robots.txt file that can be used to indicate to
: # WWW robots which parts of a web site should not be indexed.
: #
: # Following page has details on how to setup a robots.txt file and
: # related security issues:
: #
: # http://www.chami.com/tips/internet/010198I.html
: #
:
:
: # 1. Uncomment following two lines by removing "#" at the beginning to
: # indicate that all robots should stay away from the /cgi-bin/ folder
:
: #User-agent: *
: #Disallow: /cgi-bin/
:
:
: # 2. Following two lines indicate that robots named "WWW_Robot_Name"
: # should stay away from this site.
:
: #User-agent: WWW_Robot_Name
: #Disallow: /
:
:
: # 3. Following two lines indicate that robots named "WWW_Wanderer_Name"
: # should not index files in the /private/ folder.
:
: #User-agent: WWW_Wanderer_Name
: #Disallow: /private/
:
:
: : > If there is a robots.txt file in the root directory, how can spiders
still
: > list all my files even in the MEMBERS-ONLY section
:

 

[Hot-text]

And he need to Generator a Google Sitemap/XML and a Yahoo Map/Text for the
Robots will help too Tom [Pepper] Willett
If he Read to here he see this link
http://wonderwebware.com/sitemap-generator/
But he get the robots.txt he ask for right!