My Robots.txt file

[Zylstra]

k..

i'll try to make one myself and have you guys check it over..

how do i find the robots names??

Find it here

[unlovedwarrior]

k thanks

[Rob Pomeroy]

if you have a forum, you will want to block the admin directories and the member directories. This will keep important information about your users from being searched if the search engines manage to do that.

Still not convinced you've quite go this, by virtue of the fact you're using the word "block".  It's best not to have (or give) the impression that robots.txt files can actually block anything.  Consider them more as a polite request, which will sometimes be ignored.

You need to protect your /admin tree in much more robust ways.

[Zylstra]

if you have a forum, you will want to block the admin directories and the member directories. This will keep important information about your users from being searched if the search engines manage to do that.

Still not convinced you've quite go this, by virtue of the fact you're using the word "block".  It's best not to have (or give) the impression that robots.txt files can actually block anything.  Consider them more as a polite request, which will sometimes be ignored.
[highlight]
You need to protect your /admin tree in much more robust ways.[/highlight]

Mind explaining how?

[Dilbert]

Yes, please elaborate, I can use this info.

[unlovedwarrior]

im confused rob explain plz so do i really need robot.txt??

[Rob Pomeroy]

Mind explaining how?

Yes, please elaborate, I can use this info.

Sure.  WIth Apache, you can use an .htaccess file as an added layer of protection.  I suggest using password and IP-based protection.  If you will only access /admin from your local subnet, then only allow access from that subnet.  If you are connecting to a remote server, but from a fixed IP address, only allow access from that IP address.

You can do something similar with IIS and WIndows-based authentication.

im confused rob explain plz so do i really need robot.txt??

Well this is my point.  The Computer Hope article explains it quite well.  You tell spiders not to index parts of the web tree that it would be pointless for them to access - e.g. recursive directories (not something you're likely to encounter for a while) or any thread on YaBB that contains a post by Mac...

[Zylstra]

Hmm. I already have HTACCESS, but not on my YaBB directories...

[unlovedwarrior]

i have the htaccess file also im using phpbb

[Dilbert]

or any thread on YaBB that contains a post by Mac...

ROTFL

Thanks, Rob, I'll keep that in mind.

So now I know what robots.txt is... thanks, Rob.

[Google]

You have reached the point of a "VERY HOT TOPIC" at [timestamp=1155312980]!
[highlight]CONGRATULATIONS![/highlight]