Re: UKNM: robots.txt - are you at risk?

[ben]

[Sam says: that's probably as much of robots.txt as anyone can take for the

moment, shall we draw a vale over this one (misspelling intended)]


Craig Pickup <

c [dot] pickupdigitrade [dot] co [dot] uk

> wrote ..

>

> Correct me if I am wrong (which is often the case) but the search engine

> bots, spiders, etc will only index pages either specifically submitted or

> linked to from somewhere else. If it ain't linked to the bot won't find

it.

> So if something is of such merit that you don't wish crackers to find it

> then don't link to it (or they will find it anyway) and you will not need

> to include it in the robots.txt file.


You're right but robots.txt allows you to specify certain directories as not

to indexed even though links  may take the bot there.


It is these areas that you have to be careful about and ensure that either:-


1) they are password protected in some way of other (.htaccess)

2) Directory Indexing is switched off

3) The directory includes an index page (what it does, doesn't matter it

merely stops anyone else seeing the other stuff in the directory).


Remember we are assuming that kiddie hackers are looking at the robots file

and then going to the areas that are excluded.


Also be careful if like us, you do an awful large amount of on the fly

redirecting of requests. robots.txt operates on the URL version of the name

not the actual location make sure the security is correct on both versions.

(Thankfully for us, everything ends up in 1 of 6 directories so we know

whats going on but each case is different).


Ben



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 the UK's totally managed affiliate marketing solution.

 ukaffiliates.com >> the net.working

 http://www.ukaffiliates.com /

salesukaffiliates [dot] com (mailto:salesukaffiliates [dot] com)

 telephone: 020 7691 1880 / fax: 020 7691 1881

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  To unsubscribe or change your list settings go to

http://www.chinwag.com/uk-netmarketing or

helpchinwag [dot] com