Re: UKNM: Spidering frames

[Paul Sanders]

>Well, if you have a very, very cunning database driven site, you

>could proably build a spiderable version of it which didn't have

>frames, for arachnid consumption.


Cunning sites are as much of a headache as frames if you want the spiders

to come crawling.


We struggled with this one - take a look at http://motion.state51.co.uk/ It

runs on Apache with mod_perl, nearly all dynamic, and faking HTTP headers.

It's a big fat fly as far as the spiders are concerned.


One really great unforseen benefit - if you are detecting agents before you

deliver the content you can deal with those evil EmailSiphon bastards and

their like. Another of our sites redirects them to a certain large software

company's site...


Cheers


Paul


-

Paul Sanders - a member of the state51 conspiracy

http://www.state51.co.uk/

email:

paulstate51 [dot] co [dot] uk

      tel: +171 729 4343

    state51, Rhoda Street, London, E2 7EF, UK


********************

UKNM is sponsored by Excite UK, visit us at http://www.excite.co.uk.

Email Khalil Ibrahimi

khalilexcitecorp [dot] com (mailto:khalilexcitecorp [dot] com)

 to advertise on Excite.

********************

Change your UKNM subscription use http://www.chinwag.com/uknm.html