queso made mention of this,
Jul. 22nd, 2001 02:58 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
scottobearand I think brad could be looking at this, to keep them that want to be 'crawler free' free
The three good ones so far seem to be standards for robot exclusion,killing scrapers,spider info resource.
of course, this goes counter to what I think the web is used for, and if you publish to the net, folks will find you, sooner or later.
personally, if it's not a private entry or friend's only, I want to be indexed like crazy, and to be found by whoever wants to find me.
The three good ones so far seem to be standards for robot exclusion,killing scrapers,spider info resource.
of course, this goes counter to what I think the web is used for, and if you publish to the net, folks will find you, sooner or later.
personally, if it's not a private entry or friend's only, I want to be indexed like crazy, and to be found by whoever wants to find me.
