queso made mention of this,

[scottobear]

and I think brad could be looking at this, to keep them that want to be 'crawler free' free

The three good ones so far seem to be standards for robot exclusion,killing scrapers,spider info resource.

of course, this goes counter to what I think the web is used for, and if you publish to the net, folks will find you, sooner or later.

personally, if it's not a private entry or friend's only, I want to be indexed like crazy, and to be found by whoever wants to find me.

Comments

[ex-foom23660.livejournal.com]

Whore!

Re:

[scottobear.livejournal.com]

Nah, I'm a one woman man.

But yeah, if you want to read me, I'm easy. you don't even have to buy me dinner.

Re:

[ex-foom23660.livejournal.com]

I meant LJ whore! Hee hee!

Re:

[scottobear.livejournal.com]

That i am! :)

[rejectomorph]

I decided not to block search engines from my journal, since it was a search engine that brought me here in the first place. I did a Google search on a company that had sent me some bit of spam, and LJ users who had gotten the same thing wrote about it and ended up in my search results. Turned out to be the most helpful results, too. First time I was ever grateful for spam. I probably wouldn't have found LJ so soon without it.

Re:

[scottobear.livejournal.com]

Always nice to find the dark cloud's silver lining, hm?