One strange thing though, they didn’t adhere to my robots.txt file which tells them not to index odp.php – heck, they even threw robots.txt right in there with the rest of the search results.

Well, as long as I’m in the index I don’t care much

The new way is to do it here, from “My Sites”. Yes, that means you will have to have registed your site there first. You’ll also have to agree that your “site has violated Google’s quality guidelines” and that you’ve made changes. In my case I don’t know why I’ve been banned, but I just agreed to it anyway.

I guess when you have all your sites registered one place it makes it easier for Google to keep track on you .. so if you’ve got one banned site maybe the other looses some PR or something? (just speculating, or course).

I’m don’t how well this development model works with dynamic data etc. but I’m sure they have a clever way to do that too. Will report more later when I’ve tested it

One strange thing though – even though I have a robots.txt disallowing Google from indexing odp.php on my site they seem to have indexed it nonetheless.

Good the be back in – even though most people find this site using Yahoo! anyway – which has finally gotten the point and made a search portal that looks just like Google

I like the idea of web based word processors, like Writely. I have a laptop, a computer in my bedroom, and a computer at work – and I like the fact that I can just sit down and edit my documents without thinking about where they are physically located. Also, it’s ideal for people who want to work together – sharing documents, editing them – and be able to view the changes of others.

And best of all – you can save as Word, HTML or export to PDF! ..

.. but what I’d really like it a tool like this who could save as DocBook XML. I mean, it’s not hard – if you can export to HTML you can export to DocBook, so why aren’t these companies doing it? Maybe not enough requests for such a feature? Well, I for one would like to pay for such a service, so sign me up

PS! I know I promised you I’d let the java crowd retaliate for the crude comments about java forwarded by random PHP geeks – and I will. Tomorrow. Promise.

I worry that search engines are sucking out too much of the Web’s value, acting as leeches on companies that create the very source materials the search engines index.

He argues that people are using the search engines as ‘answer engines‘, and that they often do not visit the pages that create the content at all.

I see this myself. If I search for something, I often specify the search good enough so that I don’t have to visit the site. Especially when it comes to technology-related searches (e.g. config settings, etc.) – Google almost always display what I want in the text next to the title.

Why is this a problem? Well, the sites create the content, but don’t get the traffic. So, the search engines get the traffic (and the money that follows it) while the ones who actually create the content get nil.

And if the ones who create content want something, then they will have to outbid eachother on AdWords. I must say Jakob has a point.

Here’s the story: I’ve created a script, called phpODP that lets people put the content of dmoz (the open directory project) on their own pages. This is perfectly fine according to the dmoz licence, and my script is even listen on dmoz itself. A lot of people have downloaded this script, and are currently using it.

A little while ago I read Googleguy say something about Google wanting to do something about duplicate content. I think probably banning my script was one of the things he was talking about. They don’t want 100′s of dmoz out there, and frankly I can’t blame them (although I do think entire site banning is a bit to harsh). However, I can see a couple of positive things about including multiple dmoz sites in the index:

• The main site is often slow, so other sources may be preferable
• Sites can add content/functionality (e.g. user comments about the sites, rating, etc.)

Who is going to do that now? No one.

I see odp @ bie.no is still listed in Yahoo! – I don’t know if they’re heading in the same direction as Google, but I have a feeling they are.

The psychology of a Google Outlaw (me that is).. the first days after being banned Day 1: Something must be wrong. ‘I must’ve spelled “site:bie.no” incorrectly – is that how you search to tell if you’re included in the index? It’s always been, hasn’t it?’ etc.etc. Day 2: Realising one has been banned, searching for “google banned”, sending a complaint to Google Day 3: “I don’t need Google” – start using Yahoo! and actually learning it isn’t all that bad. It can be used as a replacement. Almost a full day. Day 4: Logging into Amazon Associates and checking the damage. Damn. Day 5 – next Google Dance: Apathy

Anyhow, today I put up a robots.txt that should help things:

User-agent: Googlebot Disallow: /products/phpodp/odp.php Disallow: /products/phpodp/go.php

I was delighted to find out that I could have User-agent in there, so I can make some special rules for Google. As you see, it shouldn’t access the odp script anymore.

Hopefully, this will get me unbanned – but it’s not for sure. I’ve got all those people using my script linking back to me from odp content as well. So Google might think “Duplicate content links to bie.no, bie.no must be bad”. I don’t know what I’m going to do with that if that’s a problem. But if it is a problem, then I’m not the only one in trouble (other scripts do the same, although maybe not with so much duplicate content).

Gaaah .. I’m tired after a long day of refurbising, so I’ll stop writing now