Paul O'Flaherty

Stop AboutUs.org from scraping your blog

Tom’s posted some good tips for preventing AboutUs.org from scraping your site:

Block their robot
Add the following lines to your robots.txt (in the “root” folder of your website)
User-agent: AboutUsBot
Disallow: /

Block their IP range
In your .htaccess file (if you’re on Apache) add the following lines:

deny from 66.249.16.

Block the bot’s user agent
If you do user agent blocking, block the bot’s user agent:
(currently Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot))

Block the DomainTools.com IP Range
AboutUs.Org uses Domaintools services to generate thumbnail images of site content, so block their IP range too:

deny from 66.249.4.

Update: Tom appears to have deleted the post from his site. (It’s back) Thankfully AboutUs.org has a page stating how to block their bot from your site and it’s a bit simpler than what’s posted above.

To prevent the AboutUsBot from collecting your site content in the future, please include the following lines in your /robots.txt file.

User-agent: AboutUsBot

Disallow: /

The AboutUsBot will include the following in it’s User-Agent string:

Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)

Please note that the current AboutUsBot behavior is to visit each site only once to initialize the AboutUs.org page.

Also, they’ve posted a page for people to voice their concerns! So, go voice them…

Ray King from AboutUs.org just doesn’t get it!

Following up on what I wrote about the AboutUs.org uproar yesterday, I noticed that Ray King (owner of aboutus.org) left a rather lengthy reply to on Ajay D’Souza’s blog.

Here are some excerpts (any emphasis is mine):

Whois data is readily and publicly available from many sources, as is historical whois data, which often pre-dates a domain being switched to “private”. I am not aware of any whois data service that first asks the registrant for permission to provide it — not that this wouldn’t be good, but it seems impractical. We list contact information because it is valuable to users wanting to contact website owners. The information can be changed simply by hitting the edit button.

AboutUs is a wiki, and as such, its content is openly editable. Some site owners may feel the desire to check AboutUs to see if anyone else has modified the page referring to their site or commented positively or negatively — just as they might want to know if someone made a blog or del.icio.us entry about their site. Watching for commentary on one’s own site is par for the course if you publish a website in the first place. To make this easier however, it is our plan to add the ability to get an e-mail alert for any pages users put on their watch list. This will be coming as soon as we are able.

As you may remember, I fired off an email to Ray yesterday, to which I haven’t had a response, although I can imagine he’s getting plenty of emails at the moment about this.

Now, lets get right to the point! Ray, you just don’t get it! Yes we understand that you are trying to provide a service here by having a central place where everybody can go and find out information about websites and their owners. We also understand that the information is readily available from Whois services.

The major problem here is that it’s a wiki that ANYBODY can edit.

You have no system to ensure the accuracy of any information posted on the site. Heck, the algorithms you are currently using to extract site names aren’t even working properly.

You also intend to put the burden of maintaining accurate information on regular site owners by requiring them to come and check the accuracy of information posted on your site.

Not only that, but if they want to ease that burden they will have to register for email updates, which of course requires the handing over of their email address. Then there’s the fact that you have no privacy policy in place, so they have know way of knowing what will be done with that email.

It’s one thing for bloggers and site owners to watch technorati and other such sites to see who’s writing about them, but your site is meant to be a service, and considering that you’re partnered with Name Intelligence one can only assume that you intend to monetize this service.

It’s very easy for you to say that you’re providing a service and dismiss the fact that information can simply be edited by anybody and inaccurate by saying, hey, it’s a wiki.

However, I don’t think it’s that simple. If you really wanted to provided a service, one that would be valuable to everybody and that could be trusted, you would not have a wiki. You would have a searchable database with moderated submissions and a commenting system, also moderated. Although funnily enough that sounds almost exactly like the whois services that are out there, except without the comments.

If you did employ moderators you’d be talking about a tremendous undertaking. I know, I used to run Blog Resource, (check here for the latest info on the future of Blog Resource) and that was just a small directory of blogs, but because submissions were free and didn’t require registration (much like AboutUs.org because they don’t require an email to register) the task of moderating and ensuring the accuracy of submissions became too much for us, and we had to kill off the directory. We had 3 people moderating it.

You see, having a wiki for this provides Ray with a number of things:

• The ability to say that errors are not the fault of aboutus.org
• Any abuse of the system, or defamation of information about peoples sites is not the problem of aboutus.org. It’s the site owners responsibility for not checking in everyday to make sure the information is correct.
• It’s an easy way to stick advertising (currently Google ads) on millions of pages and have other people do the work of updating the sites content.

Maybe I’m overreacting here, but I don’t think I am.

Stop writing AboutUs.org

There’s a lot of buzz today about AboutUs.org and whether or not their practice of posting the whois information of peoples sites to their wiki is ethical.

From AboutUs.org homepage:

AboutUs is a fully editable wiki, a type of Web site, that has been prepopulated with information about several million websites. Enter a domain name in the search box (for example: “Yahoo.com” or “AboutUs.org“) and see what comes up!

There are many Web sites which are not yet in the system, so feel free to use the “add missing domain” box on the left hand side to add them. Currently, we only support addresses in the form “second-level-domain.tld” (i.e. no third-level domains/subdomains yet).

Now, my problems are two fold:

First you’ve got the questionable issue of copyright, as they scrape your site for content, much like a search engine, but their use of the information is to populate a wiki, which can then be edited by anyone!

Due to fact that anybody can edit posts it is now incumbent of me to regularly visit AboutUs.org and verify that my information is correct

Second, and perhaps more importantly is the fact that their is currently NO way to delete your site from the listing.

So you are forced to register in order to edit or delete information. Okay, registration is just a username and password, but you’ll also need to enter your email address if you want to retrieve your password at a later date.

As for the privacy policy of AboutUs.org, well, it’s nonexistent! It merely pleads with us not to be nasty (in best English):

Please Do not abuse with the System

So, what do you do to prevent a listing. well currently you can’t but it didn’t stop me firing off a brief letter to the site owner who can be contacted here (scroll to the very bottom for the email address).

Here’s the email I sent him. It wasn’t well thought out but it got my point across I hope (I’ve deleted the list of domains form the email!)

Hi Ray,

While I appreciate your efforts to develop a service at AboutUs.org, I feel that the posting of information about me and content from my sites is questionable in terms of copyright and makes me feel uncomfortable.

I would request that you remove all information on AboutUs.org that refers to the any of the following domains and/or their sub-domains:

(Domain list deleted)

While some of these domains may not currently have information listed, I would like you to ensure that all of the above domains are deleted form AboutUs.org and remain unlisted.

I am aware that AboutUs.org is a wiki, however, I do not believe that it is my responsibility to visit the site and edit information that I DID NOT post or give permission to be posted. Nor, should I be required to register in order to do so.

If I choose to list my information, I will email you, register, and post it myself. Until such time, please keep it off of AboutUs.org

In the meantime, I wish you the best with your endeavour.

Paul O’Flaherty

So, now we’ll just wait for a response from Ray.

What do you folks think about this?

AIM PRO – 98 seconds of WTF??

I received one of those AIM Member Messages in my email this morning:

Get Secure Instant Messaging with AIM Pro — Free Download.

A quick glimpse of the email promised such wonders as

• Integration with Microsoft Outlook – Get all IMs, calendar events, and send email in one convenient place.
• Business Grade Security through encrypted communications
• One-Click WebEx meetings

Always being one to give something new a bash I followed the link, downloaded the 12.3mb (AIMPro208.exe) file and happily started the installer.

Once complete I started the app. It asked for my screen name and password, opened up my browser to AOL.com and then nothing.

I was expecting this:

Instead I got noting.. Just a blank white box on the side of my screen that sat there like a useless lump.

I force the application to quit. I restarted the machine. Tried again with the same result.

I uninstalled, restarted, installed, restarted and still the same bloody result.

It looks to me like AIM PRO, ain’t so Pro!!!

Woah! Okay, as I type this AIM PRO has decided to open up and actually work! I’m going to shut it down and re-open it and time how long it takes.

1 minute and 38 seconds! That’s right 98 seconds for an IM client to open! Unbelievable!

No wonder I thought it had crashed when I saw this for more than a minute:

Skype has in my experience always been the slowest of the IM clients to open, but on this machine even Skype only takes 5 seconds from click to working.

Windows Live Messenger takes 4 seconds.

GAIM only takes 4 seconds and that’s including logging into AIM, Yahoo, MSN, ICQ and 2 G-talk accounts.

Google Talk takes an amazing 1 second!

I would give times on YIM, ICQ and regular AIM, but I don’t have them installed as I use GAIM as my main client.

And as for all the integration promised in Outlook etc.. well, the calendar integration does appear to work, but some of the menu options in the program for integrating with Office and Explorer are completely grayed out.

Needless to say, I wouldn’t be booting AIM PRO to chat to my friends and make calendar alterations when using regular AIM client (or GAIM) is way faster even with booting Outlook. Oh, and Outlook only takes 1 second to boot!

Easy Blog platforms turn users away from blogging.

I spent two hours earlier tonight chatting to my cousin James on Skype. James is what you would call the average computer user.

He’s used computers for years but every time he’s had an issue with his machine he would call me. Since I’ve been living in Denmark that hasn’t been possible.

When we were chatting we got talking about blogs, and the fact that a month ago I got him to set up a blog on WordPress.com.

James hasn’t posted to his blog much, mostly because he was in Prague for the past while on holiday, and one thing that struck me during our conversation tonight was the difficulty faced by the average user.

After returning from his holiday, Jim wanted to post some of his pictures online. He also wanted them to appear on the side of his blog. Ever geek out there probably had the same idea, use Flickr and a widget.

That’s exactly the idea I had, and told Jim about it. The first problem Jim had was finding the Sidebar Widgets menu. The net was figuring out how to configure the widget. The instructions given are far from clear.

We got him set up after some step by step instruction for me, but it was still a lot of hassle for what I, and I’m sure all other geeks, consider a rather trivial thing.

Blogging, in many respects is accessible to just about anybody, with services like Blogger and WordPress.com providing fire and forget services which require minimal configuration. The problem lies when you have an average user, like Jim, who wants more from his blog.

Any user that is even slightly serious about blogging will want to start down the road of customization, custom templates etc. But todays blog platforms offer only two real choice. Have things insultingly easy to use with almost zero customization options, or have customization be an arduous task involving the editing of PHP, HTML and other web languages.

It’s the degree of separation between the apparent ease of blogging and the requirements for customization that turns many potential long term bloggers away from blogging.

Platforms like WordPress (self hosted) and WordPress.com are making inroads with tools like the sidebar widgets, which do make editing the side bar easier. But it’s not enough.

What we need is an interface where the user can customize everything, via drag and drop, replace interface text with a click, and select colors from a pallet without worrying about hex or RGB values.

Okay, for those who want customized graphics, they still have to bang them out themselves, but lets make inserting them as easy as possible.

Trackbacks, comments, gravatars, should all be just a checkbox away and drag and drop to position.

If we can get a platform that is powerful and fully featured with an easy to use interface that can do all of the above, then I think we will see a lot more bloggers who stick at it beyond the first couple of posts.

Oh, before I go, just one more thing, I made a little pre-announcement over on OFlaherty.dk tonight. You might wan to check it out if you’re into Tech!

Writer has the write potential but…

Okay, pardon the crap pun in the title, but I’ve been playing with Windows Live Writer and I must say that unlike Ed Bott (who positively adores it) I’ve got mixed feelings about this Microsoft effort.

I know this is beta software, so I’ll try not to be unduly harsh, but I find myself agreeing with Pirillo that Live Writer is a long way from winning me over.

Perhaps the biggest problem I have with Live Writer is when using it with WordPress. The style importing simply doesn’t work with any level on consistency. I have a number of blogs, two of which have identical style sheets, yet the import works with one and not the other. Also, having installed Live Writer on both my work and home machine, I was surprised that Writer successfully imported the style of my blog on my work machine, but not on my home.

When Writer detects the style it makes a temporary entry on your blog which it is supposed to delete, unfortunately it failed to do the deleting on my blogs.

I’ve noticed issues with the interface as well. After adding multiple accounts to the program the appear erratically in the drop down menu to the right of the screen. Sometimes they all appear. Sometimes they don’t. Strange!

On a happier note, I do feel that this offering from the big MS has the potential to become a “Killer App” of blogging! It’s off to a relatively good start for a beta, and with it’s ability to be extended by user developed plugins there’s little that it couldn’t be expanded to accomplish.

I have one major question though! Since when did beta software start at version 1.0?

The wasted voice of Vista!

I just read an entry over on AMCP about Microsoft shipping voice recognition in Vista. I posted a comment about this but Co-comment doesn’t support pop-up comments on Blogger, so I’ve quoted it here:

Voice recognition in Vista could prove to be a thorn in the side of Vista if it doesn’t function correctly.

My experience of current voice recognition programs is that they tend to be both resource hogs and not accurate enough to use for practical application (except perhaps for small blog entries that I don’t mind correcting manually!).

Of course there are benefits to having this technology embedded directly into the OS (should lower the resource demands), and it will definitely be of great benefit (and a cash saver) for the disabled and those that simply must have voice recognition.

But, and there’s always a but, if the Vista version of voice recognition proves to be more inaccurate than the current commercial versions, then it will be seen as a waste of resources, as well as a waste of programming resources that could have been used to improve other areas of vista.