Screen scraping to my mobile with Grease Monkey...idea #67/100...

In idea #62, I wanted to scrape info from a designed-for-big-screen web page and make it available for a tiny screen on a mobile, perhaps as a WAP page or a text message.

The example I gave was the "My Rental List" from the Blockbuster Online account page. It's just three tiny lines of info (my current DVD rentals), but buried in a mass of HTML in a frame set.

I described an approach, which is essentially to ensure that all data elements in a page have unique labels, so that elements could be fetched separately to some other container than the web page that holds them.

I've now found a way to do something similar, albeit only partially and with some limitations. It's called Grease Monkey (GM). This is a modification to the Firefox browser. The mod allows a programmer to "take charge" of the web page output and re-display it, within the data-manipulation possibilities of JavaScript.

For example, in the Blockbuster page in question, there is a table that contains the rental list that I'm after. Fortunately, those nice people who programmed Blockbuster's website have nice "id" tags in each of their tables. The one I want is called "Onrentlistcontrol1_OnRentList". I can therefore extract the table contents using GM and then I have the list, separated from all the Blockbuster page graphics and other stuff on the page.

However, at this point, it's only a partial solution. What I really need is a web(/WAP) proxy that runs a similar script, so that when I call the page from my mobile, I just get the table and nothing else. So, what I really need is a Grease Monkey proxy, which is my idea for an interim solution!

What I like about GM is the whole notion that the user can create his/her own service on the back of an existing one, just by re-purposing the content. The notion of giving the user control over their own service experience was the essence of idea #10, which so far has been one of my favourite (possibly the favourite) idea in the 100 ideas series.

Of course, I don't expect most users to start creating their own services. However, what I hope is the following might happen:

1. A better (i.e. open) environment emerges for innovative new services
2. A market emerges for service "plug ins" (output from 1)
3. Operators to have an "ah-ha" experience (OK, that's too hopeful)

Screen scraping has limited scope in a mobile environment. The "GM-like proxy" in a mobile network should go a stage further and allow manipulation of underlying service elements, such as voicemail, but still in a "proxy" fashion that doesn't stand a chance of actually changing (i.e. harming) the elements themselves, so other users get the unadulterated vanilla service.

With the emerging possibilities of all-IP networks in 3G (such as the IMS), this idea becomes even more possible and potentially very powerful. To an extent, the advent of IMS will be a step towards an open network that the operators no longer control, despite their best efforts. Many of the services hanging off the IMS will be under the control of 3rd parties. On the Northbound interface, the 3rd party might well have a web presentation of their service. With a GM-enabled browser, we can potentially take control of the service in ways unintended by the service provider.

(p.s. using Grease Monkey, I have appended some Technorati tags at the end of this post using a GM script to override the standard Blogger posting page.)

Buy my book (Amazon US/UK)
Join my email list
Subscribe to my "100 Mobile Product Ideas" free e-book

Tags: greasemonkey mobile IMS WAP