nsftools.com - The April 2005 Blog

Since Rich has been putting a lot of HHGTTG quotes out there in anticipation of the upcoming movie, I thought I might join in with a Douglas Adams quote of my own.

This one is actually from Dirk Gently's Holistic Detective Agency, but I like it because it has to do with programming. It's Richard (a software developer) talking to Reg (a "Regius Professor of Chronology") during dinner at Cambridge.

"Well, what we called a computer in 1977 was really a kind of electric abacus, but..."
"Oh, now, don't underestimate the abacus," said Reg. "In skilled hands it's a very sophisticated calculating device. Furthermore it requires no power, can be made with any materials you have to hand, and never goes bing in the middle of an important piece of work."
"So an electric one would be particularly pointless," said Richard.
"True enough," conceded Reg.
"There really wasn't a lot this machine could do that you couldn't do yourself in half the time with a lot less trouble," said Richard, "but it was, on the other hand, very good at being a slow and dim-witted pupil."
Reg looked at him quizzically.
"I had no idea they were in such short supply," he said. "I could hit a dozen of them with a bread roll from where I'm sitting."
"I'm sure. But look at it this way. What really is the point of trying to teach anything to anybody?"
This question seemed to provoke a murmur of sympathetic approval from up and down the table.
Richard continued. "What I mean is that if you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your own mind. And the more slow and dim-witted the pupil, the more you have to break things down into more and more simple ideas. And that's really the essence of programming. By the time you've sorted out a complicated idea into little steps that even a stupid machine can deal with, you've certainly learned something about it yourself. The teacher usually learns more than the pupil. Isn't that true?"
"It would be hard to learn much less than my pupils," came a low growl from somewhere on the table, "without undergoing a prefrontal lobotomy."
"So I used to spend my days struggling to write essays on this 16K machine that would have taken a couple of hours on a typewriter, but what was fascinating to me was the process of trying to explain to the machine what it was I wanted it to do. I virtually wrote my own word processor in BASIC. A simple search and replace routine would take about three hours."
"I forget, did you ever get any essays done at all?"
"Well, not as such. No actual essays, but the reasons why not were absolutely fascinating."

I can completely relate to that. Mr Adams, wherever you ended up out there... thanks for all the fish.

The other day I ran across links to two really interesting products, dealing with giving you more universal access to your data.

Orb Networks
Via Russell Beattie, I visited the Orb Networks site. Check this out:

Orb gives you secure access to the digital media on your home PC through a simple web interface. Orb streams your content safely to virtually any internet-connected device. Just install the Orb software on your PC, then access your media anywhere.

So, I can listen to MP3s from my home PC on my cellphone? Or stream videos to my Pocket PC? They even talk about streaming live TV, as long as you've got a tuner card. Sounds fun.

GMail Drive Shell Extension
Essentially, turn your GMail account into a mapped drive on Windows. Drag and drop files. Use it as offline storage.

These types of technologies (or maybe they're technology extensions) are really interesting to me. Take your existing hardware/software/setup, and do things with it that you never thought about doing before.

In this case, make your data more ubiquitous. Access your media files or your backed up data from anything that has a browser. Securely. Using free software.

Cool.

Nice to see all the Greasemonkey activity starting up! Jake Howlett published a nice script to add some features to LDD search result pages, and Rob McDonagh actually wrote one to allow you to see the comments of my blog entries inline (which has to be about the coolest script that anyone could write!).

I got the itch to touch my monkey just a little more (my LDDMonkey script that is), and I updated it so it not only puts a little [+] next to response links on messages, but it puts them next to any link that has "?OpenDocument" in it. So now it also puts a [+] next to message links in the main views and the search results.

Here's the updated script (same place as before):

http://www.nsftools.com/tools/lddresponse.user.js

Unfortunately, using "?OpenDocument" as my search criteria puts the [+] next to a few links that won't work properly -- like the forum FAQs or the login link. But it was just so nice to have all the messages in the main view and the search results view expandable that I thought it was worth having a few "false positives".

If you think that the new functionality is obnixious though, you can install the older version of the script instead (the one that only does response links) using this script:

http://www.nsftools.com/tools/lddresponseold.user.js

As always, bug reports and questions can be directed at the project location on the OpenNTF site.

I made a small update to the LDDMonkey script -- I fixed the problem where it was inserting a linefeed between all of the response links. If you installed the script on Saturday, please reinstall to get the latest version (you can install right over the existing script). Here's the local link:

http://www.nsftools.com/tools/lddresponse.user.js

After you have Greasemonkey installed, you can just right-click the link above and choose the "Install user script..." option.

If you're trying to figure out why LDDMonkey is a good thing to have, please read below.

What is Greasemonkey?
Greasemonkey is a Firefox extension that allows you to run one or more "user scripts" against a page after it loads in your browser. Essentially, this means that you can run client-side JavaScript against the page, and manipulate it in any way you want. See the Greasemonkey Script Repository for examples.

What is LDDMonkey?
LDDMonkey is a Greasemonkey script that runs when you're viewing messages on the Lotus developerWorks site. Currently, when you read a message in one of the LDD Forums, the responses to the message are listed at the bottom as separate links, like this:

While that's a nice way to display the links, it's not a very convenient way to read the links, because you have to visit each link individually to read the separate responses, and on a thread with a lot of responses you can end up going back and forth between pages quite a bit. Not only is it easy to lose context when you're doing this, but it's not a very efficient way to read the thread.

LDDMonkey will change the response list ever so slightly, so it looks like this:

Notice the little [+] next to each response link. If you click on one of the [+] thingies, it will display the text of that response below the link, like this:

You can open as many of the responses as you want this way. If you click the [+] again, the response text disappears, so you can toggle the display of the individual responses. And you never have to reload the page, because I'm using that XmlHttp/Ajax stuff to get the responses.

So there you go. Try it and see if you like it. If you don't, it's easy enough to uninstall.
;-)

I just published a Greasemonkey script that makes it a lot easier to read message threads on the LDD forums:

http://www.openntf.org/projects/pmt.nsf/ProjectLookup/LDDMonkey

Look at the "Project releases" links on the main OpenNTF LDDMonkey project page, click the latest release, and follow the install instructions on that page.

What this script does is add a little [+] next to each response link at the bottom of a message that you're looking at. If you click the [+] next to a response, the text of the response will appear below the link, so you can read it inline instead of having to open a new window.

This will work for all the responses to a thread, so you can read all the responses to a message inline, on a single page.

If you find any bugs or have any suggestions for improvement, please use the OpenNTF project discussion forums.

Enjoy.

A few days ago, Rob McDonagh asked me about accessing unread docs in a user's Notes mailbox. I know you can just set an agent to run against "All unread documents in a view" and then use NotesDatabase.UnprocessedDocuments to do such a thing, but that only helps if you (A) are using the Notes client UI, and (B) currently have the database open to the view in question.

Well, since I didn't see a good built-in way of accessing the unread docs, I started going down the C-API path. All things are possible with the C-API, right? Here are the fruits of my labor:

LotusScript Class To Get Unread Docs In A Database Or View

I hope that helps someone out there.

This one's for Spuggy (via CNN Money, of all places):

Watch Dukes, earn $100,000
The job may seem onerous to some, however: watching "The Dukes of Hazzard" reruns five nights a week.
Viacom's Country Music Television channel is running help wanted ads for this position -- Vice President, CMT Dukes of Hazzard Institute. That's right. That's what the ads say, noting the pay is $100,000 for the duration of a one-year contract.
The vice president will have to watch Dukes of Hazzard every weeknight on Country Music Television, know all the words to The Dukes of Hazzard theme song and write the Dukes of Hazzard on-line blog for CMT.com, Country Music Television's Web site.
The person selected will also have to be available for media interviews to "share his or her expertise and passion" for the TV program and make appearances at events such as Dukefest 2005 in Bristol, Tenn., in June, according to the ad.
A valid driver's license is required, the advertisement says, because the applicant may be required to drive the General Lee, an orange 1969 Dodge Charger driven by the fictional Bo and Luke Duke in the show, which originally ran from 1979 to 1985.

Yeehaw...

I've been trying to figure out a good way to convert PDF files to images. I know that sounds kind of backwards, since the whole world seems to want to convert their files to PDF format, not away from it. But I have two reasons for wanting to do such a thing:

create a thumbnail image of the first page of a PDF
convert an entire PDF to TIFF as an alternate archive format

So I started looking around for tools that would do the conversions for me -- preferably something I could either access programatically (Java or DLL), or run as a batch job. I found a few tools that did a few things, but most of the ones that were really useful were kind of expensive. Converting something to PDF isn't that bad, but if you want to convert away I think there are just too many exceptions.

Then I happened to look at Ghostscript. The only reason I didn't look at it first is because I was under the impression that it only converted things to PDF. I mean, that's what everyone seems to use it for. Well it turns out that Ghostscript also allows you to convert all or part of a PDF to PNG, JPEG, TIFF, BMP, PCX, PNM, and PSD formats too (here's the list).

And it's a piece of cake to batch it. For example, if you want to convert the first page of a PDF to a JPEG file at 150 dpi, you just run something like:

gswin32c -dSAFER -dBATCH -dNOPAUSE -sDEVICE=jpeg -r150 
-dTextAlphaBits=4 -dGraphicsAlphaBits=4 -dFirstPage=1 -dLastPage=1 
-sOutputFile=page1.jpg test.pdf

(That's all supposed to be one line, so remove the linefeeds if you copy and paste that somewhere.) Want to convert an entire PDF to a multi-page TIFF file? Try:

gswin32c -dSAFER -dBATCH -dNOPAUSE -sDEVICE=tiffg4 -r150 
-dTextAlphaBits=4 -dGraphicsAlphaBits=4 -dMaxStripSize=8192 
-sOutputFile=test.tif test.pdf

(Again, without the linefeeds.) Need to run it on your server? There are Windows and Linux/Unix versions. Heck, it even runs on OS/2. Got no budget? No problem: it's free. In fact, you can use the GPL version for certain kinds of commercial distribution if you're so moved.

I actually started playing around with accessing the DLL programatically, but finally decided that it was more productive to stab myself with a fork, so I ended up with a LotusScript agent that batches up and runs all my jobs for me at the command line. Works like a champ.

BTW, if you want to get the first page of the PDF as an image and then resize it as a thumbnail as I was discussing before, you'll need to use something else for the JPEG resizing once you've got the image. If you want an easy Java solution, please see my Java JpegImage class. I've also got an example of calling that from LotusScript in my LS2J Examples Database.

Two weekends ago, I was browsing the books in the Border's in front of my house, and I saw a nice leather-bound (or leather-looking-bound) edition of The Ultimate Hitchhiker's Guide by Douglas Adams. It was a really nice looking book, and it was only $16.99 (which seemed like a really good price). I looked it over, flipped through the pages, and eventually put it back and moved on.

Well, maybe because The Hitchhiker's Guide Movie is coming out later this month, or maybe just because I thought it was a good deal, I kept thinking that I should go back and buy the book. After all, I don't have all the Hitchhiker's books anymore, and it would be fun to re-read them.

So I went back last weekend, and it was gone. "No problem," I thought, "I'll just order it off Amazon." I did a quick search, and sure enough there it was, but there was something strange... that edition of the book isn't supposed to be released until November 1, 2005!

That was odd. I knew I had seen and touched the book just days before, but even after I checked other online merchants, they all said it wasn't going to exist for 7 more months. It was perhaps a fitting paradox for that particular book (although probably even more fitting for Dirk Gently), but nonetheless troublesome in my little universe. I wanted the book. I saw it, and now I wanted it, dammit.

Well, I went back to Border's today (after checking a few other bookstores) and it hadn't reappeared, but as I was leaving I happened to look at the shelves behind the checkout counter -- usually reserved for books that people have ordered or put on hold. And there were 3 copies of the book. My book.

I asked checkout counter lady if those books were on hold for someone, and she said they weren't. They were just back there for some reason. "Um, can I buy one?" She gave me a strange little look and said of course I could buy one. And I did.

Well it turns out that this particular edition was actually published by Portland House in 1997, but it looks almost identical to the soon-to-be-released edition from Gramercy Press you can pre-order on Amazon. Which is fine with me. I don't know why my bookstore happens to have several copies of this old edition of the book (which I can't even find a reference to on Amazon), and I'm not asking any questions.

So now I've got The Hitchhiker's Guide To The Galaxy, The Restaurant At The End Of The Universe, Life, The Universe And Everything, So Long And Thanks For All The Fish, Young Zaphod Plays It Safe, and Mostly Harmless, all bound up into a nice little package. That should keep me busy for a few nights.

Yesterday I read an article saying that the US Congress is considering extending daylight saving time (that's "saving" without an "s"), so it will go from March to November. Supposedly this will save 10,000 barrels of oil per day during the extended months.

On one hand, my knee-jerk reaction is: "Heck, go ahead and extend it from January until December while you're at it. If you're thinking about extending it, just get rid of it instead." I think that would make a lot of people (like me) happy.

However, Dave Delay made a good point today, saying that extending daylight saving time would actually break a large number of computer applications -- including operating systems.

Wow, that's a really good point. No matter how silly you think daylight saving time is, you have to remember that computer programmers have been writing exceptions to deal with that silliness for a long, long time. It's like when Joel Spolsky talks about rusty code -- there's a lot of extra crap that we programmer types have had to write to deal with it, but it's now a part of the program, and it's important.

Of course, I can imagine that some software companies would in fact like it if the daylight saving time rules changed and their customers suddenly had to rush out and purchase new versions of software. What better opportunity to tell your customers, "Oh, sorry, you'll have to spend loads of money on those upgrades now... I really hate it for you, but we're just not patching those old versions any more, and the US Congress made us change it."

Yeah, we'll have to see what side of this issue the software lobbyists are on.

When I was reading some of the stories about WebFountain during/after yesterday's blog entry, I noticed this quote:

WebFountain has found that 30% of the web is porn...

My, my, aren't we a naughty bunch. But you know, that's not the first time I've heard that sort of thing. Take, for instance, this Reuters article from a few months ago:

As goes pornography, so goes technology. The concept may seem odd, but history has proven the adult entertainment industry to be one of the key drivers of any new technology in home entertainment. Pornography customers have been some of the first to buy home video machines, DVD players and subscribe to high-speed Internet.
One of the next big issues in which pornographers could play a deciding role is the future of high-definition DVDs.

Heck, some people have even claimed that the porn industry is one of the big reasons why VHS beat the pants off of Betamax in the early video format wars (the author of that particular essay used the name "Peter Johnson", which may or may not indicate a bias to the hypothesis, but he at least used footnotes...).

And there's money in them hills, too. A USA Today article from last year said that the online porn industry generated $2 billion in revenue in 2003. Another recent Reuters article said "Mobile phone users around the world spent $400 million on pornographic pictures and video in 2004, an amount that is expected to rise to $5 billion by 2010". Heck, Playboy even launched their "iBod" service for the iPod this past January, seeking out the mobile masses (maybe they were appealing to the mythical toothing crowd).

Anyway, there's no explosive point or deep meaning to all this, I'm just offering a little distraction from the normal tech-talk. Consider this some conversation fodder for the water-cooler today.

(By the way, all the links in this blog entry should be "safe", in case you're reading this at work.)

About two years ago, I started noticing some traffic in the logs of my website that looked like this:

"GET /tools/secdom.txt HTTP/1.0" 200 5254 "-" 
"http://www.almaden.ibm.com/cs/crawler   [c01]"

Curious, I went to the almaden.ibm.com link and saw a prepared statement about the "Almaden Crawler" (I won't repeat it to you -- I'm sure you're clever enough to click the link and read it yourself).

For some reason I was thinking about that today, and realized that I never did hear anything about the ol' Almaden Crawler since then. So I looked it up.

Apparently it's now called WebFountain, and there have been a few mentions of it in the press (although the last article on the list was published exactly a year ago). Some of the researchers on the project even wrote an interesting paper about the "bursty" nature of blogs.

So I guess what I'm wondering is: where is this thing? Will we ever see it? Is it one of those cool projects that will never "escape" from the IBM research labs? Certainly there's money to be made with search engines (based on the fortunes of the Google creators), but maybe it's just not money the way that IBM makes money. I don't know.

In any case, there are a lot of really smart people working in IBM research, so I don't think the problem is that they couldn't figure out how to write a search engine. I'm guessing it's a marketing dilemma.

Kind of like the VitalFile Backup Application on the alphaWorks site: I've played around with it and it's a really cool app that "just works" (it automatically backs up local files to a file server when they're created or they change). But in speaking with one of the project managers for VitalFile a few times, it sounds like they're just having a hard time marketing it.

That's the sad truth for us programmers, I suppose. You can write really cool software, but if you don't have good marketing then it'll never go anywhere.

UPDATE: John Battelle visited the WebFountain project last March and wrote a really interesting piece about it. Lots of good stuff there.

After I finished reading the Baroque Cycle (see previous entry), I took a little break by reading a collection of short stories by Philip K. Dick.

If you don't recognize the name, Philip K. Dick was a fairly prolific science fiction author who died in the early 1980's. The movies Blade Runner, Total Recall, Minority Report, and Paycheck were all based on his stories. He was a bit of an odd guy in the biographies, but he certainly knew how to put a lot of interesting ideas to paper.

Anyway, at the end of the book there was a reprint of the introduction he wrote for a 1968 collection of his short stories called "The Preserving Machine". In it, he talks about the difference between a short story and a novel. I thought it was interesting, and since I know that several of my readers are either (A) aspiring writers or (B) science fiction buffs, you might find it interesting too. Here ya go:

The difference between a short story and a novel comes to this: a short story may deal with murder; a novel deals with the murderer, and his actions stem from a psyche which, if the writer knows his craft, he has previously presented. The difference, therefore, between a novel and a short story is not length; for example, William Styron's The Long March is now published as a "short novel" whereas originally in Discovery it was published as a "long story". This means if you read it in Discovery you are reading a story, but if you pick up the paperback you are reading a novel. So much for that.
There is one restriction in a novel not found in short stories: the requirement that the protagonist be liked enough or familiar enough to the reader so that, whatever the protagonist does, the reader would also do, under the same circumstances... or, in the case of escapist fiction, would like to do. In a story it is not neccesary to create such a reader identification character because (one) there is not enough room for such background material in a short story and (two) since the emphasis is on the deed, not the doer, it really does not matter -- within reasonable limits, of course -- who in a story commits the murder. In a story, you learn about the characters from what they do; in a novel it is the other way around: you have your characters and then they do something idiosyncratic, emanating from their unique natures. So it can be said that events in a novel are unique -- not found in other writings; but the same events occur over and over again in stories, until, at last, a sort of code language is built up between the reader and the author. I am not sure that this is bad by any means.
Further, a novel -- in particular the sf novel -- creates an entire world, with countless petty details -- petty, perhaps, to the characters in the novel, but vital for the reader to know, since out of these manifold details his comprehension of the entire fictional world is obtained. In a story, on the other hand, you are in a future world when soap operas come at you from every wall in the room... as Ray Bradbury once described. That one fact alone catapults the story out of mainstream fiction and into sf.
It is in sf stories that sf action occurs; it is in sf novels that worlds occur. The stories in this collection are a series of events. Crisis is the key to story-writing, a sort of brinkmanship in which the author mires his characters in happenings so sticky as to seem impossible of solution. and then he gets them out... usually. He can get them out; that's what matters. But in a novel the actions are so deeply rooted in the personality of the main character that to extricate him the author would have to go back and rewrite his character. This need not happen in a story, especially a short one (such long, long stories as Thomas Mann's Death in Venice are, like the Styron piece, really short novels). The implication of all this makes clear why some sf writers can write stories but not novels, or novels but not stories. It is because anything can happen in a story; the author merely tailors his character to the event. So, in terms of actions and events, the story is far less restrictive to the author than is a novel. As a writer builds up a novel-length piece it slowly begins to imprison him, to take away his freedom; his own characters are taking over and doing what they want to do -- not what he would like them to do. This is on one hand the strength of the novel and on the other, its weakness.

About a month ago, I finally finished The Baroque Cycle trilogy, by Neal Stephenson. I can't remember when I actually started reading the books. I see that I talked about Quicksilver (book #1 of 3) in May 2003, so it's certainly been over a year. All 3 books are over 800 pages long, so I read at least 2,400 hardback pages to get through the story (which is about as much as Duffbert reads in a weekend).

I've got to say, as much as I was intrigued by the books, I probably wouldn't have finished them all if I didn't love Neal Stephenson so much. There were just so many... words. And for me to say that, well, that's saying something (see Addendum below).

But I still love the guy as a writer, and I'm still going to buy and read The Cobweb and Interface when they come out next month.

If you've never read any of his books, you can start by downloading (or buying) In The Beginning... Was The Command Line, an essay about operating systems, the Microsoft monopoly, and why Linux is so cool. Then go buy or borrow yourself a copy of Cryptonomicon and lock yourself in a room with it.

Not sure if I have a real point to all this. I'm just proud of myself for finishing the books, I suppose.

Addendum: to give you a feel for what I was talking about when I said there were a lot of words, here's an excerpt. It's Daniel Waterhouse (one of the main characters) riding in a coach with Isaac Newton (yes, that Isaac Newton) in the early 1700's, trying to eat a meal:

In an apt demonstration of the principle of Relativity, as propounded by Galileo, the bawdy platter, and the steaming morsels thereon, remained in the same position vis-a-vis Daniel, and hence were, in principle, just as edible, as if he had been seated before, and the pies had been resting upon, a table that was stationary with respect to the fixed stars. This was true, despite the fact that the carriage containing Daniel, Isaac Newton, and the pies was banging around London. Daniel guessed that they were swinging round the northern limb of St. Paul's Churchyard, but he had no real way of telling; he had closed the window-shutters, for the reason that their journey to Bedlam would take them directly across the maw of Grub Street, and he did not want to read about today's adventure in all tomorrow's papers.
Isaac, though better equipped than Daniel or any other man alive to understand Relativity, shewed no interest in his pie -- as if being in a state of movement with respect to the planet Earth rendered it somehow Not a Pie. But as far as Daniel was concerned, a pie in a moving frame of reference was no less a pie than one that was sitting still: position and velocity, to him, might be perfectly interesting physical properties, but they had no bearing on, no relationship to those properties that were essential to pie-ness. All that mattered to Daniel were relationships between his, Daniel's, physical state and that of the pie. If Daniel and Pie were close together both in position and velocity, then pie-eating became a practical, and tempting, possibility. If Pie were far asunder from Daniel or moving at a large relative velocity -- e.g., being hurled at his face -- then its pie-ness was somehow impaired, at least from the Daniel frame of reference. For the time being, however, these were purely Scholastic hypotheticals. Pie was on his lap and very much a pie, no matter what Isaac might think of it.
Mr. Cat had lent them silver table-settings, and Daniel, as he spoke, had tucked a napkin into his shirt-collar -- a flag of surrender, and an unconditional capitulation to the attractions of Pie. Rather than laying down arms, he now picked them up -- knife and fork. Isaac's question froze him just as he poised these above the flaky top-crust. "Is it the Clubb's intention to remain idle for the entire month of July?"
"Each member pursues whatever lines of inquiry strike him as most promising," Daniel returned. "As you and I are doing at this very moment." And he stabbed Pie.

Now, you could argue that those first two [rather long] paragraphs about the relativity of Daniel's Pie could easily have been shredded and tossed by the editor. However, reading them does indeed make your mind start churning, especially if you're the science geek sort like me.

Are they extraneous? If you think so, then you'd never make it through the trilogy. If you give yourself up to the style of writing, and try to appreciate that sort of verbal cleverness, then you'll enjoy the ride.

In any case, Neal Stephenson is certainly on my top 5 list of people I'd really like to have dinner with.