Archive for the 'mac' Category

Solving the evolving File Format problem

Tuesday, August 14th, 2007

Andy Matuschak brought up a really good obstacle to providing an easier upgrade and downgrade path for applications: changing file formats.

Changing file-formats is far from free. It fractures your user base as your users find they cannot share their documents with their friends. It makes it difficult for you to regression test your code. And it dissuades third parties from supporting your file-format, because you’re giving them more work each time you change. Overall changing file-formats should be avoided where possible. How then can you extend your application?

Future proof your file-formats!

I try to keep my file-format forwards compatible: the same file should work with older versions of my software. This makes it easier to find bugs that I may inadvertently have added: I can regression test against an older version of the software.

I use a file-format that lets the Application store but not use the fields it does not understand. There are many such solutions: SQLITE database, Trees, XML, python pickles… The reason to store fields one does not understand is that one can save them back out later.

For example consider an HTML document. Even if your HTML editor does not know what the “blink” tag means, it can remember the fact that that there is a “blink” tag around the text over there. Even if you edit the text within the blink tag, it can make reasonable guesses about grouping. Of course if you delete the text and type it back in, the blink tag will be gone. But most of the time you don’t just lose your formatting because a colleague used a different tool to edit your file. The key idea here is that you can use an extensible hierarchical data format to keep things associated with their attributes.

Keep the conversion code separate!

Sometimes, to simplify the main application’s code, it does makes sense to change file-formats. In that case, write a bidirectional converter. A version field in the file-format specifies (by naming convention) the converter to use so that the App when encountering a file of a different file-format can figure out the chain of conversions it must apply to get something it understands.

Now if you downgrade, you’re in almost the same position as your friend who never upgraded: your application can’t read the new format files. There is however a difference: as part of the upgrade process a bidirectional converter was added. It does not need to be removed on a downgrade. Because the the older version of the application knows how to figure out which converters to call, it will just find them and call them.

Note that because the original file-format is extensible, it’s possible for the bi-directional converter not to throw away any data. Similarly because the oldest version of the app stores the stuff it does not understand and saves it back out again, changes made by a co-worker with an older version of the software should cause little data loss.

Additional benefits of independent converters

Because you have independent converters you can earn bonus points with people who have not yet been able to upgrade. You can give or sell them the tool to convert the files. The tool can even advertise the features they’d get by upgrading.

You benefit because your code is simpler. The application code is simpler because it only knows its native file-format, the converter’s code is simpler because it’s focussed on one task. You also benefit because it’s easier to require a new base OS for your next version without fracturing your user-base.

Of course writing writing a converter is a pain, and a bidirectional one even more of a pain, but as I said in the beginning changing file formats should be avoided. Writing a bidirectional converter forces you to consider more cases, leading to a better upgrade converter. Furthermore you can reduce the pain for third parties by giving or licensing your converter to them.

What about standard file formats?

I think the same idea of independent converters can be applied, although it may not be possible to keep the additional information known to later versions of the file-format in earlier versions of the file format.

IPhone apps: they weren’t kidding

Friday, June 22nd, 2007

The new WebKit inspector may be the demonstration vehicle that convinced Apple that AJAX UIs could work, despite many people’s concerns.

Very few applications so far use HTML for rendering. Find It! Keep It!’s Database view is rendered as HTML and uses Javascript within the application. NetNewsWire’s Combined View is also implemented in this way. WebKit Inspector goes one step further, by emulating what looks to be a Cocoa app in Javascript extremely effectively. Like Find It! Keep It!, it calls native functions to perform operations that would otherwise be impossible with Javascript. However it also uses the canvas element to make its bar graphs. Its only flaw is that it does not scale correctly when you press Command + to make the text bigger.

Google Gears looks promising

Thursday, May 31st, 2007

Google has just released a new API called Google Gears. It will allow web applications to work even if you’re disconnected from the internet for a period of time. And it will let them install their files onto your harddrive so that they can be run “locally”, by using a small “local server” that’s loaded into your application as an Internet Plugin (much likef Flash).

It’s New BSD licensed which means the source code is available and people will be able to make sure it doesn’t do anything dubious. Currently it’s Firefox only, but there’s source code for Safari, which when it’s released should theoretically also work with Find It! Keep It!.

It consists of three services: a database extension based on sqlite, a local server that saves and loads the application files from disk and provides an HTML interface to them, and full blown multithreaded Javascript interpreter. The javascript interpreter allows “worker threads” to run without blocking the browser. Basically they seem to run in a separate application that includes a Javascript interpreter taken from FireFox. The advantage is that the worker thread code can run on Linux, Windows and Mac OS X without needing browser specific tweaks. The UI will still run inside the browser.

Google Gears is providing a solution to a real problem without requiring a whole new programming paradigm as Adobe’s Apollo and Microsoft’s Silverlight do. It maintains the AJAX promise (runs everywhere) while providing a benefit should you choose to install it (faster and more resilient applications). Also, it’s open source which means other application providers can use it. I think in this case Google have a winning solution.

Update Dojo Offline now supports Google Gears.

Update2 It seems that the WebKit Google Gears only works with the WebKit nightlies (think Leopard) and will require an Input Manager… ho hum. (via Daring Fireball).

Neat Mac tricks

Sunday, January 21st, 2007

Every Mac comes with tons of hidden features. The problem is that they’re hidden, and the Mac doesn’t come with much of a manual. Here are a few that I’ve stumbled upon:

Argh… Beta 3

Sunday, December 3rd, 2006

The first bug I got back from Beta 2 showed that the new Crash Reporter functionality wasn’t always started up correctly… despite my testing for that case. Turns out that what I thought was memory given to me by Python was in fact garbage collected (and therefore … disappears after an indeterminate amount of time). Since the Crash Reporter is the means for users to signal bugs, I’ve had to rush the third beta.

Second Beta released & Input Manager disabler

Sunday, December 3rd, 2006

I released the second beta of Find It! Keep It! today.

It contains a few bug fixes, and a crash reporter to help people report bugs.

The biggest change is that Input Managers are disabled within Find It! Keep It! by default. They’ll still work in your other applications.

Why? All but one of the reported crashes at launch were due to Input Manager conflicts: 50% of all reported bugs! Understandably, if a crash is your first experience with Find It! Keep It!, you’ll assume Find It! Keep It! is at fault.

Contacting the author of every Input Manager that fails to ask them for a fix isn’t ideal: my users need to tell me what crashed and work with me to isolate the fault, and the Input Manager author needs to be willing to fix the problem. In the real world, most users will just give up.

Furthermore, in many cases, Input Managers are targeted at a specific application. For instance, Safari Tidy targets Safari, Chax targets iChat. At best they are harmless when loaded into other applications. At worst they’ll crash them.

So what if you actually want to access an Input Manager’s functionality from within Find It! Keep It!? I provide a Preference Panel that allows you to enable any of your Input Managers. When loading a new combination of Input Managers, Find It! Keep It! will step you through the process to help isolate crashes on load. It also disables any Input Manager it recognizes as having crashed on load, and restarts without it.

Hopefully, this is the best of all worlds: the ability for users to customize their systems without the penalty of instability.

I would like to see Apple providing stronger system controls over Input Managers, and other customizations: the user should always be in control. In particular Input Managers make adware trivial to implement on the Mac. Apple could improve overall user experience by requiring users to explicitly go to a preference panel to allow a given Input Manager to load into a given application.

Sunday’s Beta feedback

Monday, November 27th, 2006

The beta has been out two days.

Spam Filters

Again it seems some spam filters are preventing people from receiving the download information… No one has been turned away from this beta, so if you requested it, please tell your spam filter that mail from is not spam. If you email me again, I’ll be happy to send you the information a second time.


  • Input Managers, aka Plugins: Two crashes on launch were due to third party plugins
  • Cocoa: One bug seems to be that Cocoa gives me the wrong information (weird!)
  • Flash plugins disabled: I believe this is due to a misconfigured computer, but I need to dig into it further.
  • Rosetta: One report of Rosetta crashing on an Intel Mac, and thereafter Find It! Keep It! would not start again. To deal with this case I made a small program for Intel Mac owners that restarts Rosetta before starting Find It! Keep It!

Overall observations

Because people who are willing to run betas are people who try new things, they run plugins I’ve never even heard of, and have interestingly configured computers. Beta testing is trial by fire for the software being tested! :-)

People downloading the tool use a wide spread of browsers: 60% use Safari, 21% use Firefox (1.5 & 2.0), followed by OmniWeb, Camino, Opera and something called iGetter

A few people seem to be hoping that it will work on 10.3.x… I’m afraid it won’t.

Find It! Keep It! is Feature-Complete

Thursday, November 23rd, 2006

At 1 am today (Thanksgiving) Find It! Keep It! became feature-complete. It also has no known bugs.

I’m currently running it through my test plan to check that no bugs crept back in while I was sleeping…

This afternoon after the traditional meal with friends, I hope to send it out tonight. At the worst… it should be tomorrow.

Intel Mac, encryption and beta delay

Thursday, November 16th, 2006

I finally received my Intel Mac, and was able to test Find It! Keep It! on an Intel Mac. It works fine under Rosetta, which is very encouraging. A previous test failed badly. Apple’s fixes in 10.4.8 must have done the trick…

The beta was ready to go out on Monday, but before letting it out I was given the advice to add some form of protection. Apparently piracy of Mac shareware is a thriving industry: Spending less that 10 minutes searching, I found licenses to every piece of shareware I know!

Wil Shipley doesn’t believe it’s a big deal, as pirates would not buy in the first place, but could be the software’s loudest advocates. To some extent, I agree with him.

On the other hand, Colin Messitt makes a very convincing case that unprotected software does not sell well. Having spent a year and my savings working on this tool I cannot afford to lose 4/5ths of my customers.

So… I’ve been adding encryption and registration code to Find It! Keep It!. Many simple solutions violate people’s privacy or are annoying. For instance, I could tie each license to a single machine. But then, people could not upgrade easily. But doing it right takes time… Once this hurdle is over, I’ll release Find It! Keep It!.

First Pre-review feedback and banks…

Tuesday, November 7th, 2006

I’ve been getting my first feedback, and it’s really great: people use software in such amazingly different ways. Two bugs I never encountered in a year of usage have been fixed.

The PowerPC beta looks to be delayed a few days to fix any other issues that crop up, and to allow me to catch up from the lost days due to the recent storm and talking to my bank: see below.

On the other hand, it looks like the Intel beta will have to wait until next week: somehow my company credit card application got lost going from my bank’s branch manager’s desk to my bank’s headquarters… losing me at least a week. Anyway, the Intel Mac is now ordered and is supposed to arrive on Friday.