James Governor’s MonkChips: digital lard for the enterprise: DRM meets document formats: “What I am saying is that DRM creates new escrow challenges, and organisations should know exactly what they are using it for, and why, and what risks they are mitigating, before embarking on an enterprise DRM strategy.”

Firstly only did Massachusetts make their decision in a very public way (so we can all “see their working”), clearly documenting why they felt Open Formats were important from a business perspective, and how they defined what one way. Now the resulting backlash from Microsoft is producing an awful lot of good analysis about the problems of file formats, which is very relevant for companies considering ELN purchases.

To me, this is increasingly looking like the end of proprietary file formats (and hence the use of file formats as a commercial weapon) and the more the vendors kick & scream, the faster their customers will understand they need demand the protection that Open Formats provide.

Groklaw has a really good wrap up of all the issues and considerations with pointers to other material. If you haven’t been watching this then it is a good first place to start your research into why Open File Formats are a crucial consideration in any ELN purchase.

Here’s a particularly relevant quote:

Few leading vendors ever really want to have an agreed-on open standard, because that opens the market to competitors. But once a technology stabilizes, wise customers demand standards that let them pick between suppliers, and customers are the ones paying the bills. This happens all the time, in all technology areas.

As a result of this, the community has had the opportunity of seeing Microsoft’s “Talking Points” and soundly rejecting them. We’ve also seen how desperate Microsoft is to protect their Office cash cow.

Why is this important for Electronic Lab Notebooks?

• It demonstrates (yet again) that Open File formats are an issues that needs to be taken seriously at the highest levels. Hopefully as governments around the world begin to insist on open formats, corporates will be being to take the problem seriously too.
• The arguments vendors use to justify their proprietary file formats are increasingly being shown to be false.
• The increasing experience with open formats shows that it isn’t that hard to build systems that are open, and give customers first-hand experience of the benefits of Open Systems.

Electronic Lab Notebooks systems benefit more than normal from open file formats because:

• Almost ELN systems are made up of software from more than one source (commercial vendors, as well as a large number of small, focused applications developed in-house).
• The information in ELN systems often needs to be accessible in 30 years or longer (”Lifetime of the Company” is very typical). The chances of the original application being available is minimal.
• ELN systems often need to do an awful lot of unusual stuff, which wasn’t anticipated by the original designers. Yet the market for ELNs is comparatively small compared to the overall market for products like Office. So ELN functionality is often an after-the-fact addon implemented internally or by niche vendors (like my company, Amphora).

My feeling is that the market is going to punish ELN vendors who are still holding to their closed proprietary file formats, and preventing their users getting access to their data (by not releasing APIs). I spoke to one vendor at a conference a few months ago and he was genuinely confused why anyone would need their data outside their ELN - and felt strongly that moving to an open file format would be unduly hard for them. Duh. All they’d have to do would be to publish their internal documentation and they’d be OK - no code changes required. But that would require giving up control….

In contrast, there are ELN suppliers who do strongly support open file formats, and encourage our customers to access their data which is stored in our systems, etc. Our PatentSafe product is one, but there are many others. Did I mention we’re growing like crazy….. I guess the market likes vendors who look after their customers’ interests

For more information you might want to readWhy OpenDocument Won (and Microsoft Office Open XML Didn’t) by David A. Wheeler. There’s also a very interesting letting from Adam Barr (of Microsoft) to Jeff Raikes (Group VP at Microsoft) explaining the reasons for Massachusetts’s choice, and why it shouldn’t be a problem for Microsoft to work with open formats rather than resisiting. Replace “Microsoft” with your favourite ELN vendor and see how it feels…

I hope it won’t be long before companies start to require “that all electronic documents “created and saved” by state employees would have to be based on open formats”. That will be a major advance for the interests of customers.

How long will it be before industry also wakes up to the problem? There’s no need for proprietary file formats, all it does is increase user lock-in to that specific vendor.

Hashes are an important part of any Digital Signature scheme, and SHA-1 is one of the more popular (and until now well-respected) Hash algorithms. Any flaw found in the hashing algorithm is a serious problem for any ELN system that uses Digital Signatures to prove a record has not been changed. As Jon Callas, PGP’s CTO, puts it: ‘It’s time to walk, but not run, to the fire exits. You don’t see smoke, but the fire alarms have gone off.’

So this isn’t a crisis - if you were using digital signatures yesterday, they haven’t suddenly become worthless today. But it is a reminder that nothing lasts forever, especially in Cryptography. Flaws are found in algorithms, computers get faster, and soon it costs a mere $1m to forge the Digital Signature on the document that proves you have the rights to a Blockbuster drug. Picking a “better” algorithm or longer key isn’t going to help - erosion of Cryptographic tools is a fact of life.

Any Evidence system needs to have a series of controls and capabilities to show that a document has been unaltered. Digital Signatures are one of the tools you would use in such a system (we use them in our PatentSafe product) but they shouldn’t be the only thing you rely on. Indeed PatentSafe has a whole series of checks & balances in it and if you run the system properly, even if it turned out the signature algorithm was worthless, you’d still be able to use your records in court.

Unfortunately there are some vendors who are relying on Cryptography alone to prove authenticity. Listening to their marketing, they’re treating the technology as a silver bullet - “Buy this magic, and all your concerns are over”. This has always made me intensely uncomfortable, especially given the timescales that our customers expect to be able to use the records they put into our systems. Fortunately given the level of concern over Electronic Records for patents, few customers have actually implemented such systems - the laggards have been proven correct!

It is possible to do perfectly safe, effective, Electronic Records systems for Patent Evidence Creation and Preservation. Unfortunately, it is a lot harder than just implementing some Cryptographic magic, as the demise of SHA-1 shows. My problem is that it is much easier (and more seductive) to talk about apparently sexy technology rather than the real (but less exciting) issues around good electronic records systems