Netherlands

Where’s my Canonical Data Model?

I think it was last year that Jan Baan spoke to the assembled Atos Origin Enterprise Architects. At one point he said “Data is still the king” which for some reason I found rather comforting. How does an architecture work if data isn’t managed properly?

In the brave new world of SOA the term canonical data model appears. Canonical is yet another word that’s been borrowed by the IT world and then used in an imprecise fashion. In mathematics the term has a precise and formal definition (see Wolfram’s Mathworld if you’re into that sort of thing) and something that is canonical is universally applicable – well, at least in the world of mathematics. Of course canonical data models aren’t, so I can’t buy one just yet. 

Concept

Let’s quickly (and imprecisely) review what a canonical data model is. The basic idea is that we need a means of passing data between our loosely-coupled applications. Of course these applications can store their data in different ways and to avoid the necessity of point-to-point conversions we introduce the canonical (common) data model. Now you can transform the data from source application 1 to canonical format and then transform from canonical format to the format required by target application 2.

This idea is not new. The concept first emerged into public prominence, under the name conceptual schema, with the publication of the ANSI / SPARC Database Model in 1977! The ANSI three-schema architecture was not extensively implemented at the time but currently the idea has re-emerged in the SOA context. By the way, there’s an interesting description of the concept of a canonical data model in Jack van Hoof’s blog which despite its title doesn’t address the semantic issues. 

Issues

So what’s my gripe? My first point is that a canonical data model obviously isn’t universal. The common data model of everything hasn’t been built yet and when it is it will probably turn out to be the 42 from the Hitchhiker’s Guide to the Galaxy. What then is the scope of the model? Is it a model of data objects for generic subject areas, for a specific company, or maybe industry-wide? Will my canonical data model look like your canonical data model and will it matter if it doesn’t? Typically the approach seems to be the creation of a model for your enterprise (on the inside looking out) where I guess “enterprise” would be business group for some large corporations. What also happens is that external standards are incorporated in the model (for example Dunn & Bradstreet numbers for customers/suppliers or UNSPSC for product classification). It’s not obvious to me what exactly to build into the model – it’s not a trivial problem. I also get the impression that many architects see the model building as a straightforward analytic exercise whereas some people in the data management world see data modelling as a creative design activity. How am I going to arrive at my canonical data model and how will I maintain it?

My second issue is that having a canonical data model doesn’t really address the semantic issues. For example what happens if you want to exchange information about customers between two systems? The canonical model maps data structures and not individual data values so how will you know that customer A in system 1 is customer 123 in system 2? It looks like we need some agreement about a common identifier (e.g. the Dunn & Bradstreet number) that will be in both systems, or we need to keep and maintain lists of all translations and that can be a hell of a job if you have millions of customers. The model also doesn’t tell us what a customer is. Is it a party who regularly orders goods from us, a party who once ordered something (some systems don’t record one-time customers), or a party with whom we have contact and may be able to sell to (e.g. a marketing database)? What are the underlying definitions for the entities and attributes in the canonical data model and how are they maintained? Achieving consensus about enterprise-wide data definitions is a major exercise and many organisations have struggled and failed with this problem. 

Question

I have a few ideas about how to address these issues and I’ll come back to that in a future blog, but first I’d like to know what other people think. Do all you architects recognise these problems or have you all solved them already?       

Footnote

 A break with the trend: a blog in English. Seems a shame to limit our audience when we’re on the World Wide Web!