Made my day

This article just made my day. Fantastic!

SOA Design Patterns released

I've been writing a book on SOA and ESB for a while with a couple of friends. It's due to be part of Thomas Erl's SOA series later this year. While writing it Thomas asked us to produce some patterns for his book on SOA. Well the finished book, SOA Design Patterns, is now out and it's well worth a read. A number of people throughout the industry have helped to contribute some of the patterns within it so this is definitely a cross-industry collaborative effort. Thomas also wants it to be a live-work in as much as new patterns can be contributed by anyone. So take a look at the book and if you see something missing consider contributing it.

Star Trek interlude

Anyone who's known me for long enough knows that I'm a Star Trek fan. (Not the variety that dresses up as Andorians or learns Klingon!) Being born in the late sixties sci-fi was fairly influential to me in one way or another, with my fondest TV memories coming from shows like Star Trek, Space 1999, Dr Who, The Six Million Dollar Man and Blakes 7.

I still say that the first couple of seasons of The Original Series were better than much of what was created in recent years. So it's with some apprehension that I look ahead to the new movie. I'm definitely not going to prejudge, but I had hoped they would retain some of the look-and-feel of the original.

On thing that has stayed with me over the past few years is a quote from Kirk to Picard from Star Trek Generations (watch the film if you want to understand the context):

"Don't let them transfer you... don't let anything take you off the bridge of that ship... Because while you're there, you can make a difference."

I've found that it applies to more than just starship captains!

Virtual conference

We're doing a virtual conference next month. I'm recording some sessions at the moment. Very strange, but I think it'll be fun to attend. Maybe this will encourage me to look into SecondLife.

SOA is dead (again?)

For the umpteenth time SOA is dead apparently. It's been shot so many times over the past year or so that you'd think it had a role in a Sam Peckinpah movie! However, maybe it's got 9 lives or you need to use a silver bullet, because each time it dies it keeps coming back.

This time it's my friend Anne who is reporting the demise of SOA. I've known Anne for many years and respect her, so unlike a few other obituaries I take note of this one. However, like Duane I have to disagree that SOA is dead. OK, the term may have been overloaded over the past few years almost to the point of being meaningless (analysts are as much at fault as anyone in the industry) but the concepts behind it are still very much alive and relevant (more so in today's economy.) Where we have lost our way is perhaps in having an agreed architectural definition for SOA: it is the 'A' after all, so surely we can agree on what that means (REST doesn't suffer this problem) as well as pushing the mantra that "SOA is not just a technology" (though given the number of people who have been saying this over the years I suspect that this is more a case of individuals simply not listening, certain vendors muddying the waters or it being lost in the noise.)

There is no global panacea for IT woes. There never has been and I doubt there ever will be one. For the most part software engineering practices evolve over time (Darwin would be proud). There are a few evolutionary dead ends on the way. I doubt we'll see any extinction-level events though (not unless they're associated with human extinction.) But these things take time (I'm sure the dinosaurs laughed at the evolutionary path that created the Coelacanth, but who's laughing now?)

We can't afford to keep jumping from one lifeboat to another just because some use cases don't match, or someone didn't quite understand that you need to think about how to use SOA before coding. The underlying requirements of loose coupling, security, federation, reliability etc. date back decades. The term SOA should have been good enough. It still is as far as I'm concerned. If it's not SOA then are we in for yet another round of acronym generation, like SOA 2.0, WOA etc? None of which really add much to help the people at the sharp end of IT problems!

I think in Anne's recent SOA obituary she is trying to indicate that the principles are fine but it's the term that needs to go. In the first case we can agree. In the second I don't think it's worth the industry wasting another 4 years to disagree on yet another acronym (if we do, I'd like to nominate XYZZY.) As I said above, for people actually using these principles and associated technologies it won't make a difference to them. (It may be good for analysts though.) So let's just stop this madness and start concentrating on where it really matters.

In this case I think it's definitely more a case of what Mark Twain once said, "The reports of my death have been greatly exaggerated".

Scala

My friend Mic and I have discussed a few languages I've been playing with over the past year or so, which include D and Erlang. I think I managed to persuade Mic that there is some benefit to Erlang, but maybe the jury's still out on D. Well a few weeks back Mic mentioned that he was looking at Scala, so I decided to take a look. I didn't find the time until I took a break from work for the festive season and have just spent a few days giving it a go. My first impressions are pretty positive (I've mentioned a few times in the past that I'm a fan of functional programming languages and particularly lisp which I've used for over 20 years). I'm trying to use it through Eclipse but am finding the plugin a bit clunky, so I may go back to the command-line and emacs (I'm a shell/emacs person at heart!)

So far I haven't got into any really complicated projects or applications so there may be problems with the language that I don't know about yet. But at the moment I like it and will persevere with some pet projects that I've been thinking about for a while. Even if it doesn't lead to anything ultimately, it's another language to learn and experience gained. But I have a suspicion it'll be more than a passing fad for me. Thanks Mic.

Eventual Consistency and XTP

I meant to post something about this article a while ago but only found the time due to the Christmas vacation.

It's a nice article and what Werner's talking about is part of what I consider real extreme transaction processing (not the hype term). As I've said elsewhere several times before, a few of us in the transaction community have been working on this type of problem dating back to at least the work we did in early SOA/Web Services standards. Even longer if you consider the work on extended transaction models. It's nice to see practical implementations of some of the theory behind these models actually being used outside of academia.

I'm hoping that in the next year we'll see some fruit coming from research and development we've been doing on real XTP, including the REST approach I've mentioned before as well as general approaches to large scale (extended) transactions. This should further illustrate what's possible with relaxation of the ACID properties we know and love, particularly when combined with high performance messaging and replication techniques. Sometimes it's hard to believe that this has been going on for so long.

Since this article started with a reference to Werner, it's worth ending with him too: congratulations my friend for your latest award. Well done! I think I could have predicated it ;-)

Sad day

I just read that Oliver Postgate has died. It's sad because I grew up in an age when Bagpuss and the Clangers were the high points of television. Those were some of my fondest memories.

SOA Symposium follow up

I seem to have forgotten to mention that I presented a couple of times at this year's first ever SOA Symposium a couple of months ago.

The Eternal Golden Braid

I first read An Eternal Golden Braid about 14 years ago and thought it was a great book. I still do. If you haven't read it then I would encourage you to do so: it is a wonderful journey through mathematics, computer science, philosophy, art and so much more. (I notice that there's a 20th anniversary edition, so that may be going on my present list to Santa since my original copy is a bit dog-eared.) So it was with some surprise that I noticed a translation of Godel's original work available! However, the reviews indicate that it may not be a good translation. But it may be worth a try anyway!

EDA followup

JJ commented on my earlier posting. Unfortunately he doesn't seem to have comments enabled, or I'd have responded there. (JJ, if you do have comments on that entry, let me know and I'll cross link).

Anyway, he talks about "the main failure of distributed computing is precisely the prevalent monotheism of traditional distributed computing approaches". Ignoring the fact that I don't believe distributed computing has failed, I think JJ read more into my posting than was intended. I didn't mean to suggest that any approach to distributed computing (e.g., RPC or message based) should be used to the exclusion of all others. I think I made that clear elsewhere too: you should use the best approach to hand. After all, isn't that what being a good engineer or architect is all about? I'm very fond of saying that one-size doesn't fit all, and by that I mean what I've just said. Of course systems exist that are entirely based on the concept of resources, services, events, RPC or message exchange, as examples. But there are examples of implementations that combine these different approaches very successfully too.

So in a way I disagree when JJ says "It is not being about fence sitting, or applying the right solution to a given problem." At this stage in computer science that is precisely what it is all about. However, I know what he's after and I've said elsewhere that in many ways computer science is a soft science and more of an art at the moment.

So looking forward, would I like to see a formal approach that combines all of the different approaches I've discussed to distributed computing, and which JJ mentions? Yes, I would. But would everyone use it in its entirety? No, of course they wouldn't: they would use the right aspect of it for the task at hand. Does that invalidate it? No. Does it offer an improvement over today? Yes. Is it going to happen soon? I doubt it, and I'm fairly sure JJ knows the reasons for that too (and the majority of them aren't technical). In fact I could probably write an entire article on that very subject because it relates to how close to a hard science computing science is and can come, with analogies between Galileo/religion and vendor values. But it's way too dark a subject to cover today.

DOA 2008 is over

I just finished chairing the last session for DOA 2008. It's been a good conference with some excellent presentations and invited speakers. Now for the long trip home.

Thoughts on EDA and CEP

Over the past year or so I've been seeing more and more discussions on Event Driven Architecture (EDA). I suppose the thing that really made me take notice was when Gartner tried to push SOA 2.0 and mentioned EDA in there. For me, event driven architectures (notice the case) have been around for as long as I can recall. I've used the concept of events and triggers in many systems I've been involved with without thinking it was somehow novel. The reason for this? Well as I say every time I'm asked about event driven architectures (still notice the case), I believe that they are all around us and have been since the very first computer system: they are a useful abstraction for building certain types of systems (not necessarily just distributed systems).

It's possible to model everything in terms of events when you think about it. For example, in a distributed object system that is based on RPC and uses client and server stubs to make the distribution opaque, the ultimate server object (the one that does the real work) is essentially registered with the server stub to be called (triggered) by it upon the receipt of the right message (i.e., the right opcode in the recipient message). Likewise, the client application (or thread of control) can be considered to register itself with the client stub to be triggered (woken, since we're working in a synchronous environment in this example) when the response message comes back. Pub/sub based messaging systems, such as JMS, support the concept of events as well. When you subscribe to a blog you're essentially registering to be triggered when an event (a new blog entry) becomes available. Anyone remember the CORBA Event Service or the DCE Event Management Service?

Back when I was starting my PhD one of the first things I did (for fun, as it really had nothing to do with any of the various subjects I was considering for my PhD at the time) was to write a hardware bus simulator in C++. It had the concept of components being connected to the various buses and registering to be triggered when specific events occurred. In the real world, hardware is event based, with IRQs as the events and the bus as the delivery mechanism. A very good example of an asynchronous system that is often ignored by the software community these days. Maybe we should be looking at calling the next wave Interrupt Driven Architectures instead?

I've written applications on a number of windowing systems over the past 20 years. Probably the best one was InterViews, but they all share a very common pattern: components (e.g., windows, scroll bars, minimisers, etc.) register for events (e.g., left mouse click, mouse move, etc.)

Over the past 5 years or so we've also seen a move away from the synchronous request/response approach to one-way messages with various Web Services standards and SOAP implementations. Services register their interest in specific events (receipt messages) and clients register their interest in specific events (response messages). But it's still very similar to the stub approach I mentioned earlier. But what we're seeing in this approach is making the concept of events and triggers more explicit than it was before (at least in the Web Services world). This is definitely a good thing for loose coupling when you look at asynchronous environments. But is is a good thing for developers and users? Maybe.

So why have we not seen more event based frameworks pushing events as first class entities up to the developer, which seems to be what the current discussions around EDA are about? I think the reason for this is abstractions: we typically try to hide things like events because most of the time we don't think in terms of them. When a sound wave hits my ear drum and is converted into electrical impulses that trigger a response in me, I don't normally equate the sound wave or the electrical impulses as events. When I receive a letter through my post box telling me I've won the lottery (I wish!) I don't think about the steps it took to be delivered as a series of events (though they clearly are). If you look back at what I said earlier concerning how event driven architectures have been with us for many years, you'll see that what's really happened is that we've tended to hide events with abstractions like stubs and message exchanges.

I've said before that I think an event driven approach to SOA (or pretty much anything) is good if it helps you to better reason about the system in question. If it gets in the way, though, there are alternatives that could be better. As with anything, you use the right abstraction for the job. Is this going to be like RPC, where we realise that hiding distribution is a bad thing (we knew all along really)? Is hiding events bad? Well as I just said, that will depend upon what you're doing. But just because you start thinking in terms of events doesn't solve all of your problems immediately. There may be benefit to exposing some business events to the higher level, but we shouldn't think that rewriting everything we currently use in an event-explicit manner will be a global panacea - it may even be a step backwards. As an industry we need to stop pushing point solutions as global solutions and remember that one-size doesn't fit all.

And what about Complex Event Processing (CEP)? In a nutshell CEP gives you the ability to correlate events from different streams over time (time should really be modeled as an event too) to determine patterns, manage and predict other events (such as failure conditions) etc. The data streams can contain messages at various levels of a system architecture, so it's possible to use CEP at the low level (e.g., allowing a system administrator to predict when machines a more likely to fail given daily usage patterns) as well as the higher level (e.g., allowing a business analyst to determine the buying habits of specific individuals given the market trends elsewhere in the world).

So if you've got CEP do you have an EDA? Well you obviously have something that is generating the event (data) streams that are input to the CEP implementation, but that doesn't mean the entire system is (or should be) architected using an event abstraction.

If you've got an EDA do you need CEP? This is more clear-cut: of course you don't need CEP if you've got events. There are more examples of event driven architectures that don't use (and don't need to use) CEP than those that do. But in some cases, it can make things easier to use CEP.

DOA 2008

I wasn't going to be able to come to DOA 2008, which was a shame because I enjoyed DOA 2007. However, neither of my co-chairs could attend either, so it fell to me to come. It's in Monterrey, Mexico this year and the trip here wasn't something I would like to repeat very often. But I'm here now and looking forward the conference. It'll also give me a chance to catchup on some work that I haven't been able to do (because of work) and blog entries that have been filling up my to-do queue. So one way or another the next few days should be interesting.

MAX sounds like it'll be fun

I was originally booked to presented at Adobe MAX as well as QCon SF (since both are back-to-back it was very convenient). However, I had to pull out. It's a shame because both events sound like they'll be a lot of fun and I wish I could attend.

New Macs

I've been a Mac user for a few years (keep promising myself that I'll return to Linux one day), so I was looking forward to the announcement of new Macs. But I obviously missed something because I'm not really impressed with what I've seen. No button on the track-pad? No 1900x1200 resolution? Slightly improved performance. But gone is the metal screen surround, to be replaced by black plastic?! I'm not sure what I expected from Apple, but it wasn't this. Plus, maybe the thing will look better when viewed up close rather than through a web page. If the new design does get around the erosion problem then it'll be worth an upgrade eventually.

Transactive Memory

I've been doing some research on Software Transactional Memory recently and as a result started to read about Transactive Memory. Now it quickly became apparent that the two things are unrelated, but I found transactive memory to be very interesting (not to indicate that STM isn't interesting!) Put simply, transactive memory is the name for the process whereby people selectively remember things and rely on their relationships with other people, or other things, to remember everything else. So for example, you don't remember every single phone number for everyone you know: you're selective (because that's how the brain works) and remember the "top" 6 or 7, but will probably rely on a telephone book or a PDA for the others. In the home, you may not remember how all of your friends are related to one another, maybe relying on your partner to do that. If you are a technophobe then you may rely on someone in your family to program the DVD player so you don't have to remember how to do it. If you've got kids then you may rely on your partner to do the bulk of their welfare. And the list goes on.

So what has this to do with STM? Well as I said at the start, absolutely nothing. But it does have relevancy to something else I've been interested in for the past few years: repositories. I've always believed that systems such as repositories are better implemented in a federated manner: they scale much better. This means that although you may have a logical notion of a repository, each physical instance is responsible for some different aspect(s) of the whole "mind". This is important because, for example, how you store and index service binaries is different to how you would store and index service contracts, or workflow definitions etc. In this kind of set up if you ask one repository instance "do you know about X" it should say either "no, but I know a man who does" or "no, but hold on and I'll get the answer for you". I'd never been able to put a name to the approach (beyond federation), but it seems to me that this is precisely transitive memory from a software system. DNS works in the same way.

CSP book available for free

I'll never know how I missed this announcement from 4 years ago, but the original CSP book is one of the most dog-eared in my library. If you don't have a copy, then I encourage you to download this and read it: it's as applicable today as it was 20 years ago (if not more so!)

To Twitter or not to Twitter?

A number of friends are "on" Twitter and apparently I should be too. I'm not quite sure why though. If blogging is anything to go by, it'll probably take me a couple of years to get round to it.

Nice article on REST

I haven't blogged much recently because I really haven't had the time. The same is true at the moment, since I'm sitting in yet another airport lounge, this time coming back from the first SOA Symposium (hopefully there'll be a blog entry or two from me on that subject later). But I wanted to say that I really enjoyed the latest article from Jim, Savas and Ian. It's been a few years since Jim and Savas worked for me and blogs/articles are often the only regular way we keep in touch. Nice work guys!