I work for Red Hat, where I lead JBoss technical direction and research/development. Prior to this I was SOA Technical Development Manager and Director of Standards. I was Chief Architect and co-founder at Arjuna Technologies, an HP spin-off (where I was a Distinguished Engineer). I've been working in the area of reliable distributed systems since the mid-80's. My PhD was on fault-tolerant distributed systems, replication and transactions. I'm also a Professor at Newcastle University and Lyon.
Friday, November 28, 2008
The Eternal Golden Braid
I first read An Eternal Golden Braid about 14 years ago and thought it was a great book. I still do. If you haven't read it then I would encourage you to do so: it is a wonderful journey through mathematics, computer science, philosophy, art and so much more. (I notice that there's a 20th anniversary edition, so that may be going on my present list to Santa since my original copy is a bit dog-eared.) So it was with some surprise that I noticed a translation of Godel's original work available! However, the reviews indicate that it may not be a good translation. But it may be worth a try anyway!
Sunday, November 16, 2008
EDA followup
JJ commented on my earlier posting. Unfortunately he doesn't seem to have comments enabled, or I'd have responded there. (JJ, if you do have comments on that entry, let me know and I'll cross link).
Anyway, he talks about "the main failure of distributed computing is precisely the prevalent monotheism of traditional distributed computing approaches". Ignoring the fact that I don't believe distributed computing has failed, I think JJ read more into my posting than was intended. I didn't mean to suggest that any approach to distributed computing (e.g., RPC or message based) should be used to the exclusion of all others. I think I made that clear elsewhere too: you should use the best approach to hand. After all, isn't that what being a good engineer or architect is all about? I'm very fond of saying that one-size doesn't fit all, and by that I mean what I've just said. Of course systems exist that are entirely based on the concept of resources, services, events, RPC or message exchange, as examples. But there are examples of implementations that combine these different approaches very successfully too.
So in a way I disagree when JJ says "It is not being about fence sitting, or applying the right solution to a given problem." At this stage in computer science that is precisely what it is all about. However, I know what he's after and I've said elsewhere that in many ways computer science is a soft science and more of an art at the moment.
So looking forward, would I like to see a formal approach that combines all of the different approaches I've discussed to distributed computing, and which JJ mentions? Yes, I would. But would everyone use it in its entirety? No, of course they wouldn't: they would use the right aspect of it for the task at hand. Does that invalidate it? No. Does it offer an improvement over today? Yes. Is it going to happen soon? I doubt it, and I'm fairly sure JJ knows the reasons for that too (and the majority of them aren't technical). In fact I could probably write an entire article on that very subject because it relates to how close to a hard science computing science is and can come, with analogies between Galileo/religion and vendor values. But it's way too dark a subject to cover today.
Anyway, he talks about "the main failure of distributed computing is precisely the prevalent monotheism of traditional distributed computing approaches". Ignoring the fact that I don't believe distributed computing has failed, I think JJ read more into my posting than was intended. I didn't mean to suggest that any approach to distributed computing (e.g., RPC or message based) should be used to the exclusion of all others. I think I made that clear elsewhere too: you should use the best approach to hand. After all, isn't that what being a good engineer or architect is all about? I'm very fond of saying that one-size doesn't fit all, and by that I mean what I've just said. Of course systems exist that are entirely based on the concept of resources, services, events, RPC or message exchange, as examples. But there are examples of implementations that combine these different approaches very successfully too.
So in a way I disagree when JJ says "It is not being about fence sitting, or applying the right solution to a given problem." At this stage in computer science that is precisely what it is all about. However, I know what he's after and I've said elsewhere that in many ways computer science is a soft science and more of an art at the moment.
So looking forward, would I like to see a formal approach that combines all of the different approaches I've discussed to distributed computing, and which JJ mentions? Yes, I would. But would everyone use it in its entirety? No, of course they wouldn't: they would use the right aspect of it for the task at hand. Does that invalidate it? No. Does it offer an improvement over today? Yes. Is it going to happen soon? I doubt it, and I'm fairly sure JJ knows the reasons for that too (and the majority of them aren't technical). In fact I could probably write an entire article on that very subject because it relates to how close to a hard science computing science is and can come, with analogies between Galileo/religion and vendor values. But it's way too dark a subject to cover today.
Tuesday, November 11, 2008
DOA 2008 is over
I just finished chairing the last session for DOA 2008. It's been a good conference with some excellent presentations and invited speakers. Now for the long trip home.
Sunday, November 09, 2008
Thoughts on EDA and CEP
Over the past year or so I've been seeing more and more discussions on Event Driven Architecture (EDA). I suppose the thing that really made me take notice was when Gartner tried to push SOA 2.0 and mentioned EDA in there. For me, event driven architectures (notice the case) have been around for as long as I can recall. I've used the concept of events and triggers in many systems I've been involved with without thinking it was somehow novel. The reason for this? Well as I say every time I'm asked about event driven architectures (still notice the case), I believe that they are all around us and have been since the very first computer system: they are a useful abstraction for building certain types of systems (not necessarily just distributed systems).
It's possible to model everything in terms of events when you think about it. For example, in a distributed object system that is based on RPC and uses client and server stubs to make the distribution opaque, the ultimate server object (the one that does the real work) is essentially registered with the server stub to be called (triggered) by it upon the receipt of the right message (i.e., the right opcode in the recipient message). Likewise, the client application (or thread of control) can be considered to register itself with the client stub to be triggered (woken, since we're working in a synchronous environment in this example) when the response message comes back. Pub/sub based messaging systems, such as JMS, support the concept of events as well. When you subscribe to a blog you're essentially registering to be triggered when an event (a new blog entry) becomes available. Anyone remember the CORBA Event Service or the DCE Event Management Service?
Back when I was starting my PhD one of the first things I did (for fun, as it really had nothing to do with any of the various subjects I was considering for my PhD at the time) was to write a hardware bus simulator in C++. It had the concept of components being connected to the various buses and registering to be triggered when specific events occurred. In the real world, hardware is event based, with IRQs as the events and the bus as the delivery mechanism. A very good example of an asynchronous system that is often ignored by the software community these days. Maybe we should be looking at calling the next wave Interrupt Driven Architectures instead?
I've written applications on a number of windowing systems over the past 20 years. Probably the best one was InterViews, but they all share a very common pattern: components (e.g., windows, scroll bars, minimisers, etc.) register for events (e.g., left mouse click, mouse move, etc.)
Over the past 5 years or so we've also seen a move away from the synchronous request/response approach to one-way messages with various Web Services standards and SOAP implementations. Services register their interest in specific events (receipt messages) and clients register their interest in specific events (response messages). But it's still very similar to the stub approach I mentioned earlier. But what we're seeing in this approach is making the concept of events and triggers more explicit than it was before (at least in the Web Services world). This is definitely a good thing for loose coupling when you look at asynchronous environments. But is is a good thing for developers and users? Maybe.
So why have we not seen more event based frameworks pushing events as first class entities up to the developer, which seems to be what the current discussions around EDA are about? I think the reason for this is abstractions: we typically try to hide things like events because most of the time we don't think in terms of them. When a sound wave hits my ear drum and is converted into electrical impulses that trigger a response in me, I don't normally equate the sound wave or the electrical impulses as events. When I receive a letter through my post box telling me I've won the lottery (I wish!) I don't think about the steps it took to be delivered as a series of events (though they clearly are). If you look back at what I said earlier concerning how event driven architectures have been with us for many years, you'll see that what's really happened is that we've tended to hide events with abstractions like stubs and message exchanges.
I've said before that I think an event driven approach to SOA (or pretty much anything) is good if it helps you to better reason about the system in question. If it gets in the way, though, there are alternatives that could be better. As with anything, you use the right abstraction for the job. Is this going to be like RPC, where we realise that hiding distribution is a bad thing (we knew all along really)? Is hiding events bad? Well as I just said, that will depend upon what you're doing. But just because you start thinking in terms of events doesn't solve all of your problems immediately. There may be benefit to exposing some business events to the higher level, but we shouldn't think that rewriting everything we currently use in an event-explicit manner will be a global panacea - it may even be a step backwards. As an industry we need to stop pushing point solutions as global solutions and remember that one-size doesn't fit all.
And what about Complex Event Processing (CEP)? In a nutshell CEP gives you the ability to correlate events from different streams over time (time should really be modeled as an event too) to determine patterns, manage and predict other events (such as failure conditions) etc. The data streams can contain messages at various levels of a system architecture, so it's possible to use CEP at the low level (e.g., allowing a system administrator to predict when machines a more likely to fail given daily usage patterns) as well as the higher level (e.g., allowing a business analyst to determine the buying habits of specific individuals given the market trends elsewhere in the world).
So if you've got CEP do you have an EDA? Well you obviously have something that is generating the event (data) streams that are input to the CEP implementation, but that doesn't mean the entire system is (or should be) architected using an event abstraction.
If you've got an EDA do you need CEP? This is more clear-cut: of course you don't need CEP if you've got events. There are more examples of event driven architectures that don't use (and don't need to use) CEP than those that do. But in some cases, it can make things easier to use CEP.
It's possible to model everything in terms of events when you think about it. For example, in a distributed object system that is based on RPC and uses client and server stubs to make the distribution opaque, the ultimate server object (the one that does the real work) is essentially registered with the server stub to be called (triggered) by it upon the receipt of the right message (i.e., the right opcode in the recipient message). Likewise, the client application (or thread of control) can be considered to register itself with the client stub to be triggered (woken, since we're working in a synchronous environment in this example) when the response message comes back. Pub/sub based messaging systems, such as JMS, support the concept of events as well. When you subscribe to a blog you're essentially registering to be triggered when an event (a new blog entry) becomes available. Anyone remember the CORBA Event Service or the DCE Event Management Service?
Back when I was starting my PhD one of the first things I did (for fun, as it really had nothing to do with any of the various subjects I was considering for my PhD at the time) was to write a hardware bus simulator in C++. It had the concept of components being connected to the various buses and registering to be triggered when specific events occurred. In the real world, hardware is event based, with IRQs as the events and the bus as the delivery mechanism. A very good example of an asynchronous system that is often ignored by the software community these days. Maybe we should be looking at calling the next wave Interrupt Driven Architectures instead?
I've written applications on a number of windowing systems over the past 20 years. Probably the best one was InterViews, but they all share a very common pattern: components (e.g., windows, scroll bars, minimisers, etc.) register for events (e.g., left mouse click, mouse move, etc.)
Over the past 5 years or so we've also seen a move away from the synchronous request/response approach to one-way messages with various Web Services standards and SOAP implementations. Services register their interest in specific events (receipt messages) and clients register their interest in specific events (response messages). But it's still very similar to the stub approach I mentioned earlier. But what we're seeing in this approach is making the concept of events and triggers more explicit than it was before (at least in the Web Services world). This is definitely a good thing for loose coupling when you look at asynchronous environments. But is is a good thing for developers and users? Maybe.
So why have we not seen more event based frameworks pushing events as first class entities up to the developer, which seems to be what the current discussions around EDA are about? I think the reason for this is abstractions: we typically try to hide things like events because most of the time we don't think in terms of them. When a sound wave hits my ear drum and is converted into electrical impulses that trigger a response in me, I don't normally equate the sound wave or the electrical impulses as events. When I receive a letter through my post box telling me I've won the lottery (I wish!) I don't think about the steps it took to be delivered as a series of events (though they clearly are). If you look back at what I said earlier concerning how event driven architectures have been with us for many years, you'll see that what's really happened is that we've tended to hide events with abstractions like stubs and message exchanges.
I've said before that I think an event driven approach to SOA (or pretty much anything) is good if it helps you to better reason about the system in question. If it gets in the way, though, there are alternatives that could be better. As with anything, you use the right abstraction for the job. Is this going to be like RPC, where we realise that hiding distribution is a bad thing (we knew all along really)? Is hiding events bad? Well as I just said, that will depend upon what you're doing. But just because you start thinking in terms of events doesn't solve all of your problems immediately. There may be benefit to exposing some business events to the higher level, but we shouldn't think that rewriting everything we currently use in an event-explicit manner will be a global panacea - it may even be a step backwards. As an industry we need to stop pushing point solutions as global solutions and remember that one-size doesn't fit all.
And what about Complex Event Processing (CEP)? In a nutshell CEP gives you the ability to correlate events from different streams over time (time should really be modeled as an event too) to determine patterns, manage and predict other events (such as failure conditions) etc. The data streams can contain messages at various levels of a system architecture, so it's possible to use CEP at the low level (e.g., allowing a system administrator to predict when machines a more likely to fail given daily usage patterns) as well as the higher level (e.g., allowing a business analyst to determine the buying habits of specific individuals given the market trends elsewhere in the world).
So if you've got CEP do you have an EDA? Well you obviously have something that is generating the event (data) streams that are input to the CEP implementation, but that doesn't mean the entire system is (or should be) architected using an event abstraction.
If you've got an EDA do you need CEP? This is more clear-cut: of course you don't need CEP if you've got events. There are more examples of event driven architectures that don't use (and don't need to use) CEP than those that do. But in some cases, it can make things easier to use CEP.
DOA 2008
I wasn't going to be able to come to DOA 2008, which was a shame because I enjoyed DOA 2007. However, neither of my co-chairs could attend either, so it fell to me to come. It's in Monterrey, Mexico this year and the trip here wasn't something I would like to repeat very often. But I'm here now and looking forward the conference. It'll also give me a chance to catchup on some work that I haven't been able to do (because of work) and blog entries that have been filling up my to-do queue. So one way or another the next few days should be interesting.