Friday, September 28, 2007

Sunday, September 23, 2007

Thursday, September 20, 2007

More from Bill

Bill responded to my response around his original posting. I'm not going to cut-and-paste everything here, but will concentrate on the main points again. I'll have to delete some of the text from Bill's blog, but hopefully I don't lose the context as a result:

"... I think the layers you have to have in sync are very different in size and complexity. While a REST over HTTP web service only really requires an HTTP client, HTTP web service, and maybe an XML parser, WS-* requires those plus all the WS-* libraries, plus any generated stubs. What I like about REST over HTTP you can actually tell the server what data format you’re sending (content-type) and what format you want back (accept)."

I think Bill's confusing implementations with what's defined by the standards. All WS-* does is state what flows on the wire, e.g., the format of a prepare call. It doesn't define interfaces per se (let's not get into the WSDL versus IDL debate) and really only defines the legal and standard ways in which endpoints can interact. How implementations choose to support this is outside the scope of all of the standards/specifications. Some do it well, some don't. In terms of REST, you still need some kind of programmer support when you think about it (what do you think is going on in the browser?)

Bill then looks at my point around defining the message formats so endpoints can communicate.

"Don’t you have to do the same thing with WS? You still have to agree on the message you are sending. What is the name of the remote method call you are invoking. What are its parameters. What is the data format of its parameters, and so on."

Yes, but that's precisely my point (remember, I said that there's no technical impediment to doing this - it's entirely political and monetary). The companies behind Web Services include IBM, BEA, Red Hat/JBoss, Oracle, IONA, TIBCO, webMethods/Software AG, HP etc. etc. etc. and they've spent the last 7 years defining these payloads and message exchange patterns. Sure it's complex, but when you have to do complex things the infrastructure becomes complex too, either explicitly or implicitly (hidden behind service endpoints). But my point is that it's done in Web Services and we have a set of agreed standards that allow us to do business over the intranet or internet with arbitrary vendors and an arbitrary number of those vendors. We don't need to sit down and argue what it means to associate a transaction policy with a service endpoint, or what it means to have a service fail in the middle of a business transaction. Those things are hard to agree on. Yes, we could do it all again around REST, but I don't think that's going to happen any time soon.

"I am officially in my late 30s. I did work through both the short-lived DCE and long-lived CORBA eras that touted the same story of interoperability. Sorry if I am skeptical and worried that the industry will yet again re-invent itself in another 5-10 years."

Same here, but add a few more years and distributed systems (yes, I feel old at times!) These things do go in cycles. I also have no doubt that Web Services are not the end result and we'll see more evolution and revolution to come. But at the moment Web Services have it. Plus, as I've said before, this is the only time since the start of the WWW that all of the major players have agreed on something like this. It never happened with CORBA. It isn't happening with REST.

"The idioms I described with jBPM use all three of these subsystems, but they do not require a remote protocol definition. As responsible engineers, we should be questioning the need for WS-*. The amount of investment is just too huge for any organization. There’s too much money and time to be lost."

I don't see it as black and white: this is precisely why I don't believe there is one good way of tackling everything. WS-* has its place. So has REST. But then again, a good ESB can help tie all of this together as well. As for the investment reference, I've done some of this in REST before and it was just as time consuming as WS-*. More so now when you have to fight against the WS-* wave of supporters (of which I am one, sitting on my fence).

"Why the need for BA if it is not going to do things automatically for you? If the line between your business process and your compensation engine is starting to blur, why not have your bpm and compensation engine be the same thing?"

BA can do things for you automatically. It just doesn't have to. Plus, don't think of a BA coordinator as being a separate (centralized?) service: all WS-BA defines is a protocol engine; where it sits is entirely up to the developer. So it could well be at the heart of a workflow system. One of the things I've been trying to show is that you still need a coordinator somewhere: something that remembers who and what needs to be done in a reliable manner. It doesn't have to have a narrow interface (commit or rollback), but can have complex ways in which its internal protocol is driven. That was one of the reasons behind the CORBA (J2EE) Activity Service. Hey, we have coordinator technology that'd let us do that!

"Not so sure this is true. If your content type is text/xml, you’ll probably have an XSD associated with it. If you use the design by contract to implement your web services that people like the Spring guys are pushing, you really have the same thing."

OK, so go and persuade Oracle, Amazon, Google, Adobe, Microsoft, TIBCO, company ABC, company XYZ etc. to agree to this and maybe we can start talking about a new global standard for information interchange.

"What I really want here is to have the client drive the application and simplify the complexity of the distributed protocol."

You should take a look at WS-CAF too BTW.

"This is why I said I really liked the idea of using BA with a web application. It would all be local and controllable. After reading up on BPEL compensation handlers, I also like the idea of using them with RESTful web services. Again, put all the responsibility on the client for managing the coordination. Then we don’t have a dual dependency on an in-sync client and server."

Well I still think you'll find that there's a coordinator hidden in there somewhere.

Tuesday, September 18, 2007

Some comments around REST and compensations

Bill has written an interesting post which is essentially REST+compensations versus WS-*. While I agree with some of what he has to say (we did REST+compensations at HP back in 2000 - yes Jonathan, I'm still looking for the papers ;-), I have to disagree with several of his observations. Before I look at them in turn, it's best to remind everyone that I'm a "fence sitter" as Mark calls me: I do believe that REST and WS-* have their places and it's not an either/or situation. I sit on the fence for a number of reasons, but one of them is that I don't believe in the 'one-size fits all' argument.

Before I go on, it's important for me to mention that I don't want this to degenerate into a REST versus WS-* debate. In general I agree with the underlying concepts that Bill is discussing. I just know it's not as easy as it seems: it's not clear cut. But neither should you believe that Web Services are necessarily the answer to Life, The Universe and Everything.

Anyway, down to business. The first thing Bill mentions that I have to disagree with is around maintainability.

"One of my concerns is the complexity of the WS-* layer and how it affects maintainability and evolvability of a distributed system. WS-* requires both the client and server to have the necessary middleware components and libraries installed to work."

Yes, but that's the case for any distributed system: the endpoints have got to be able to understand each other or go through some intermediary that can do the protocol/data translation. The same is true for HTTP though.

"What I continually like about REST over HTTP is that there is little to no requirement on either the client or server side for additional libraries or generated stubs other than a http client library and a http server.

So let's assume we can get everyone to agree to use HTTP (not a big assumption, so I'm happy with this). Then we have to get them to agree to the protocol data, the format, representation, etc. that actually flows between the endpoints. It's no good someone saying "because we agree on PUT or GET that's the end of the story." That's a bit like saying that because we agree on using letters and envelopes to send mail around the world the recipients will be able to understand what's being sent! Sure, assuming everyone in the world has an address and a letter box, the mail will get through, but that's not good enough to reliably place an order for goods or send money etc.

Don't overlook what has been driving WS-* over the past few years: interoperability. Yes, we have interoperability on the WWW (ignoring the differences in HTML syntax and browsers). But we do not have interoperabilty for transactions, reliable messaging, workflow etc. That's not to say we can't do it: as I said before, we did manage to do REST+transactions in HP but it was in a small-scale deployment involving only a couple of partners. There is no technical impediment to doing this: it's entirely political. It can be done, I just don't see it ever being done. Until it happens, REST/HTTP cannot compete with the kinds of heterogeneous out-of-the-box interoperability that we have demonstrated with WS-*. And don't point me at Amazon (Werner and I are good friends and I know why they offer REST and WS-*): they're a single company and while they could work with partners to agree on formats etc. that doesn't scale. It's ad hoc.

Bill's next point was around flexibility. He mentions the ordering of compensations can sometimes (often?) be important and that you can't rely on ordering within WS-BA. Unfortunately that's not correct, as was pointed out several years ago. WS-BA works with scopes (similar to nested transactions) and you can control when and how different scopes go off, effectively controlling the order (and individual atomicity) of each scope. WS-BPEL started life using WS-BA for its compensation handlers and there's a lot of flexibility there. Obviously if the API that is layered on the WS-BA implementation does not support this richness, then that's a narrowing of the capabilities, but the underlying protocol is not to blame.

"While WS-BA puts the onus on both the client and server to have WS libraries that are of the same version and can interact with one another. This all works beautifully in controlled environments. You have a very nice decoupling between compensations and business logic in both client and server side code. But…. What happens when BA versions are out of sync, there’s interoperability problems, or even worse, one or more of the services you have to coordinate does not support WS-BA?"

I understand what Bill is on about, but I keep coming back to the same thing: how is this any different from getting REST developers to agree on the payload format, version etc? Just because something supports HTTP (or REST) does not mean it somehow magically understands every single PUT or GET request you send to it. This isn't a dig at Bill, but it is something that has annoyed me for the past 7 years: some REST advocates seem to think that just because the Web works on REST principles, if you use them to develop your own applications/services you'll benefit from them too. D'Oh! Why should that be the case? The Web works because the standards/specifications on which it is based were laid down back in the early 1990's and haven't really changed. The W3C manages them and everyone agreed to abide by them very early on; there's strong commercial incentive not to change things. That has nothing to do with REST and everything to do with protocol agreement: same thing has happened with WS-*. So just because I decide to use REST and HTTP doesn't mean I get instant portability and interoperability. Yes, I get interoperability at the low level, but it says nothing about interoperability at the payload. That's damn hard to achieve, as we've seen over the past 7+ years.

At least with WS-BA (and many of the WS-* specifications/standards) there's an agreed document (standard) that we can point to and test interoperability against. Yes it requires everyone to use it, but that's no different to telling everyone to use HTTP or understand English when reading my letters.

"You still have strong decoupling in a bpm driven activity. The difference is that instead of compensation being hidden by the server, it is visible, and in your face, embedded directly in the business process. With jBPM, you can look at the process diagram and know exactly the flow of success and failure conditions of your distributed application. With BA, you would have to dive into the documentation and code to determine how your application functions."

Yes, that's right, but no different to what you'd find with a good WS-BPEL implementation. WS-BA can be used stand-alone, but as I said before, it was originally developed to compliment WS-BPEL. If you look at the compensation handlers, for instance, you can see how WS-BA can play in that space. Now not all WS-BPEL implementations use WS-BA (political reasons removed the dependency on the specification back in 2003), but some do and do it well, giving you all of the capabilities mentioned above.

"Transactions that have heuristic outcomes many times require human intervention as well as action by the framework itself. Yes, there’s a lot a transaction manager could do to make it easier to record business activities, but you’re still going to have interact with a real person to resolve many issues."

Agreed, but as I said in the original blog entry, sometimes dealing with a problem there and then can save you a lot of time, be more efficient, reliable and fault-tolerant. Since the transaction coordinator records the necessary compensation information, it can try to deal with the compensation immediately (and reliably), based on the information provided to it at enlistment time. In cases where the backend system/service cannot/will not expose compensation capability to the user (there are organisations that do not do this for a number of reasons, including security) then self-compensation followed by sys-admin logging/tasking (which could be done via something like jBPM) is the only option.

So what kind of conclusions can you draw from this? First, I like the idea of REST+X, where X could be transactions, reliable messaging, workflow etc. I wish it could be used in the large, but for the forseeable future it'll be for smaller scale (number of vendors, not physical distribution) applications. Web Services are beneficial, particularly if used right. But as with all technologies, they can be misused. Also, not all things are meant to be used in isolation (WS-BA). Finally, you can't get away from needing a coordinator in these situations: you may hide it, but if you need guaranteed coordination in the presence of failures then you really can't get away from something like a transaction coordinator. How I miss the CORBA Activity Service: it helps unify all of this stuff!

Saturday, September 15, 2007

D'Oh!

I wonder if I can think of a good reason to be in Atlanta?!

Projects

I need a project to occupy my spare time ("spare time" - kind of a laugh).

Many years ago (towards the end of the 1980's) whilst working on our PhDs, Dan McCue and I developed a SIMULA-like simulation package in C++, cunningly called C++SIM. It was all tied up in the work we were doing at the time around replica placement to achieve high availability. It quickly took on a life of its own and we got requests to use it from a number of academic and commercial organisations. When Java came along it seemed like a good idea to port it across and JavaSim was born. (That "seemed like a good idea" principle was also what lead to JavaArjuna coming from Arjuna.)

Anyway, over the years JavaSim and C++SIM have had quite a few users with several of them asking for feature requests or bug fixes. It's basically been me running the project since Dan left for pastures new. With my own work in Bluestone, HP etc. etc. I've found it harder and harder to get the time to do anything meaningful with either project. So I eventually persuaded the University to allow me to put the source code into open source, which would at least allow other people to work on it and take me out of the bottleneck. Unfortunately that was about a year ago and I've managed to do zero since then! Until today, when I finally requested a project on Codehaus. So maybe sometime in 2008 the code will appear when I find the time and inclination to do more.

In the meantime, work on a D-based transaction manager continues ;-) Why? Because it's there. I'm finding I like D more and more so this is at least an excercise in coming to grips with the language as anything else. Plus, look what happened to JavaArjuna!

Vacation, contracts, annotations, papers etc.

Took a couple of days off this week, trying to use up what vacation I've got before the end of the year. It wasn't planned much in advance, so I didn't have any ideas of what to do. In the end I did some thinking about service contracts and how Java annotations could help developers define them within our ESB. I've also got yet more papers to review (these days each conference or workshop seems to bleed into one another and I'm finding it difficult to tell from the SOA/Web Services papers where one ends and one begins - which might not be a good thing for all of these workshops/conferences!) Plus I found time to write some more of the book that Thomas, Arnaud and I are doing with Thomas Erl (not sure if I've mentioned that before).

So not exactly a vacation but more relaxing than work!

Wednesday, September 05, 2007

A good day's diving!

I took a day off yesterday (which means I only have to take 5 weeks vacation between now and the end of the year!) to go SCUBA diving. It's really hard finding good dive sites in the UK: OK we're an island, but the weather isn't great! Usually it's the North Sea or a lake in North Yorkshire that the Army uses for training. It does get repetitive after a while, so a new site is always nice to investigate. Which is what we did yesterday.

My diving buddy and I had heard about Capernwray, on the other side of the country to us, but somewhere the other divers talked a lot about. It only took us a couple of years to get round to it, but it was definitely worth the trip! A flooded quarry, with sunken planes, helicopters, horses and even a gnome garden. At its deepest it's 21 metres (yes, we went down to check) and the visibility was very good. Managed to get a couple of dives in before heading home. I can see this becoming a regular thing!

Sunday, September 02, 2007