Some comments around REST and compensations

Bill has written an interesting post which is essentially REST+compensations versus WS-*. While I agree with some of what he has to say (we did REST+compensations at HP back in 2000 - yes Jonathan, I'm still looking for the papers ;-), I have to disagree with several of his observations. Before I look at them in turn, it's best to remind everyone that I'm a "fence sitter" as Mark calls me: I do believe that REST and WS-* have their places and it's not an either/or situation. I sit on the fence for a number of reasons, but one of them is that I don't believe in the 'one-size fits all' argument.

Before I go on, it's important for me to mention that I don't want this to degenerate into a REST versus WS-* debate. In general I agree with the underlying concepts that Bill is discussing. I just know it's not as easy as it seems: it's not clear cut. But neither should you believe that Web Services are necessarily the answer to Life, The Universe and Everything.

Anyway, down to business. The first thing Bill mentions that I have to disagree with is around maintainability.

"One of my concerns is the complexity of the WS-* layer and how it affects maintainability and evolvability of a distributed system. WS-* requires both the client and server to have the necessary middleware components and libraries installed to work."

Yes, but that's the case for any distributed system: the endpoints have got to be able to understand each other or go through some intermediary that can do the protocol/data translation. The same is true for HTTP though.

"What I continually like about REST over HTTP is that there is little to no requirement on either the client or server side for additional libraries or generated stubs other than a http client library and a http server.

So let's assume we can get everyone to agree to use HTTP (not a big assumption, so I'm happy with this). Then we have to get them to agree to the protocol data, the format, representation, etc. that actually flows between the endpoints. It's no good someone saying "because we agree on PUT or GET that's the end of the story." That's a bit like saying that because we agree on using letters and envelopes to send mail around the world the recipients will be able to understand what's being sent! Sure, assuming everyone in the world has an address and a letter box, the mail will get through, but that's not good enough to reliably place an order for goods or send money etc.

Don't overlook what has been driving WS-* over the past few years: interoperability. Yes, we have interoperability on the WWW (ignoring the differences in HTML syntax and browsers). But we do not have interoperabilty for transactions, reliable messaging, workflow etc. That's not to say we can't do it: as I said before, we did manage to do REST+transactions in HP but it was in a small-scale deployment involving only a couple of partners. There is no technical impediment to doing this: it's entirely political. It can be done, I just don't see it ever being done. Until it happens, REST/HTTP cannot compete with the kinds of heterogeneous out-of-the-box interoperability that we have demonstrated with WS-*. And don't point me at Amazon (Werner and I are good friends and I know why they offer REST and WS-*): they're a single company and while they could work with partners to agree on formats etc. that doesn't scale. It's ad hoc.

Bill's next point was around flexibility. He mentions the ordering of compensations can sometimes (often?) be important and that you can't rely on ordering within WS-BA. Unfortunately that's not correct, as was pointed out several years ago. WS-BA works with scopes (similar to nested transactions) and you can control when and how different scopes go off, effectively controlling the order (and individual atomicity) of each scope. WS-BPEL started life using WS-BA for its compensation handlers and there's a lot of flexibility there. Obviously if the API that is layered on the WS-BA implementation does not support this richness, then that's a narrowing of the capabilities, but the underlying protocol is not to blame.

"While WS-BA puts the onus on both the client and server to have WS libraries that are of the same version and can interact with one another. This all works beautifully in controlled environments. You have a very nice decoupling between compensations and business logic in both client and server side code. But…. What happens when BA versions are out of sync, there’s interoperability problems, or even worse, one or more of the services you have to coordinate does not support WS-BA?"

I understand what Bill is on about, but I keep coming back to the same thing: how is this any different from getting REST developers to agree on the payload format, version etc? Just because something supports HTTP (or REST) does not mean it somehow magically understands every single PUT or GET request you send to it. This isn't a dig at Bill, but it is something that has annoyed me for the past 7 years: some REST advocates seem to think that just because the Web works on REST principles, if you use them to develop your own applications/services you'll benefit from them too. D'Oh! Why should that be the case? The Web works because the standards/specifications on which it is based were laid down back in the early 1990's and haven't really changed. The W3C manages them and everyone agreed to abide by them very early on; there's strong commercial incentive not to change things. That has nothing to do with REST and everything to do with protocol agreement: same thing has happened with WS-*. So just because I decide to use REST and HTTP doesn't mean I get instant portability and interoperability. Yes, I get interoperability at the low level, but it says nothing about interoperability at the payload. That's damn hard to achieve, as we've seen over the past 7+ years.

At least with WS-BA (and many of the WS-* specifications/standards) there's an agreed document (standard) that we can point to and test interoperability against. Yes it requires everyone to use it, but that's no different to telling everyone to use HTTP or understand English when reading my letters.

"You still have strong decoupling in a bpm driven activity. The difference is that instead of compensation being hidden by the server, it is visible, and in your face, embedded directly in the business process. With jBPM, you can look at the process diagram and know exactly the flow of success and failure conditions of your distributed application. With BA, you would have to dive into the documentation and code to determine how your application functions."

Yes, that's right, but no different to what you'd find with a good WS-BPEL implementation. WS-BA can be used stand-alone, but as I said before, it was originally developed to compliment WS-BPEL. If you look at the compensation handlers, for instance, you can see how WS-BA can play in that space. Now not all WS-BPEL implementations use WS-BA (political reasons removed the dependency on the specification back in 2003), but some do and do it well, giving you all of the capabilities mentioned above.

"Transactions that have heuristic outcomes many times require human intervention as well as action by the framework itself. Yes, there’s a lot a transaction manager could do to make it easier to record business activities, but you’re still going to have interact with a real person to resolve many issues."

Agreed, but as I said in the original blog entry, sometimes dealing with a problem there and then can save you a lot of time, be more efficient, reliable and fault-tolerant. Since the transaction coordinator records the necessary compensation information, it can try to deal with the compensation immediately (and reliably), based on the information provided to it at enlistment time. In cases where the backend system/service cannot/will not expose compensation capability to the user (there are organisations that do not do this for a number of reasons, including security) then self-compensation followed by sys-admin logging/tasking (which could be done via something like jBPM) is the only option.

So what kind of conclusions can you draw from this? First, I like the idea of REST+X, where X could be transactions, reliable messaging, workflow etc. I wish it could be used in the large, but for the forseeable future it'll be for smaller scale (number of vendors, not physical distribution) applications. Web Services are beneficial, particularly if used right. But as with all technologies, they can be misused. Also, not all things are meant to be used in isolation (WS-BA). Finally, you can't get away from needing a coordinator in these situations: you may hide it, but if you need guaranteed coordination in the presence of failures then you really can't get away from something like a transaction coordinator. How I miss the CORBA Activity Service: it helps unify all of this stuff!