JBossTS/Narayana blog - a reminder

I haven't been cross-posting links to the JBossTS/Narayana blog recently and it's well worth taking a look at the team have written a number of great articles over the year, particularly around microservices and transactions.

A Java EE Interlude

I want to take a quick break from the other series for a moment to do something I've been remiss about for a while: addressing a recent report by Gartner on the state of Java EE. Before doing so I decided the other day to read the article my friend and colleague John Clingan made a few weeks ago and I realised that John had done a great job already. In fact so good I couldn't see myself adding much to it, but I'll try below. Note, there are other responses to this report and I haven't had a chance to read them but they might be just as good.

To start with, it's worth noting that I've known Anne for a number of years and we've had one or two disagreements during that time but also some agreements. Her announcements about the death of SOA seven years or so ago falls somewhere in between, for example. It's also important to realise, if you don't already, that I do believe everything has a natural cycle ("the circle of life") and nothing lasts forever; whether it's dinosaurs giving way to mammals (with some help from a meteor and/or the Deccan Traps), or CORBA shifting aside for J2EE, evolution is a fact of life. Therefore, whilst I disagree with Anne about the short-to-medium term future of Java EE, longer term it will pass into history. However, before doing so it will evolve and continue to influence the next generation of technologies, just as the dinosaurs became the birds and aspects of CORBA evolved into J2EE. For more details on my thinking on this topic over the years I leave it as an exercise to the reader to check this blog or my JBoss blog.

John covers my next point very well but it's worth stressing: I've mentioned on many occasions that my background is heavily scientific and I continue to try to employ objective scientific method to everything I do. So when I see blanket statements like "fade in relevance" or "lighter-weight approaches", whether it's in a Gartner report, a paper I'm reviewing as a PC member, or even something one of my team is writing, I immediately push back if there's no supporting evidence. And I read this report several times searching for it but kept coming up empty handed. It was one subjective statement after another, with no real attempt to justify them. That's disappointing because with my scientific hat on I love to see facts backing up theories, especially if those theories contradict my own. That's how we learn.

One other thing I did get from the report, and I'm prepared to admit this may be my own subjective reading, was that it seemed like a vague attempt to look into the future for middleware frameworks and stacks without really understanding what has brought us to where we are today. I've written about this many times before but weak assertions by the report that "modern applications" are somehow so different from those we've developed for the last 50 years that existing approaches such as Java EE are not useful really doesn't make any sense if you bother to think about it objectively. Distributed applications, services and components need to communicate with each other, which means some form of messaging (hopefully reliable, scalable and something which performs); there's really no such thing as a stateless application, so you'll need to save data somewhere (again, reliable, scalable and performant); hey, maybe you also want some consistency between applications or copies of data, so transactions of one form or another might be of help. And the list goes on.

Of course application requirements change over time (e.g., I recall doing research back in the 1980's where scale was measured in the 10's of machines), but new applications and architectures don't suddenly spring into being and throw away all previous requirements; it's evolution too though. I've presented my thoughts on this over the past couple of years at various conferences. In some ways you can consider Java EE a convenient packaging mechanism for these core services, which are typically deployed in a co-located (single process) manner. Yet if you look beyond the Java EE "veneer" you can still see the influence of enterprise distributed systems that predate it and also the similarities with where some next generation efforts are heading.

I suppose another key point which John also made well was that the report fails miserably to differentiate between Java EE as a standard and the various implementations. I'm not going to cover the entire spectrum of problems I have with this particular failure, but of course lightweight is one of them. Well over 5 years ago I wrote about the things we'd done years previously to make AS7 run on a plug computer or Android phone and we've kept up that level of innovation up throughout the intervening time. There's nothing in the Java EE standard which says you have to build a bloated, heavyweight application server and we, as well as other implementations, have proven that time and time again. It's disappointing that none of this is reflected in the report. I suppose it's one of those terms the authors hope that if they say it enough, people will attribute their own subjective opinions and assumptions to it.

Another one of those throw-away statements the report is good at is that "[...] at this point, Java developers eschew Java EE". I admit not every Java developer wants to use Java EE; not every Java developer wanted to use J2EE either. But you know what? Many Java developers do like Java EE and do want to use it. You only have to go to conferences like JavaOne, Devoxx or other popular Java events to find them in abundance. Or come and talk to some of our customers or those of IBM, Payara, Tomitribe or other members of the MicroProfile effort.

I could go on and on with this but the entry has already grown larger than I expected. John did a great job with his response and you should go read that for an objective analysis. Probably the only positive thing I could attribute to the original Gartner report is that its very existence probably proves that time travel is possible! It's a theory which fits the facts: a report which criticises something based on data which is largely a decade old means it was probably written a decade ago.

Architecture First

We've all heard the story of The Three Little Pigs. We all know that building houses from straw isn't a great idea if your use case is to survive a wolf's breath. We all know that sticks aren't that much better either. Of course brick is a far better medium and the pig that built from it survived. Now what the original story doesn't say, probably because it's a little more detail than children really need to understand, is that before building their houses all of the pigs went to architecture school and studied at length about arches, the truss, stylobates and other things necessary to design and then construct buildings from various materials. Now it's likely the pig that used straw didn't listen when they were talking about the tensile strength of straw and the pig that used sticks ignored the warnings that they're really only good for building forest dens (or birds nests). But if they'd been listening as much as their brother then they'd have known to give him a hand with the bricks and not waste their time.

Now even the best architects make mistakes. It's not possible to suggest a single reason for these though. Sometimes it's because the architect didn't understand the physical properties of the material being used (a bit like the straw and stick pigs). Sometimes it's because they didn't fully understand the environment within which their building would reside. But fortunately for us, the vast majority of buildings are successful and we feel safe to be within them or around them.

You may be wondering why I'm talking about architecture here. I think I've mentioned this before but my biggest worry about the rush towards microservices is the lack of focus, or discussions, around architectures. I'm sure many of the established groups that have been building systems with services (micro or not) understand their architectures and the impact service-orientation has on it, or vice versa. But I'm also convinced that many groups and individuals who are enamoured by the lustre of microservices aren't considering architectures or the implications. That worries me because, as I said at the JavaOne 2016 presentation I gave recently, launching into developing with microservices without understanding their implications and the architecture is neither going to solve any architecture problems you may have with your existing application nor will it result in a good/efficient distributed system. In fact it's probably the worst thing you could do!

Even if you've got "pizza teams", have a culture that has embraced DevOps, have fantastic tools supporting CI and CD, if you don't understand your current and new architecture none of this is really going to help you. That's not to suggest those things aren't important after you've done your architecture design and reviews because they clearly are. The better they are the quicker and more reliably you can build your new application using microservices and manage it afterwards. But you should never start such an effort just because you've got the right tools, building blocks and support infrastructure. It could be argued that they're necessary, but they most certainly aren't sufficient. It needs to start with architecture.

Update: I should also have mentioned that after any architecture review you find that you don't need many, or any, microservices then you shouldn't feel a sense of failure. A good architect (software or not) knows when to use and when not to use things. Remember, it's quite probable the pig who used the bricks considered straw and sticks first but decided they just weren't right this time around.

Microservices and distribution

OK so following on from my previous article on inferring the presence of microservices within an architecture, one possibility would be to view the network traffic. Of course it's no guarantee, but if you follow good microservices principles that are defined today, typically your services are distributed and communicating via HTTP (OK, some people say REST but as usual they tend to mean HTTP). Therefore, if you were to look at the network traffic of an "old style" application (let's not assume it has to be a monolith) and compare it with one that has been re-architected around microservices, it wouldn't be unreasonable to assume that if you saw a lot more HTTP requests flowing then microservices are being used. If the microservices are using some other form of communication, such as JMS, then you'd see something equivalent but with a binary protocol.

We have to recognise that there are a number of reasons why the amount of network traffic may increase from one version of an application to another, so it could be the case that microservices are not being used. However, just as Rutherford did when searching for the atomic nucleus and which all good scientists follow, you come up with a theory that fits the facts and revise it when the facts change. Therefore, for simplicities sake, we'll assume that this could be a good way to infer microservices are in place if all other things remain the same, e.g., the application is released frequently, doesn't require a complete re-install/re-build of user code etc.

Now this leads me to my next question: have you, dear reader, ever bothered to benchmark HTTP or any distributed interaction versus a purely local, IPC, interaction? I think the majority will say Yes and of those who haven't the majority will probably have a gut instinct for the results. Remote invocations are slower, sometimes by several orders of magnitude. Therefore, even ignoring the fault tolerance aspects, remote invocations between microservices are going to have a performance impact on your application. So you've got to ask: why am I doing this? Or maybe: at what point should I stop?

Let's pause for a second and look back through the dark depths of history. Back before the later 19th Century/early 20th Century, before electrification of factories really took off, assembling a product from multiple components typically required having those components shipped in from different parts of the country or the world. It was a slow process. If something went wrong and you got a badly built component, it might prevent assembly of the entire product until a new version had been sourced.

In the intervening years some factories stayed with this model (to this day), whereas others moved to a production factory process whereby all of the pieces were built on site. Some factories became so large, with their constituent pieces being built in their own neighbouring factories that cities grew up around them. However, the aim was that everything was built in one place so that mistakes could be rectified much more quickly. But are these factories monoliths? I'm not so sure it's clear cut simply because some of the examples I know of factories like this are in the Japanese car industry which has adapted to change and innovation extremely well over the years. I'd say these factories matured.

Anyway, let's jump back to the present day but remembering the factory example. You could imagine that factories of the type I mentioned evolved towards their co-located strategy over years from the distributed interaction approach (manufacturers of components at different ends of the planet). They managed to evolve because at some point they had all of the right components being built but the impediment to their sales was time to market or time to react. So bringing everything closer together made sense, Once they'd co-located then maybe every now and then they needed to interact with new providers in other locations and if those became long term dependencies they probably brought them "in house" (or "in factory").

How does this relate to microservices and the initial discussion on distributed invocations? Well whilst re-architecting around microservices might help your application evolve and be released more frequently, at some point you'll need to rev the components and application less and less. It becomes more mature and the requirements for change drop off. At that stage you'd better be asking yourself whether the overhead of separate microservices communicating via HTTP or even some binary protocol, is worth it. You'd better be asking yourself whether it's not better to just bring them all "in house" (or in process) to improve performance (and probably reliability and fault tolerance). If you get it wrong then of course you're back to square one. But if you get it right, that shouldn't mean you have built a monolith! You've just built an application which does it's job really well and doesn't need to evolve much more.

Microservices and subatomic particles - an end-user perspective?

For a while now we've seen various debates around microservices, such as how they compare to SOA, whether the emphasis should be on size (micro), whether HTTP (and REST) is the preferred communication style, where and why you should adopt them as well as when you shouldn't? The list goes on and on and I've participated in a few of them.

Recently at work we've been focusing on how best to consider microservices within an existing architecture, i.e., how, why and when to breakdown so-called monoliths into microservices. We've had a number of our teams involved in these discussions, including Vert.x, WildFly Swarm and OpenShift. We've made great progress and this article isn't about that work - I'll leave it to the various teams and others to report once it's ready.

However, during this work I also went on vacation and that gave me time to ponder on life, the universe and everything microservices related! During the time away I kept coming back to two fundamental questions. The first: why use microservices? The second: how can end-users tell if they're being used to (re-) construct (distributed) applications? Much of what we've heard about microservices has been from the perspective of developers who will use microservices, not necessarily the end-user of (re-)architected applications. And of course you're probably asking a third: how does all of this relate to subatomic particles? Patience and all will be revealed.

To answer the first question, there are a lot of reasons why people, vendors, analysts etc. suggest you should consider microservices, either as a building block for new applications or, as seems more common at the moment, as a way of refactoring your existing application or service(s) which may be monolithic in nature. At the core though is the requirement to have an architecture which allows for constituent components to be developed, revised and released independently of the entire application. The so-called "Pizza Team" approach, for instance.

This then leads us nicely to the second question: how can you tell an application has been developed, or re-architected, using microservices? If you're a user of a service or application, chances are that unless the source code is available to review and you've got that inclination, "microservices enabled" isn't necessarily going to be one of the slogans used to market it. And in fact should you care? Ultimately what you're probably more interested in is a mixture of things such as cost, reliability, performance and suitability for purpose. But let's assume you do want to know. How can you tell?

Well this is where the subatomic particles come in. Given my university degree majored in physics and computing I share an equal love for both and at times when my mind wanders I like to try to see similarities between the two areas. In the past, for instance, I've used Heisenberg's Uncertainty Principle to describe weak consistency transactions protocols. This time around I was again recalling Heisenberg; those of you who have also studied physics or have a passing interest will know that the wave-particle duality of subatomic particles cannot be view directly but can be inferred, for instance using Young's Slit experiment and firing a single "particle" at two slits to observe an interference pattern which is reminiscent of those produced by wave interference. This is a pretty extreme example of how we can infer the properties of particles we cannot view directly. Others exist, including Rutherford's original experiment to infer the existence of the atomic nucleus; I'll leave that as an exercise to the interested reader, but will say it's a fascinating area of science.

Now where all of this comes full circle is that if you're an end-user of some piece of software that has been well architected and does its job, is released frequently enough for you to do your work efficiently and basically doesn't get in the way, could you tell if it was architected or re-architected using microservices? The answer in this case is most probably no. But on the flip side, suppose you've been using an application or service which is released too slowly for you (e.g., bug fixes take months to arrive), and maybe requires a rebuild of your code each time it is released. Then let's assume things change and not only do you get updates on a daily basis but they often fit seamlessly in to your own application usage. Does this mean that the developers have switched to microservices? Unfortunately the answer is no less definitive than previously because whilst a correct use of microservices would be an answer, there are other approaches which could give the same results - despite what you may have read, good software development has existed for decades.

Therefore, without looking at the code how can an end-user know whether or not microservices are being used and why is that important? It's important because there's a lot of hype around microservices at the moment and some people are making purchasing decisions based on whether or not they are present, so you probably do need some way to confirm. Architecture diagrams are great but they're no substitute for code. But if you can't see the code, it's tricky to infer one way or the other. However, on the flip side maybe as an end-user you really shouldn't care as long as you get what you want from the application/service? Good architectures and good software architects win out in the end using a variety of techniques.

Note: yeah, the other obvious analogy between microservices and subatomic particles could be that maybe microservices are the smallest divisible aspects of your application that make sense; you can't really re-factor your code smaller than a microservice in just the same way that you can't go beyond sub-atomic particles. However, since there are things smaller than subatomic I didn't want to go there.

Fault tolerance and microservices

A while ago I wrote about microservices and the unit of failure. At the heart of that was a premise that failures happen (yes, I know, it's a surprise!) and in some ways distributed systems are defined by their ability to tolerate such failures. From the moment our industry decided to venture into the area of distributed computing there has been a need to tackle the issue of what to do when failures happen. At some point I'll complete the presentation I've been working on for a while on the topic, but suffice it to say that various approaches including transactions and replication have been utilised over the years to enable systems to continue to operate in the presence of (a finite number of) failures. One aspect of the move towards more centralised (monolithic?) systems that is often overlooked, if it is even acknowledged in the first place, is the much more simplified failure model: with correct architectural consideration, related services or components fail as a unit, removing some of the "what if?" scenarios we'd have to consider otherwise.

But what more has this got to do with microservices? Hopefully that's obvious: with any service-oriented approach to software development we are inherently moving further into a distributed system. We often hear about the added complexity that comes with microservices that is offset by the flexibility and agility they bring. When people discuss complexity they tend to focus on the obvious: the more component services that you have within your application the more difficult it can be to manage and evolve, without appropriate changes to the development culture. However, the distributed nature of microservices is fundamental and therefore so too is the fact that the failure models will be inherently more complex and must be considered from the start and not as some afterthought.

An update to my microservices rant ...

I spent some time writing something more positive than my previous rant. I hope.

Types of microservices

I've started to post a few things over on the new Red Hat Developer Blog. My first entry was about microservices.

You keep using that word (REST) and I don't think it means what you think it does

A long time ago in a galaxy far, far away ... OK, not quite. But some time ago our industry spent a lot of time and effort on discussion the pros and cons of REST, SOA, SOAP (!) etc. I even had a few things to say on the subject myself. To misquote Jeff Wayne, "minds immeasurably superior to my own" on the topic of REST and HTTP at the time had a lot more to say and do in this area. It's something which is easily Googled these days. Covered in books too. Probably even several video master classes on the topic of REST and HTTP, let alone their relevance to SOA. And yet it seems that some people either have a very short memory, didn't understand what was said, or perhaps didn't do their homework?

I'm specifically talking about how REST plays into the recent microservices wave. Yes, I think there are some problem areas with microservices and yes, one of them is the dogmatic assertion by some that REST (they tend to really mean HTTP) is mandatory. I don't believe that. I do believe that different message exchange patterns may be suitable for a range of microservices and to limit ourselves to just one (HTTP) does not make sense.

Oh and yes, it still frustrates me to this day when people talk about REST and they're really talking about HTTP - my only consolation here is that if I find it frustrating it must really annoy the h*ll out of the RESTafarians! Not all REST is HTTP and likewise not everything which uses HTTP is necessarily REST. Simple, huh? Well you'd think ...

Anyway, putting that aside, what's got me more frustrated recently is that some people are suggesting that REST (really HTTP) can't do async for microservices and therefore can prevent you breaking apart your monolith. I'm not even going to attempt to explain in detail here how that is wrong except to suggest that a) those people should go and read some InfoQ articles from the early 2000's (yes, even theserverside had things to say on the topic), b) do a Google search, c) read something from my friend/colleague Jim Webber, Savas and others on the topic of REST (and maybe Restbucks - hint, hint), d) in case they're confused between REST and HTTP, maybe go and read the HTTP response codes and look at those in the early 200's range (again, another hint). As I said above, I'm not suggesting there aren't some issues with REST/HTTP for microservices. And as I've also said over the last decade or so, maybe there are some issues with building some types of distributed systems with REST/HTTP. But this isn't one of them!

And this then brings me to a related topic. And if he hadn't been saying it around REST, I'm pretty sure Inigo Montoya would've been saying "You keep using that word. I do not think it means what you think it means" about "asynchronous". Yes, some of the authors try to make the distinction between a blocking call within a single address space and a one way interaction with a service across the network, but that really doesn't instil me with confidence that they know what they're talking about. If they threw in a bit of Fischer, Lynch and Patterson, rather than the oft overused reference to CAP, then maybe I'd be slightly less concerned. But that'd require some homework again, which they don't appear to want to do! Oh well, good luck to those who follow them!

Note, I have deliberately not included many links to things in the post in the hopes it will encourage the reader to follow up.

Micromonoliths?

I'm not even sure it's a word, but I wrote something on micromonoliths elsewhere - basically just some words on architecture.

Some cross postings on microservices

As I mentioned earlier, I've had some time on holiday recently and spent some time musing on microservices amongst other things. I've written a couple of articles over on my JBoss.org blog, one on microservices and co-location, and one about the mad rush to make everything a microservice. As Rod Serling often said: submitted for your approval.

Poor scientific method

One of the nice things about being on holiday is that the mind can wander down paths that might otherwise not be trodden, particularly because to do so typically needs more available time. This holiday I spent some of that time pondering microservices, amongst other things. As I've mentioned and been quoted elsewhere, I'm not against the fundamental concepts behind microservices architectures (MSA) - I see them as embodying an evolution of services-based architectures which we've seen happening over several decades. And yet I have a few concerns or niggles about the way in which some groups are positioning it as so significantly different that it represents a fundamental new approach or wave.

It wasn't until I read the recent NIST document that I managed to put my finger on at least one of these concerns: they're making data fit their own definitions of SOA and MSA, whilst conveniently ignoring those facts which don't fit. Any good scientist knows that goes against scientific method, whereby we come up with a theory based upon all of the available data and that theory lasts until some new data appears which blows a hole in it (yes, I'm paraphrasing). The NIST document is particularly guilty of this by choosing a definition of SOA which is conveniently at odds with their definition of MSA - it's not even the one that the standards, such as the OASIS SOA Reference Model, define! The table they have to compare and contrast is so badly flawed - go read the SOA-RM and look at the NIST MSA definition to see how close SOA and MSA are!

Look, I'm not suggesting there's nothing different in MSA than what we've been doing in the past with service oriented architectures. But let's please be objective, stick to the facts and define theories or models based upon all of the available data. Then also be prepared to change if new data appears. But please don't select data to match your own preconceived notions. In our more fully connected world there's really no excuse for being selective of the available data or ignorant of what is available!

HPTS 2015

I've been so busy travelling to conferences and customer engagements that I haven't had a chance to write about my trip to HPTS 2015. I've written several times about previous trips to this workshop and how it's my favourite of them all, so won't repeat. The workshop had the usual high standard of presentations and locating it at Asilomar is always a great way to focus the mind and conversations.

Because of its highly technical nature of the workshop I always like to use this event to try out new presentations - I know the feedback I receive will be constructive and worth hearing. This time my submission was essentially about what I'd written earlier this year concerning the evolution of application servers (application containers) driven by immutability and operating system containers, such as Docker. And I threw in a smattering of microservices since the topic is obviously relevant and I figured Adrian would be there! My presentation was well received and the feedback clearly showed that many people at the event agreed with it.

One other positive thing to come from the workshop and my presentation was that my co-traveller and long time friend/colleague, Professor Shrivastava, saw the presentation for the first time at the event. He understood it and whilst much of what was said I and others take for granted, he believes that there are groups of people that would find it interesting enough that we should write a paper. Writing papers with Santosh is something I enjoy and it has been a very fruitful collaboration over the years, so I look forward to this!

I also want to thank James because it was our discussions after I started my initial entries on the evolution of application servers that helped to focus and clarify my thinking.

A little more on transactions and microservices

I had some interesting discussions this week which made me want to write something more on transactions (really XA) and microservices.

SOA done right?

I said something the other day in a presentation I was giving that I immediately regretted: I was asked how to position microservices and SOA for marketing, to which I responded "Microservices are SOA done right". A nice sound bite, but the reason I regret it is that someone could infer from it that anything termed a microservice is a good SOA citizen. That's not the case. Just as not everything termed SOA or REST was following good SOA or REST principles, so too will be the case with microservices. And the lack of a formal definition of microservices, whether in terms of recognisable SOA principles or something else, doesn't help matters.

So what should I have said? In hindsight, and remembering that it had to be short and to the point, I'd probably revisit and say something along the lines of "If the service follows good SOA principles then it's probably on the way to being the basis of a microservice". Not quite as snappy a response, but probably as far as I'd feel comfortable going in a single sentence. Of course it then could lead to follow up questions such as "What's the difference between a good SOA citizen/service and a microservice?" but then we're into a technical domain and no longer marketing.

Microservices and events

I wanted to write more on microservices (SOA) and specifically around reactive, event-orientation. We've heard a lot recently about frameworks such as Vert.x or Node.js which are reactive frameworks for Java, JavaScript and other languages (in the case of Vert.x). Developers are using them for a range of applications but because they present a reactive, event driven approach to building services and applications, it turns out they're also useful for microservices. Therefore, I wanted to give an indication of why they are useful for microservices and when I started to put proverbial pen to paper I found I was repeating a lot of what I'd written about before and specifically when I discussed some of the core reasons behind the future adaptive middleware platform.

Now I'm not suggesting that Vert.x, Node.js or any reactive, event-oriented framework or platform fulfills everything I mentioned in that earlier entry because I don't think they do, at least not today. However, they do represent a good basis from which to build. Of course that doesn't preclude existing architectures and associated software stacks from being used to build microservices. In fact I think a combination of these new approaches and existing mature stacks are the right way to go; with each new generation we learn the mistakes of the past and their successes, building upon the latter and hopefully not reproducing the former.

Unfortunately I'm not going to endorse the reactive manifesto, because I'm not entirely convinced that what's encompassed there really represents much more than good fault tolerance and message-driven architecture. For instance, responsive? I'd like to think that any useful platform should be responsive! I think we should add that to the "bleeding obvious" category. My definition of reactive would include fault tolerant, scalable, reliable, message-oriented (supporting synchronous and asynchronous), self-healing (resiliency is only part of that) and event-driven.

Therefore, when I say that reactive, event-oriented is a good fit for microservices (remember ... SOA) I'm talking about something slightly different than reactive as defined within the manifesto. It's all that and more. We've come a long way since the 1970's and 80's when things were typically synchronous and RPCs were the main paradigm. Even those are event driven if you think about it. But since then synchronous interactions fell out of favour, asynchronous rose in interest and so did event-orientation and associated efforts like CEP. Hopefully given what I wrote in that earlier article it should be fairly obvious what kind of platform characteristics I believe are necessary in the future - microservices is just a good approach that can take advantage of it.

As I said earlier though, I don't believe any implementation today really embodies where we need to be. There's a lot we can leverage from the past decades of enterprise software development but there is more work to be done. For instance, we hear a lot about asynchronous systems where in reality what most people are talking about is synchronous interactions performed by a separate thread to the main application (business logic) thread of control; there are known time bounds associated with work or interactions, even in a distributed environment. If we really want to talk about true asynchronous interactions, and I believe we do, then we are looking at unbounded interactions, where you simply do not know when a message is overdue. That causes a number of problems, including failure detection/suspicion (timeouts are a favourite way of trying to determine when a machine may have failed and now you can't use time as a factor), and distributed consensus, which has been proven to be unsolvable in an asynchronous environment.

Of course these may seem like purely theoretical or academic problems. But think about the kinds of environments we have today and which are growing in applicability: cloud, mobile, internet of things with sensors, gateways etc. These are inherently asynchronous, with unpredictable interaction patterns and message exchanges. Whilst synchronous or multi-threaded asynchronous may be a good stop-gap approach, it will inevitably present scalability and resiliency problems. True asynchronous, with all of the problems it represents, is the ultimate goal. But we're not there yet and it's going to take time as well as recognition by developers that further research and development is a necessity.

Transactions and microservices

Sometimes I think I have too many blogs to cover with articles. However, this time what I was writing about really did belong more in the JBoss Transactions team blog, so if you're interested in my thoughts on where transactions (ACID, compensation etc.) fit into the world of microservices, then check it out.

Containerless Microservices?

A few weeks ago I wrote about containerless development within Java. I said at the time that I don't really believe there's ever the absence of a container in one form or another, e.g., your operating system is a container. Then I wrote about containers and microservices. Different containers from those we're used to in the Java world, but definitely something more and more Java developers are becoming interested in. However, what I wanted to make clear here is that in no way are microservices (aka SOA) tied to (Linux) containers, whether Docker-based or some other variant.

Yes, those containers can make your life easier, particularly if you're in that DevOps mode and working in the Cloud. But just as not all software needs to run in the Cloud, neither do all services (whether micro or macro) need to be deployed into a (Linux) container. What I'm going to say will be in terms of Java, but it's really language agnostic: if you're looking to develop a service within Java then you already have great tools and 3rd party components to help you get there. The JVM is a wonderful container for all of your services that has had many years of skilled development within it to help make it reliable and perform. It also runs on a pretty much every operating system you can mention today and hardware ranging from constrained devices such as the Raspberry Pi, up to and including mainframes.

In the Java world we have a rich heritage of integration tools, such as Apache Camel and HornetQ. The community has built some of the best SOA tools and frameworks anywhere, so of course Java should be one of the top languages in which to develop microservices. If you're thinking about developing microservices in Java then you don't have to worry about using Linux containers: your unit of failure is the JVM. Start there and built upward. Don't throw in things (software components) that aren't strictly necessary. Remember what Einstein said: "Everything should be made as simple as possible, but not simpler."

However, as I said earlier, the Linux container can help you once you've got your microservice(s) up and running. Developing locally on your laptop, say, is unlikely to be the place where you want to start with something like Docker, especially if you're just learning how to create microservices in the first place. (Keep the number of variables to a minimum). But once you've got the service tested and working, Docker and its like represent a great way of packaging them up and deploying them elsewhere.

Microservices and state

In a previous entry I was talking about the natural unit of failure for a microservice(s) being the container (Docker or some other implementation). I touched briefly on state but only to say that we should assume stateless instances for the purpose of that article and we'd come back to state later. Well it's later and I've a few thoughts on the topic. First it's worth noting that if the container is the unit of failure, such that all servers within an image fail together, then it can also be the unit of replication. Again let's ignore durable state for now but it still makes sense to spin up multiple instances of the same image to handle load and provide fault tolerance (increased availability). In fact this is how cluster technologies such as Kubernetes work.

I've spent a lot of time over the years working in the areas of fault tolerance, replication and transactions; it doesn't matter whether we're talking about replicating objects, services, or containers, the principles are the same. This got me to thinking that something I wrote over 22 years ago might have some applicability today. Back then we were looking at distributed objects and strongly consistent replication using transactions. Work on weakly consistent replication protocols was in its infancy and despite the fact that today everyone seems to be offering them in one form or another and trying to use them within their applications, their general applicability is not as wide as you might believe; and pushing the problem of resolving replica inconsistencies up to the application developer isn't the best thing to do! However, once again this is getting off topic a bit and perhaps something else I'll come back to in a different article. For now let's assume if there are replicas then their states will be in sync (that doesn't require transactions, but they're a good approach).

In order to support a wide range of replication strategies, ranging from primary-copy (passive) through to available copies (active), we created a system whereby the object methods (code) was replicated separately from the object state. In this way the binaries representing the code was immutable and when activated they'd read the state from elsewhere; in fact because the methods could be replicated to different degress from the state, it was possible for multiple replicated methods (servers) to read their state from an unreplicated object state (store). I won't spoil the plot too much so the interested reader can take a look at the original material. However, I will add that there was a mechanism for naming and locating these various server and state replicas; we also investigated how you could place the replicas (dynamically) on various machines to obtain a desired level of availability, since availability isn't necessarily proportional to the number of replicas you've got.

If you're familiar with Kubernetes (other clustering implementations are possible) then hopefully this sounds familiar. There's a strong equivalency between the components and approaches that Kubernetes uses and what we had in 1993; of course other groups and systems are also similar and for good reasons - there are some fundamental requirements that must be met. Let's get back to how this all fits in with microservices, if that wasn't already obvious. As before, I'll talk about Kubernetes but if you're looking at some other implementation it should be possible to do a mental substitution.

Kubernetes assumes that the images it can spin up as replicas are immutable and identical, so it can pull an instance from any repository and place it on any node (machine) without having to worry about inconsistencies between replicas. Docker doesn't prevent you making changes to the state within a specific image but this results in a different image instance. Therefore, if your microservice(s) within an image maintain their state locally (within the image), you would have to ensure that this new image instance was replicated in the repositories that something like Kubernetes has access to when it creates the clusters of your microservices. That's not an impossible task, of course, but it does present some challenges, including how to distribute the updated image amongst the repositories in a timely manner - you wouldn't want a strongly consistent cluster to be created with different versions of the image because that means different states and hence not consistent, and how to ensure that state changes that happen at each Docker instance and result in a new image being created are in lock-step - one of the downsides of active replication is that it assumes determinism for the replica, i.e., given the same start state and the same set of messages in the same order, the same end state will result; not always possible if you have non-deterministic elements in your code, such as the time of day. There are a number of ways in which you can ensure consistency of state, but we're not just talking about the state of your service now, it's also got to include the entire Docker image.

Therefore, overall it can be a lot simpler to factor the binary that implements your algorithms for your microservices (aka the Docker image or 1993 object) from the state and consider the images within which the microservices reside to be immutable; any state changes that do occur must be saved (made durable) "off image" or be lost when the image is passivated, which could be fine for your services of course, if there's no need for state durability. Of course if you're using active replication then you still have to worry about determinism, but we're only considering state here and not the actual entire Docker image binary too. The way in which the states are kept consistent is covered by a range of protocols, which are well documented in the literature. Where the state is actually saved (the state store, object store, or whatever you want to call it) will depend upon your requirements for the microservice. There are the usual suspects, such as RDBMS, file system, NoSQL store, or even highly available (replicated) in memory data stores which have no persistent backup and rely upon the slim chance that a catastrophic failure will occur to wipe out all of the replicas (even persistent stores have a finite probability that they'll fail and you will lose your data). And of course the RDBMS, file system etc. should be replicated or you'll be putting all of your eggs in the same basket!

One final note (for now): so far we've been making the implicit assumption that each Docker image that contains your microservices in a cluster is identical and immutable. What if we relaxed the identical aspect slightly and allowed different implementations of the same service, written by different teams and potentially in different languages? Of course for simplicitly we should assume that these implementations can all read and write the same state (though even that limitation could be relaxed with sufficient thought). Each microservice in an image could be performing the same task, written against the same algorithm, but with the hopes that bugs or inaccuracies produced by one team were not replicated by others, i.e., this is n-versioning programming. Because these Docker images contain microservices that can deal with each other's states, all we have to do is ensure that Kubernetes (for instance) spins up sufficient versions of these heterogeneous images to give us a desired level of availability in the event of coding errors. That shouldn't be too hard to do since it's something research groups were doing back in the 1990's.

Microservices and the unit of failure

I've seen and heard people fixating on the "micro" bit of "microservices". Some people believe that a microservice should be no larger than "a few lines of code" or a "few megabytes" and there has been at least one discussion about nanoservices! I don't think we should fixate on the size but rather that old Unix addage from Doug McIlroy: "write programs that do one thing and do it well". Replace "programs" with "service". It doesn't matter if that takes 100 lines of code or 1000 (or more or less).

As I've said several times, I think the principles behind microservices aren't that far removed from "traditional" SOA, but what is driving the former is a significant change in the way we develop and deploy applications, aka DevOps, or even NoOps if you're tracking Netflix and others. Hand in hand with these changes come new processes, tools, frameworks and other software components, many of which are rapidly becoming part of the microservices toolkit. In some ways it's good to see SOA evolve in this way and we need to make sure we don't forget all of the good practices that we've learnt over the years - but that's a different issue.

Anyway, chief amongst those tools is the rapid evolution of container technolgoies, such as Docker (other implementations are available, of course!) For simplicity I'll talk about Docker in the rest of this article, but if you're using something else then you should be able to do a global substitution and have the same result. Docker is great at creating stable deployment instances for pretty much anything (as long as it runs in Linux, at the moment). For instance, you can distribute your product or project as a Docker image and the user can be sure it'll work as you intended because you went to the effort to ensure that any third party dependencies were taken care of at the point you built it; so even if that version of Foobar no longer exists in the world, if you had it and needed it when you built your image then that image will run just fine.

So it should be fairly obvious why container images, such as those based on Docker, make good deployment mechanisms for (micro) services. In just the same way as technologies such as OSGi did (and still do), you can package up your service and be sure it will run first time. But if you've ever looked at a Docker image you'll know that they're not exactly small; depending upon what's in them, they can range from 100s of megabytes of gigabytes in size. Now of course if you're creating microservices and are focussing on the size of the service, then you could be worried about this. However, as I mentioned before, I don't think size is the right metric on which to base the conclusion of whether a service fits into the "microservice" category. Furthermore, you've got to realise that there's a lot more in that image than the service you created, which could in fact be only a few 100s of lines of code: you've got the entire operating system, for a start!

Finally there's one very important reason why I think that despite the size of Docker images being rather large, you should still consider them for your (micro) service deployments: they make a great unit of failure. We rarely build and deploy a single service when creating applications. Typically an application will be built from a range of services, some built by different teams. These services will have differing levels of availability and reliability. They'll also have different levels of dependency between one another. Crucially there will be groupings of services which should fail together, or at least if one of them fails the others may as well fail because they can't be useful to the application (clients or other services) until the failed service has recovered.

In previous decades, and even today, we've looked at middleware systems that would automatically deploy related services on to the same machine and, where possible, into the same process instance, such that the failure of the process or machine would fail the unit. Furthermore, if you didn't know or understand these interdependencies a priori, some implementations could dynamically track them and migrate services closer to each other and maybe even on to the same machine eventually. Now this kind of dynamism is still useful in some environments, but with containers such as Docker you can now create those units of failures from the start. If you are building multiple microservices, or using them from other groups and organisations, within your applications or composite service(s), then do some thinking about how they are related and if they should fail as a unit then pull them together into a single image.

Note I haven't said anything about state here. Where is state stored? How does it remain consistent across failures? I'm assuming statelessness at the moment, so technologies such as Kubernetes can manage the failure and recovery of immutable (Docker) images. Once you inject state then some things may change, but let's cover that at another date and time.