Microservices and distribution

OK so following on from my previous article on inferring the presence of microservices within an architecture, one possibility would be to view the network traffic. Of course it's no guarantee, but if you follow good microservices principles that are defined today, typically your services are distributed and communicating via HTTP (OK, some people say REST but as usual they tend to mean HTTP). Therefore, if you were to look at the network traffic of an "old style" application (let's not assume it has to be a monolith) and compare it with one that has been re-architected around microservices, it wouldn't be unreasonable to assume that if you saw a lot more HTTP requests flowing then microservices are being used. If the microservices are using some other form of communication, such as JMS, then you'd see something equivalent but with a binary protocol.

We have to recognise that there are a number of reasons why the amount of network traffic may increase from one version of an application to another, so it could be the case that microservices are not being used. However, just as Rutherford did when searching for the atomic nucleus and which all good scientists follow, you come up with a theory that fits the facts and revise it when the facts change. Therefore, for simplicities sake, we'll assume that this could be a good way to infer microservices are in place if all other things remain the same, e.g., the application is released frequently, doesn't require a complete re-install/re-build of user code etc.

Now this leads me to my next question: have you, dear reader, ever bothered to benchmark HTTP or any distributed interaction versus a purely local, IPC, interaction? I think the majority will say Yes and of those who haven't the majority will probably have a gut instinct for the results. Remote invocations are slower, sometimes by several orders of magnitude. Therefore, even ignoring the fault tolerance aspects, remote invocations between microservices are going to have a performance impact on your application. So you've got to ask: why am I doing this? Or maybe: at what point should I stop?

Let's pause for a second and look back through the dark depths of history. Back before the later 19th Century/early 20th Century, before electrification of factories really took off, assembling a product from multiple components typically required having those components shipped in from different parts of the country or the world. It was a slow process. If something went wrong and you got a badly built component, it might prevent assembly of the entire product until a new version had been sourced.

In the intervening years some factories stayed with this model (to this day), whereas others moved to a production factory process whereby all of the pieces were built on site. Some factories became so large, with their constituent pieces being built in their own neighbouring factories that cities grew up around them. However, the aim was that everything was built in one place so that mistakes could be rectified much more quickly. But are these factories monoliths? I'm not so sure it's clear cut simply because some of the examples I know of factories like this are in the Japanese car industry which has adapted to change and innovation extremely well over the years. I'd say these factories matured.

Anyway, let's jump back to the present day but remembering the factory example. You could imagine that factories of the type I mentioned evolved towards their co-located strategy over years from the distributed interaction approach (manufacturers of components at different ends of the planet). They managed to evolve because at some point they had all of the right components being built but the impediment to their sales was time to market or time to react. So bringing everything closer together made sense, Once they'd co-located then maybe every now and then they needed to interact with new providers in other locations and if those became long term dependencies they probably brought them "in house" (or "in factory").

How does this relate to microservices and the initial discussion on distributed invocations? Well whilst re-architecting around microservices might help your application evolve and be released more frequently, at some point you'll need to rev the components and application less and less. It becomes more mature and the requirements for change drop off. At that stage you'd better be asking yourself whether the overhead of separate microservices communicating via HTTP or even some binary protocol, is worth it. You'd better be asking yourself whether it's not better to just bring them all "in house" (or in process) to improve performance (and probably reliability and fault tolerance). If you get it wrong then of course you're back to square one. But if you get it right, that shouldn't mean you have built a monolith! You've just built an application which does it's job really well and doesn't need to evolve much more.

Microservices and subatomic particles - an end-user perspective?

For a while now we've seen various debates around microservices, such as how they compare to SOA, whether the emphasis should be on size (micro), whether HTTP (and REST) is the preferred communication style, where and why you should adopt them as well as when you shouldn't? The list goes on and on and I've participated in a few of them.

Recently at work we've been focusing on how best to consider microservices within an existing architecture, i.e., how, why and when to breakdown so-called monoliths into microservices. We've had a number of our teams involved in these discussions, including Vert.x, WildFly Swarm and OpenShift. We've made great progress and this article isn't about that work - I'll leave it to the various teams and others to report once it's ready.

However, during this work I also went on vacation and that gave me time to ponder on life, the universe and everything microservices related! During the time away I kept coming back to two fundamental questions. The first: why use microservices? The second: how can end-users tell if they're being used to (re-) construct (distributed) applications? Much of what we've heard about microservices has been from the perspective of developers who will use microservices, not necessarily the end-user of (re-)architected applications. And of course you're probably asking a third: how does all of this relate to subatomic particles? Patience and all will be revealed.

To answer the first question, there are a lot of reasons why people, vendors, analysts etc. suggest you should consider microservices, either as a building block for new applications or, as seems more common at the moment, as a way of refactoring your existing application or service(s) which may be monolithic in nature. At the core though is the requirement to have an architecture which allows for constituent components to be developed, revised and released independently of the entire application. The so-called "Pizza Team" approach, for instance.

This then leads us nicely to the second question: how can you tell an application has been developed, or re-architected, using microservices? If you're a user of a service or application, chances are that unless the source code is available to review and you've got that inclination, "microservices enabled" isn't necessarily going to be one of the slogans used to market it. And in fact should you care? Ultimately what you're probably more interested in is a mixture of things such as cost, reliability, performance and suitability for purpose. But let's assume you do want to know. How can you tell?

Well this is where the subatomic particles come in. Given my university degree majored in physics and computing I share an equal love for both and at times when my mind wanders I like to try to see similarities between the two areas. In the past, for instance, I've used Heisenberg's Uncertainty Principle to describe weak consistency transactions protocols. This time around I was again recalling Heisenberg; those of you who have also studied physics or have a passing interest will know that the wave-particle duality of subatomic particles cannot be view directly but can be inferred, for instance using Young's Slit experiment and firing a single "particle" at two slits to observe an interference pattern which is reminiscent of those produced by wave interference. This is a pretty extreme example of how we can infer the properties of particles we cannot view directly. Others exist, including Rutherford's original experiment to infer the existence of the atomic nucleus; I'll leave that as an exercise to the interested reader, but will say it's a fascinating area of science.

Now where all of this comes full circle is that if you're an end-user of some piece of software that has been well architected and does its job, is released frequently enough for you to do your work efficiently and basically doesn't get in the way, could you tell if it was architected or re-architected using microservices? The answer in this case is most probably no. But on the flip side, suppose you've been using an application or service which is released too slowly for you (e.g., bug fixes take months to arrive), and maybe requires a rebuild of your code each time it is released. Then let's assume things change and not only do you get updates on a daily basis but they often fit seamlessly in to your own application usage. Does this mean that the developers have switched to microservices? Unfortunately the answer is no less definitive than previously because whilst a correct use of microservices would be an answer, there are other approaches which could give the same results - despite what you may have read, good software development has existed for decades.

Therefore, without looking at the code how can an end-user know whether or not microservices are being used and why is that important? It's important because there's a lot of hype around microservices at the moment and some people are making purchasing decisions based on whether or not they are present, so you probably do need some way to confirm. Architecture diagrams are great but they're no substitute for code. But if you can't see the code, it's tricky to infer one way or the other. However, on the flip side maybe as an end-user you really shouldn't care as long as you get what you want from the application/service? Good architectures and good software architects win out in the end using a variety of techniques.

Note: yeah, the other obvious analogy between microservices and subatomic particles could be that maybe microservices are the smallest divisible aspects of your application that make sense; you can't really re-factor your code smaller than a microservice in just the same way that you can't go beyond sub-atomic particles. However, since there are things smaller than subatomic I didn't want to go there.

Fault tolerance and microservices

A while ago I wrote about microservices and the unit of failure. At the heart of that was a premise that failures happen (yes, I know, it's a surprise!) and in some ways distributed systems are defined by their ability to tolerate such failures. From the moment our industry decided to venture into the area of distributed computing there has been a need to tackle the issue of what to do when failures happen. At some point I'll complete the presentation I've been working on for a while on the topic, but suffice it to say that various approaches including transactions and replication have been utilised over the years to enable systems to continue to operate in the presence of (a finite number of) failures. One aspect of the move towards more centralised (monolithic?) systems that is often overlooked, if it is even acknowledged in the first place, is the much more simplified failure model: with correct architectural consideration, related services or components fail as a unit, removing some of the "what if?" scenarios we'd have to consider otherwise.

But what more has this got to do with microservices? Hopefully that's obvious: with any service-oriented approach to software development we are inherently moving further into a distributed system. We often hear about the added complexity that comes with microservices that is offset by the flexibility and agility they bring. When people discuss complexity they tend to focus on the obvious: the more component services that you have within your application the more difficult it can be to manage and evolve, without appropriate changes to the development culture. However, the distributed nature of microservices is fundamental and therefore so too is the fact that the failure models will be inherently more complex and must be considered from the start and not as some afterthought.

Serverless? Really?

Our industry appears to be going through a phase of giving new or not so new approaches short names which though catchy are so inaccurate as to be meaningless and possibly dangerous. These include "containerless", when containers of one sort or another are clearly present. Now we have "serverless ".

Look, I absolutely get what's behind the term: cloud has made it so that developers don't need to worry about deploying databases, web severs or whatever is needed to run their application and also takes care of scaling and fault tolerance. But servers and databases and other infrastructure are still there because your application still needs them; just because you don't see them doesn't mean they're not there.

An update to my microservices rant ...

I spent some time writing something more positive than my previous rant. I hope.

Types of microservices

I've started to post a few things over on the new Red Hat Developer Blog. My first entry was about microservices.

You keep using that word (REST) and I don't think it means what you think it does

A long time ago in a galaxy far, far away ... OK, not quite. But some time ago our industry spent a lot of time and effort on discussion the pros and cons of REST, SOA, SOAP (!) etc. I even had a few things to say on the subject myself. To misquote Jeff Wayne, "minds immeasurably superior to my own" on the topic of REST and HTTP at the time had a lot more to say and do in this area. It's something which is easily Googled these days. Covered in books too. Probably even several video master classes on the topic of REST and HTTP, let alone their relevance to SOA. And yet it seems that some people either have a very short memory, didn't understand what was said, or perhaps didn't do their homework?

I'm specifically talking about how REST plays into the recent microservices wave. Yes, I think there are some problem areas with microservices and yes, one of them is the dogmatic assertion by some that REST (they tend to really mean HTTP) is mandatory. I don't believe that. I do believe that different message exchange patterns may be suitable for a range of microservices and to limit ourselves to just one (HTTP) does not make sense.

Oh and yes, it still frustrates me to this day when people talk about REST and they're really talking about HTTP - my only consolation here is that if I find it frustrating it must really annoy the h*ll out of the RESTafarians! Not all REST is HTTP and likewise not everything which uses HTTP is necessarily REST. Simple, huh? Well you'd think ...

Anyway, putting that aside, what's got me more frustrated recently is that some people are suggesting that REST (really HTTP) can't do async for microservices and therefore can prevent you breaking apart your monolith. I'm not even going to attempt to explain in detail here how that is wrong except to suggest that a) those people should go and read some InfoQ articles from the early 2000's (yes, even theserverside had things to say on the topic), b) do a Google search, c) read something from my friend/colleague Jim Webber, Savas and others on the topic of REST (and maybe Restbucks - hint, hint), d) in case they're confused between REST and HTTP, maybe go and read the HTTP response codes and look at those in the early 200's range (again, another hint). As I said above, I'm not suggesting there aren't some issues with REST/HTTP for microservices. And as I've also said over the last decade or so, maybe there are some issues with building some types of distributed systems with REST/HTTP. But this isn't one of them!

And this then brings me to a related topic. And if he hadn't been saying it around REST, I'm pretty sure Inigo Montoya would've been saying "You keep using that word. I do not think it means what you think it means" about "asynchronous". Yes, some of the authors try to make the distinction between a blocking call within a single address space and a one way interaction with a service across the network, but that really doesn't instil me with confidence that they know what they're talking about. If they threw in a bit of Fischer, Lynch and Patterson, rather than the oft overused reference to CAP, then maybe I'd be slightly less concerned. But that'd require some homework again, which they don't appear to want to do! Oh well, good luck to those who follow them!

Note, I have deliberately not included many links to things in the post in the hopes it will encourage the reader to follow up.

Micromonoliths?

I'm not even sure it's a word, but I wrote something on micromonoliths elsewhere - basically just some words on architecture.

Some cross postings on microservices

As I mentioned earlier, I've had some time on holiday recently and spent some time musing on microservices amongst other things. I've written a couple of articles over on my JBoss.org blog, one on microservices and co-location, and one about the mad rush to make everything a microservice. As Rod Serling often said: submitted for your approval.

Poor scientific method

One of the nice things about being on holiday is that the mind can wander down paths that might otherwise not be trodden, particularly because to do so typically needs more available time. This holiday I spent some of that time pondering microservices, amongst other things. As I've mentioned and been quoted elsewhere, I'm not against the fundamental concepts behind microservices architectures (MSA) - I see them as embodying an evolution of services-based architectures which we've seen happening over several decades. And yet I have a few concerns or niggles about the way in which some groups are positioning it as so significantly different that it represents a fundamental new approach or wave.

It wasn't until I read the recent NIST document that I managed to put my finger on at least one of these concerns: they're making data fit their own definitions of SOA and MSA, whilst conveniently ignoring those facts which don't fit. Any good scientist knows that goes against scientific method, whereby we come up with a theory based upon all of the available data and that theory lasts until some new data appears which blows a hole in it (yes, I'm paraphrasing). The NIST document is particularly guilty of this by choosing a definition of SOA which is conveniently at odds with their definition of MSA - it's not even the one that the standards, such as the OASIS SOA Reference Model, define! The table they have to compare and contrast is so badly flawed - go read the SOA-RM and look at the NIST MSA definition to see how close SOA and MSA are!

Look, I'm not suggesting there's nothing different in MSA than what we've been doing in the past with service oriented architectures. But let's please be objective, stick to the facts and define theories or models based upon all of the available data. Then also be prepared to change if new data appears. But please don't select data to match your own preconceived notions. In our more fully connected world there's really no excuse for being selective of the available data or ignorant of what is available!

Holiday thoughts ...

I just got back from a week of holidaying with family in Tenerife. We did lots of great things together, such as whale watching, snorkeling, visiting the local volcano (great mountain roads with barely enough space for one car let alone two lanes!) and generally soaking up the sunshine! It also gave me an opportunity to catch up on some articles and work related videos I'd been saving up on for a while, which resulted in me also being able to write down some of my own thoughts on a few related areas. I'm taking a bit of a down-day today before back to work tomorrow so it may take me a few days to upload the entries here or to my JBoss.org account.

Frameworks versus stacks - cross posting

Wasn't really sure if I should put this on my JBoss.org blog or here, so cross posting.

SRC-IoT 2016

I have had the honour of working with Professor Paul Watson to create the System Research Challenges Workshop. For the first year we focussed on IoT and it's possible we may keep that high-level theme next time. But it really great to see and hear the attendees embrace the underlying issues which are presented by large-scale distributed systems, which IoT embodies: reliability, fault tolerance, security, trustworthiness, data management etc. Of course all of the presentations had an IoT focus, but even there we had a wide range of examples, from field devices through gateways and including wearables (yes, there were a lot of side discussions about whether some of these IoT devices would ever really take off.)

We had attendees from industry (e.g., Red Hat, IBM and ARM) as well as SMEs and arcademia (e.g., Newcastle University, Lyon and Cambridge). It was a great mix of practical and theoretical, highlighting some of the challenges we have ahead of us in research and development. And as with many of these kinds of events, it was the discussions around the sessions that generated as much interesting conversation as during the presentations.

As well as 2 days of 30 minute presentations (maybe we'll try and get the agenda published somewhere), we also held a 2 hour lightning talk session on the first evening. Here anyone attending, whether they had a formal presentation or not during the event, was encouraged to present on a topic for 5 minutes. There hadn't been much preparation for this beforehand, so there was a little concern about whether we'd be able to fill the time. We needn't have worried - we could have gone much longer than the allotted 2 hours. It was a lot of fun. In fact my favourite talk of the entire event was probably here when Jonathan Halliday gave a presentation onf Big Data over the centuries, going back hundreds of years and managing to also touch on open source 400 years ago!

In conclusion, I thought the event went well. I'm hoping we can do it again next year, perhaps with the same theme or maybe we need to change it. We'll know closer to the time.

Elite

Back when I was just starting at university Elite came out for the Beeb. I remember going into my local town and walking to the computer shop to buy the game, then getting the slow bus home, waiting with apprehension for the time when I could put the tape into the cassette drive and slowly load up the game! By today's standards the graphics were basic, but in 1984 they were ground breaking. And I spent the next couple of years playing Elite at every opportunity. Even when I upgraded to the Atari 520 STM I longed for Elite on it but it never arrived.

I learned about Elite Dangerous back in 2012 and that it was available on Steam earlier this year. I wasn't a Steam player at the time, preferring to do my gaming on a PlayStation 3 or 4, or perhaps an XBox 360. However, I made the plunge this morning and for my first game purchase of 2016 I decided to install Steam on my laptop and buy Elite. I'm sure I'll have hours of fun ahead if it's anything like the original game!

HPTS 2015

I've been so busy travelling to conferences and customer engagements that I haven't had a chance to write about my trip to HPTS 2015. I've written several times about previous trips to this workshop and how it's my favourite of them all, so won't repeat. The workshop had the usual high standard of presentations and locating it at Asilomar is always a great way to focus the mind and conversations.

Because of its highly technical nature of the workshop I always like to use this event to try out new presentations - I know the feedback I receive will be constructive and worth hearing. This time my submission was essentially about what I'd written earlier this year concerning the evolution of application servers (application containers) driven by immutability and operating system containers, such as Docker. And I threw in a smattering of microservices since the topic is obviously relevant and I figured Adrian would be there! My presentation was well received and the feedback clearly showed that many people at the event agreed with it.

One other positive thing to come from the workshop and my presentation was that my co-traveller and long time friend/colleague, Professor Shrivastava, saw the presentation for the first time at the event. He understood it and whilst much of what was said I and others take for granted, he believes that there are groups of people that would find it interesting enough that we should write a paper. Writing papers with Santosh is something I enjoy and it has been a very fruitful collaboration over the years, so I look forward to this!

I also want to thank James because it was our discussions after I started my initial entries on the evolution of application servers that helped to focus and clarify my thinking.

High Integrity Software 2015 conference

I was asked to give one of the keynotes at this year's High Integrity Software Conference and I have to say that I enjoyed the entire event. It's probably one of the best technical conferences that I've been to for a while and I've been thinking about why that was the case. I think it's partly due to the fact that it was a very focussed themed event with multiple tracks for just a small part (4 talks) of the day so everyone at the event was able to concentrate on that main theme. In many ways it was similar to how conferences and workshops were "back in the day", before many of them seemed to need to try to appeal to everyone with all of the latests hot topics at the time.

The other thing that appealed to me was that I was asked to give a talk I hadn't given before: dependability issues for open source software. The presentation is now available and it was nice to be forced to put into a presentation things I've taken for granted for so many years. The feedback from the audience was very positive and then we were straight into a panel session on open source, which was also well attended with lots of great questions. Definitely a conference I'll remember for a long time and one I hope to go back to at some point.

Finally there was one presentation that stuck in my mind. It was by Professor Philip Koopman and worth reading. There's a video of a similar presentation he did previously and despite the fact it's not great quality, I recommend watching it if you're at all interested in dependable software for mission critical environments.

Heisenberg's back!

A long time ago (longer than I care to remember), I made the analogy between Heisenberg's Uncertainty Principle and large-scale data consistency (weak/eventual consistency). It got reported by InfoQ too. Over the weekend I came across a paper from friend/colleague Pat Helland where he made a similar analogy, so I figured I'd mention it here. What's that they say about "great minds" ;) ?

The modern production stack

Over a year ago I wrote the first of what was supposed to be the start of a series of articles on how research our industry had been doing years (decades) ago was relevant today and even in use today, i.e., had moved from blue-sky research into reality. I never got round to updating it, despite several valiant attempts. However, thanks to James I got to see a nice article called Anatomy of a Modern Production Stack, I probably don't need to, or at least not as much as I'd expected. And before anyone points out, yes this stuff is very similar to what James, Rob and the team have been doing with Fabric8, OpenShift etc.

Proud ...

I have two sons. They both make me proud on a daily basis. I've been away for a week at Summit and whilst there my youngest son, who's not quite 13, did something brave that made me very proud of him. I can't go into what it was except to say that he did it through the medium of social media - that annoyed me slightly but it seems to be the way of things today for a certain generation. I've told him that what he did was brave and made me proud but I figured I would also put it here so he can refer to it in the future; it's the closest I can get to social media.

Kids and asynchronous messaging

We've heard a lot recently about asynchronous frameworks. Typically what these really do is prevent or discourage blocking (synchronous) calls; such calls may be made, e.g., making an RPC to a remote service, but the sender thread/process either doesn't block (so doesn't actually make the call directly), or gets back some kind of token/promise that it can later present (locally or back to the receiver) in order to get the response when it needs it. Of course there's a lot more to this which I'm deliberately glossing over, but the point I'm trying to make it that most of these frameworks don't use the term "asynchronous" in the way we might understand it elsewhere. The FLP problem is a reality yet not one with which they tend to be concerned.

It's strange how the mind wanders when you're doing other things because this was precisely the problem I started to think about whilst cooking dinner the other day. With a lot of people believing that "asynchronous" is the way to go (and in many cases they're not necessarily wrong) I'm more often than not unsure as to whether they really understand the full implications. And whilst cooking I was getting frustrated with my kids, both of whom I'd tried contacting by SMS and neither of whom had responded.

So my mind wandered and I made the not-so-huge leap: I'm trying to communicate with them asynchronously. I sent each of them an SMS and I had no idea of it had arrived. Of course I could have added the SMS-delivery ack request to the messages but that can be disabled by the receiver. Therefore, until and unless they responded, I had absolutely no way to know if they'd got the message and were ignoring me, or if it had gone missing en route (SMS isn't reliable, after all). I sent more SMS messaged but with no responses I was left in the same quandary from before.

Now of course I could ring them but if they don't answer then all that does is mean I've deposited a message (voice) into some semi-persistent queue that they may or may not look at later. Another option would be to find someone who knows them and talk with them, asking for messages to be forwarded. By increasing the number of people I ask I increase the chances that one of them will eventually get through and I may receive a response (gossip protocols). However, ultimately until and unless they or a proxy (friend) for them responded (or came home) I couldn't really know directly whether or not they'd received my messages and acted upon them. Asynchronous is a bitch!

To conclude the story - they came home, had received the messages and didn't believe a response had been needed. Dinner was very tasty though!