NetKernel News Volume 7 Issue 6

November 18th 2016

• Repository Updates
• On Continuous Evolution / NetKernel 6.1.1 Updates
• Golden_Thread Listeners / The Recalculating Golden ThreadPattern
• Event: Talk in Manchester UK 23rd November

Repository Updates

Just a quick note regarding ongoing updates. By its nature ROC allows for progressive evolution. This is no more true than in NetKernel itself.

All of NetKernel (aside from the very small core of the microkernel) is implemented in ROC as a federated resource oriented architecture. It follows that the change from 5.2.1 to 6.1.1 is just a selective enhancement of a collection of independent sets of resources (sometimes known in other parts as microservices). In fact the changes could have just been released as regular Apposite updates through the 5.2.1 repository.

However the reason to go with a new version is that it is occasionally important to define a stable point in the evolution. In essence so that we can all collectively agree on the same operating set of features and capabilities of the platform.

This is a what a distinct version gives and is reflected with a new distribution image and a distinct version of the associated Apposite repositories.

NetKernel Enterprise Edition is designed to seamlessly move through this arbitrary marker. It detects a signed repository redirection in the 5.2.1 repository, it waits until all pending 5.2.1 repository updates have been applied, once it is running with the most recent 5.2.1 repository versions, it then follows the redirection and seamlessly switches to the 6.1.1 repository where the updates that bring it level to the new 6.1.1 marker are presented.

If for whatever reason you move forward with Apposite updates to 6.1.1, and want to go back then, as ever, you can just rollback the updates and revert your system. So you can go forward and backward between 6.1.1 and 5.2.1 as many times as you want. If you do do this though, you'll have to manually change the Apposite repository path to point to the relevant past version.

Lessons

Are there any general lessons we can learn from how we manage the evolution of NetKernel? Well yes, since NetKernel is a federated collection of thousands of independent logical resources. In "microservices speak" it has hundreds of containers (modules/spaces) each with hundreds of logical and physical endpoints.

The whole amorphous collection is decoupled and free to continuously evolve independently (just like the Web has done ever since the first page).

For any system supporting continuous evolution the one thing that is essential, for everyone's sanity, is to only ever have one timeline for change.

It is essential that no matter where you are in your adoption of the evolution you can only move forward or backward along one time line.

Now don't be confused, I'm not saying evolution has to be linear. Clearly any subsystem must be able to evolve independently. No, by "timeline for change", what I mean is that there is a single point of reference for the collective evolutionary change.

It is for this reason that from now on, all ongoing updates will be in the 6.1.1 repositories. 6.1.1 is now the single timeline (point of reference) for forward change.

If you are implementing federated microservice solutions then maintaining a single timeline for change is by far the most likely route to success.

With the recent update to layer1 we introduced a further refinement: the ability to associate requests with a given Golden Thread resource and for that request to be issued when the GT is cut. Thanks to Richard Perris, of an unnameable tier-one bank, for providing the insight that catalysed adding these features.

The Golden Thread pattern, of associating and controlling virtual dependencies, is a natural consequence of the underlying dependency caching innate to NetKernel.

To see the different ways in which a composite representation can be expired by its physical dependencies, please click on the image below (stand back, Pixar's got nothing on this baby)...

The Golden Thread pattern is simply the association of virtual resources as dependencies - shown below as the nasty yellowish GT-1 virtual resource associated with D-2-1 and D-2-3. Click the diagram to see what happens when we cut GT-1

The diagram above shows the expiration cycle. However what is not shown (when the animation loops) is how the primary Representation is recomputed. To date, the Golden Thread pattern relied upon external events to cause a new request for the composite Representation and so recalculating the dependent expired composites.

This Just-in-Time approach to recomposition is extremely efficient from a system perspective since the cost of obtaining the composite representation state is incurred only when the state is absolutely required and so the state of the representation is as fresh and relevant as is possible. Furthermore in the interim the total system computational state space has been free to expend effort on areas that are immediately in demand.

When the total computational complexity of calculating the composite is relatively small and/or low-latency, the JIT-model offers the optimal net solution in a dynamic system.

However, Richard presented a scenario in which three other considerations are present:

1. The composite representation is computationally costly and/or has significant latency to its computation.
2. The composite representation expiration time period is longer than the computation period.
3. The demand for the composite is discontinuous with the events that trigger the expiration - meaning JIT happens a long distance in the future after the expiration event.

In such scenarios the JIT recomputation model incurs a latency in accessing the composite representation for the first re-requestor. A latency that could be eliminated completely if instead a pre-emptive recomputation of the composite were initiated when it expired.

All of which sounds very complex but really is very simple: Sometimes its a good thing to pre-emptively start recalculating a representation as soon as you know that it is expired.

Recalculating Golden Thread Pattern

Shown below is an example using a Golden Thread Listener to trigger the immediate recalculation of the composite resource. Click the diagram to see what happens now when we cut GT-1

The moment GT-1 is expired, the GT-1 Listener's request is issued asynchronously. In this case it is for the Representation, which causes all the composite's to be recalculated and so the GT-1 golden thread is reattached.

How is this actually done. Well all we need to do is register a request associated with one, or more golden threads. We do this with active:addGoldenThreadCutListener.

The golden threads the listener is associated with can match a regex. Listener must also provide a declarative request - which is invoked every time a matching golden thread expires.

The usual way to register a listener is to set up an endpoint that makes this request when its postCommission() lifecycle is triggered. So the listener is automatically registered when your application space is commissioned. It can also deregister the listener when its preDeCommission() lifecycle event happens.

Of course you can add and remove listeners any way you like to suit your application's semantics.

GTListeners are powerful since the listening and golden thread attachment are completely decoupled - a listener can be added before any golden threads even exist.

So, as we've discussed, the Golden Thread Listeners enables a pre-emptive Recalculating Golden Thread Pattern to optimise for Richard's scenario. However this is only one possible pattern. You might instead want to know about an expiration so that you can notify an external system, or just to do some asynchronous logging, or who knows...

Last week I gave a new "Resource Oriented Microservices" talk at the MuCon 2016 conference in London. Based on the feedback I received I think it went very well.

A very well edited video of the talk is available here

I'll be in Manchester, UK next week to give a similar talk. If you're in the area on Wednesday evening please come along; there's sure to be beer afterwards...

Have a great weekend