NetKernel News Volume 2 Issue 1

October 22nd 2010

What's new this week?

• Repository Updates
• Announce: NetKernel West 2011
• Tom's Practical NetKernel Book
• On State and Consistency
• ROCket Science: How to see Entropy

Catch up on last week's news here

Repository Updates

There are no package updates this week. Steady as she goes.

Announce: NetKernel West 2011 - Denver Area, April 2011

Its been two years since the last NetKernel conference, NK4 is finished and the NKEE release is out too. There are a ton of things to talk about. We don't have a precise space-time coordinate yet, but this much is certain: Denver Area USA, April 2011

NetKernel West 2011
Location:	Denver Area, USA
Time:	April 2011

The format will be a two-day NK/ROC Bootcamp followed by the two day Conference.

Wise old heads can just come to the Conference. An optional combo-package will include the pre-conference Bootcamp and provide a fast-track zero to hero immersion in NK/ROC out of which you'll emerge a fully armed and dangerous ROCket scientist.

If you came to Fredericksburg VA last time, we can't guarantee that Barack Obama* will be able to make it again this time, but nevertheless it will be the usual mix of informative, practical, mind-expanding and entertaining content**.

If you didn't make either of our previous two conferences then you missed out. We make sure there is a high signal-to-noise ratio and encourage interactivity. Evenings come with a good-time guarantee (it has been known for beverages to be consumed).

We'll be working on the details, location, exact dates and will firm these up as soon as we can. There will be a maximum ceiling on places, so we'll make sure you get first access with a pre-general notice heads-up here (in the NK news) as soon as registration opens.

Why Denver? Where else would a bunch of ROCies meet?

Denver skyline with Rockies, by Lewis Shepherd, linked from Flickr.

**Speakers/Topics: We welcome submissions for talks if you'd like to speak.

*Yes, true story, Barack Obama really did take time out of the 2008 presidential campaign to attend the last NK conference. That's how unmissable it is.

Tom's Practical NetKernel Book

Tom Geudens has made good progress and released an update to his 'Practical NetKernel' book...

hello_netkernel_nk4_0.3.pdf

He'd welcome feedback on progress so far. He can be contacted at tom (dot) geudens "at" hush {dot} ai

On State and Consistency

Its true to say that the off-the-shelf component libraries we provide with NK are stateless. Therefore when assembling a composition (a process, a pipeline) the overall flow through tools is transient - state starts on the boundary of the system flows through, stuff gets done, responses return and the endpoints resume their initial condition.

That's the general picture. But of course as we run the system we accumulate representational state at various natural boundaries in the solution. In NetKernel all resource requests are potentially cacheable (in comp-sci terms, NK has systemmic memoization). Quite naturally representational state percolates upwards towards the client-side.

In most cases an application requests external state from persistence mechanisms (DB's etc). Importantly however, any given application is usually about building composite resources from the "atoms" of this external "normalized state". The Golden Thread pattern allows externally sourced state and the derivative composite state higher in the upper parts of your application to be kept consistent as state changes occur.

If you've not heard of the GoldenThread pattern then there's a demo you can install demo-goldenthread with Apposite . It includes embedded video presentations that discuss the details and which you can view here (part1) and here (part2).

You could debate that one definition of an "application" is to provide a contextual relation between one set of "denormalized" state (the client-side surfaces) and one (or more) sets of "normalized" state (business state). Security is then the constraint of the denormalized state such that the application channel can only reach that subset of the normal state that is deemed "suitable". This is why in the 3C's of ROC, security/validation are part of the 3rd C: Constraint, which can be applied retrospectively and orthogonally to an application architecture.

Returning to the general picture. As I said, while the off-the-shelf services we provide are stateless, there is actually nothing in the abstraction which mandates stateless endpoints. Its just that in compositional architectures statelessness means that systems remain consistent without a developer/architect needing to give it any thought.

However, imagine we have a Geiger counter endpoint: active:geigerCounter - which records external gamma ray events. This endpoint would be stateful. Each request to this endpoint would return the current event count.

But how do we enforce consistency now? Well we ensure that any representation that we return is only cacheable for as long as the internal count state is unchanged. We can associate a dependency to all representations that expires as soon as the state changes (we can do this with an expiry function on the response). active:geigerCounter is stateful but it is implemented to ensure system consistency.

Now lets think about a random number generator: active:random - this service is stateful in the sense that it has a random number generator whose operating state guarantees that it will produce random numbers and is not going to return the initial seed value on each request. For system consistency we would make this tool return a response that was always expired. Therefore no matter how this random resource gets composed up into composite state above, the dependent state will be "on a branch" that can never be reached by anyone else ever again.

We can play games with the engineering balance of the state in the system too.

We can introduce approximate consistency. For example we can use time as a consistency approximation. We can say things like, this resource is valid between t=0 and t=x and don't come back to me before x. For example, with the Geiger endpoint we could decide that its good enough that we know the count to within a tolerance of 10 seconds. We'd do this by setting the expiry on the INKFResponse...

response.setExpiry(xxxxxx, T)

Where xxxxxx is one of...

So, by playing with time you can introduce Nyquist sampling criterion to your application's architecture.

We can also set our own expiry functions too. For example, its likely that a timed expiry on a Geiger counter is a bit foolish (especially if you're monitoring a nuclear power plant) instead it would make more sense to have an expiry function that expires every 100 hits - no timing necessary!

I'd classify this sort of pattern as "windowing" state expiration, we're sub-sampling the rapidly moving state to give transient consistency over a window of stability. Again, this is where Nyquist criterion become part of your architectural armoury.

One thing to make clear is that a user-defined expiry function does not require that you implement an internal thread to monitor an endpoint's internal state and manage consistency. An expiry function is only invoked if a given representation is in the cache and could possibly satisfy a request. In this case, the thread of the requestor that wants the resource, is used to call into the endpoint expiry function. In fact this is a general principle, state consistency is not pre-emptively managed, only if/when a resource is requested is the consistency of any possible pre-computed representational state determined ("just-in-time consistency").

In some instances we might actually want to have direct accesss to the rapidly changing stateful endpoints, like the raw Geiger counter endpoint. But nothing prevents us introducing wrapped smoothing services, active:windowingGeigierCounter, which would apply the windowing expiry over the rapidly moving source. We can build up assemblies of faster and slower changing statefulness. Of course, you see that we can do the same against external data sources too.

By now, you'll recognize that nothing in the abstraction prevents you writing an endpoint that is more like a full stateful client (eg like a browser, an orchestration workflow tool). A simple example in the off-the-shelf tools, is the Cron transport - this allows you to register a "space-time request" which is held statefully in the cron transport until at the given point in time your request is issued to the given space.

Big Picture

Hopefully its clear from the discussion that with ROC you have the freedom to introduce engineering tolerances in order manage "good-enough" system state. For any given application, there is always some critereon below which a fair approximation is good enough to maintain consistency with reality.

Why does this matter? Because data and reality are not the same thing. Data is always an approximation and being able to make intelligent choices about these approximations leads to high quality, robust, tolerant, adaptable and efficient (fast) information systems.

You are one such system. Cinema and TV are possible only because your image processing system is based on an approximately consistent representation of the photons arriving at your eyeball. Your Nyquest sampling rate is about 25Hz. That's good enough to detect a lion starting to run towards you - but its too slow to see the shockwave of a bullet or the individual frames of Avatar.

With an ROC solution, accumulated operating representational state can have approximate fidelity with reality. But critically, within an ROC solution a requestor is given a guarantee that in a given context they may issue a request and receive state and that state will be systemmically consistent.

System state consistency is assured irrespective of if the system has just booted or has started to find cacheable information, or the state is "moving" within a given resource set. The consistency is contextual - its not the case that the system has to be globally consistent, only consistent such that for a given spacial context two equivalent requests will receive the same resource state (all other things being equal).

The ROC abstraction embodies an internally consistent contextual computational state space - which, for any given application, naturally discovers the minimal transient state (min E, energy) and, through transreption, the most useful form (min S , entropy).

Entropy is a really important concept. When you get a feel for it, NK and ROC gives you an environment in which you can engineer it. You maybe never thought about it but Transreption, the isomorphic restructuring of one representation state to another, is actually formally entropy transformation. Information is conserved, entropy is reduced. As with any other request, transrepting entropy reductions are cached, so NetKernel discoveres the most valuable state in the most valuable structural form (lowest entropy).

There's lots of hype in the market, but for truly green computing you must start with the thermodynamics of your information. ROC enables green software.

ROCket Science: How to see Entropy

Here's a one minute experiment you can do with the kids* (or your boss) to introduce them to the concept of entropy...

Take one rubber (elastic) band. Loop one end over a hook and hang a decent weight on the band so that it is stretched taught. Get one hairdryer. Apply the hairdryer on medium to high heat to the stretched band. Which way does the weight move? (Take care, too hot and the rubber band will give out and you'll drop the weight on your (or your boss's) foot)

[Click for Spoiler]

-----------

-^==-^=

Have a great weekend.