NetKernel News Volume 3 Issue 42

October 19th 2012

What's new this week?

• Repository Updates
• Resource Oriented Analysis and Design - Part 9
• Resources are Abstract, Representations are Concrete
• State Transfer - Keep Transformations and State Separate
• Sets - Multi-valued Representations
• Pull not Push
• Composition Composition Composition
• Normalization
• Process
• Was the TTT Exercise Realistic?

Catch up on last week's news here

Repository Updates

The following update is available in the NKEE and NKSE repositories...

• kernel 1.31.1
  • Enhancement to allow user-constructed Exception representations to be returned as response and cached.

Resource Oriented Analysis and Design - Part 9

Last in a series of articles taking a well understood domain problem and casting it as an ROC solution.

• Part 1, Part 2, Part 3, Part 4
• Part 5, Part 6, Part 7, Part 8

OK, this really is the last in the series. I don't know about you but if I ever have to think about TTT again for the rest of my life it'll be too soon.

So with this in mind, lets see if we can draw out the major themes which have been woven into the details of the story so far...

Resources are Abstract, Representations are Concrete

If there is one thing to take away from this exercise it is to appreciate the distinction between a resource and a representation of a resource.

We saw through the analysis and design phase that distancing ourselves from the "baggage of representations" allows us to identify the sets of information that will inform our solution without being forced into decisions about technological delivery of the physical solution.

We don't start with code or even language. We start with identity and allow this to connect us to the concept behind the resource abstraction.

That said, we should never be "fixated by identity".

Thinking of good names is often difficult - especially when we are formulating a design. We need to be prepared to modify the names we give to things so that step-by-step we refine our system to something that can have long term stability and have identities (within the domain of expertise) that have simple and unambiguous semantics.

What I'm trying to say is - we should not be afraid to say "we don't know the answer yet".

We should be ready to make a start, knowing that it will almost certainly be revised as we learn more.

Software is always an approximation to reality - the insight I hope we're discovering is that the process of ROC software is to allow progressive and continual enhancements to the approximation - until some level of sufficiency is attained. The map maker who succeeded in making a 1:1 scale map is yet to be born.

Fundamentally ROC is a two-phase model for computation. The logical world of the resources is dealt with in the first phase - in which resolution of resource identifiers is performed. When a resource request is resolved it then enters the second phase of execution.

It follows that we can model and remodel the logical resource domain at whim and indefinitely - without having the physical domain impose itself on our imaginations.

State Transfer - Keep Transformations and State Separate

It should also be apparent that we have carefully maintained a distinct separation between representation state and transformational operations on state.

In ROC, state is transferred to code, which acts upon it producing new representations.

We should never presume that a resource will only be used in one way. We should assume a resource will be used in many new and unanticipated ways as new requirements emerge.

At first separating state and transformation (code, methods, functions) might seem quite strange - especially as it is the antithesis of the object-oriented paradigm.

But there are three overwhelming arguments why we should do this.

1. Identity - every resource has an identity so every computation can be cached. It is not the job of software to keep CPUs warm, its job is purely to deliver information. Nobody cares if you actually run any code at all. If we already have the desired information (whether pre-configured, cached or by transcendental meditation) then the transformation should be avoided entirely.
2. Immutability - extrinsic state that is transferred to the code can be accessed concurrently. The information is free - so anyone can use it anytime. The only thing that is needed is to ensure that the state is immutable. State transfer does not modify the state- it simply accesses it (hence "Accessor") and uses it to create new state. Therefore, with the ROC paradigm immutability is far easier to attain.
3. Re-usability - the elephant in the room of Object Oriented design is re-usability. With the best intentions we seek to create reusable solutions but honestly can you tell me hand on heart that you actually achieve this? The reason OO is fundamentally not reusable is back to part one: "we don't know the answer yet". In fact, we never know the answer yet! The world changes all the time - software must recognise that it is "never forever". By keeping code and state separate we allow ourselves the freedom to combine the state in new unanticipated ways. We decouple the evolutionary trajectories of the logical and physical domains. In the logical resource domain we have infinite elbow room - we can introduce new resources without disturbing existing resources. Legacy resources can be used indefinitely and be combined with other resources as necessary. In ROC re-usability is a fact.

Sets - Multi-valued Representations

The last section implied that if resources are sets of information then its very powerful to do collective operations on multiple-values simultaneously.

Now, there is a caveat. State transfer allows for re-usability, provided you use a flexible data structure for your representations.

As we saw in this series we used Tree structured data by default. It turns out that The Tree is a really great structure for state transfer.

To understand why, I wrote a detailed discussion, earlier this year...

ROC Data Structures

Pull not Push

You might have noticed that there were several points in the series where we created composite resources with references to other resources. So called "Linked Data" resources.

You may also have noticed that at each point where we constructed the linked data resource we actually would have had the opportunity to SOURCE the resource and combined and returned the state directly.

Essentially what's happening here is that we are imposing an architectural preference for "Pull-State-Transfer".

By which I mean - we are deferring requesting state until the absolute very last moment.

If we can get away with saying "its over there when you need it" instead of "here it is use it now" then this is a "better solution".

Why? Because in the time it takes for the state to be finally requested - someone might have already computed it for us so we might not have to do it ourselves. Furthermore, if our use of the state has any logic involved - its possible that the logic branch we go down may not even need the state anyway - in which case we wasted our time.

Finally, by delegating to pull - we can always introduce architectural intermediary layers that provide context. Which (although it may have been hard going) we discussed last week in the concept of how to contextualize our solutions. In short, asking for it too soon, may prevent you from exploiting the power of contextual architecture.

A preference for pull should be no surprise. Look at HTML web-rendering. We don't embed the image state in the HTML page with the CSS and the scripts. The browser defers state transfer until it is requested by secondary pull requests as the HTML parser executes.

Composition Composition Composition

Putting all of the above together - it is almost a matter of inevitability that when working in ROC our time is mostly spent in composition.

I was deliberately somewhat extreme in trying to deliver an entirely compositional solution to the TTT. But its a fact that a team working on an ROC solution will spend 80% of its time in composition.

Normalization

With our presumption of "not knowing the answer yet". Its inevitable that we'll create bad and convoluted names for things. However as we start to understand the significance and conceptual value of resources then its a good thing to take the time to normalize their identity. To give things stable names with semantic integrity.

We saw that we used the mapper repeatedly to normalize and solidify the naming of our resources. Not for any technical reason but for conceptual simplicity.

Process

I've said this repeatedly but a scale invariant recipe for solving problems is to follow the four steps.

1. What is it that I'm promising?
2. What do I need?
3. Add value
4. Respond.

These steps work for an individual endpoint, a space and an entire architecture.

Was the TTT Exercise Realistic?

In many ways the TTT example was contrived and a pretty uninteresting problem. It was certainly not realistic in the sense that the state of the game was incredibly simple unlike the state of a business held in RDBMS systems-of-record.

Yet the approach and the emergent solution, the natural layering, the linked data resources, the compositional efforts, the retrospective and selective application of constraints are all 100% typical characteristics of solving real world problems with ROC.

If you've not built anything complex with NK and ROC yet then this has to be taken on trust. But here's something that might provide some scientific validation. This spreadsheet...

https://docs.google.com/spreadsheet/ccc?key=0AjePFFtc08iCdGFHV3pwRTNpWk5mRzB6YlpYdFU2SHc

... is an analysis of the NetKernel Portal.

The portal was developed using exactly the same approach as TTT. You'll see how many endpoints it has, how many lines of code, lines of declarative resource, use of accessors, DB tables, etc etc.

The headline is that a complete complex portal architecture was built by one person at an average rate of 45 minutes per channel (I class the entire TTT demo as equivalent to "one channel" and it took about an hour altogether - so it shows I'm getting old).

The portal has been in production for three years and has not required any bug fixes and during that time has been continually enhanced with new tools and services.