NetKernel News Volume 2 Issue 10

December 24th 2010

What's new this week?

• Repository Updates
• Holiday Support Cover
• ROC: State of the Union

Catch up on last week's news here

Repository Updates

There are no updates this week. Steady as you are.

Holiday Support Cover

For those of you with a comprehensive production support contract we will be here to deal with any critical issues over the holiday period. If you get into trouble and need us, we recommend you raise your support issue through the NK support portal so that we can triage and assign one of the team.

General support cover will resume on Tuesday 4th January 2011.

ROC: State of the Union

Given a certain information problem you have two choices:

1. buy a turn-key product, a solution that exactly satisfies your requirements, or
2. use a framework to build a solution yourself.

When I started seriously trying to do information engineering in the mid-90's I looked hard at the standard model (turn-key or framework). Whichever way you looked, I didn't see what I wanted. Either course felt like it had built-in limits, hard edges that when I reached them would leave me exposed to innate brittleness.

In my gut, I felt like the problems that had turn-key solutions and/or frameworks were small scale instances. There was something beyond them. Something more general and more powerful.

To use a horrible clichéd metaphore: information engineering is like bridge-building.

Roman Arch: State of the art for ~2000 years.

You can cross streams and smallish rivers with a wood scaffold and stone blocks to put up a Roman arch. But to bridge estuaries you need engineering models, cantilever moments, reinforced concrete stantions, suspension, box-girders; you need engineered architecture. And fundamentally you need insight and grasp of first principles.

World's First Iron Bridge - 1775: Iron Bridge, Shropshire, England

You could say that frameworks have advanced - they are more flexible. But to me there's a historical precedent...

It seems like frameworks today are analogous to when bridge construction first shifted from stone to wrought-iron. They had a new material, but they carried on thinking the same way. Look carefully at the picture on the left - yep, the world's first iron bridge was a Roman arch!

Pushing the metaphore, the history of engineering shows us its not sufficient just to change materials (languages, object frameworks, dependency injection...)

To make real progress you have to step away from the confines of the materials, to view the problem in terms of core principals. This is why I used the recent series of articles to show a progression from the novel, but familiar, language runtime pattern to sets and functions.

In ROC, first we consider the resources (fundamental problem) then we choose a language (materials: stone or concrete, steel or carbon fibre). Ideally we arrange it so that construction comes down to composing a solution from customized pre-fabricated units (box-girders).

The most beautiful bridge in the world - 2010: Millau Viaduct, France (notice lack of arches)

As I say ad nauseum: ROC separates architecture from "code". What I mean is it is a technological foundation that allows us to treat the first principles of the engineering problem, independently from the selection of materials and the mechanics of construction (languages, code and coding).

I hope that this year's newsletter articles have begun to shed light on what then happens...

Just as when the iron bridge builders began to think differently, so in ROC, by stepping out of languages, the engineered information solutions you can achieve can be breathtaking and beautiful.

Reality not Metaphore

This is a pretty limited metaphore. Information engineering isn't like bridge-building. You don't (and can't) always know in advance what the problem is - unlike bridge building, information engineering doesn't (and cannot) have tight specifications. Frameworks will help you build bridges but often in the real-world it transpires that the problem wasn't about bridges at all.

Ask yourself why large-scale government IT projects fail so often? I would argue the primary problem is one of ongoing change of requirements and partial specifications coupled to conventional software.

Information engineering has to be different to mechanical engineering. It demands an assumption of malleablilty. Rigidity and specification compliance must be introduced as extrinsic constraint.

Even here, there are lessons from physical engineering. Every mechanical/electrical engineer is trained to incorporate tolerance in a system (the famous: engineering factor of two) - conventional software is strongly typed and tightly bound. Rigidity is baked into the foundation, there's little room for tolerance.

And yet we instinctively know that to solve problems that you can't specify and/or will be required to adapt to future change, you have to look to emergently complex solutions. Biology is just such an information engineering problem...

You can think of biology as the information problem of sampling the environment with a sufficient Nyquist critereon, that the computational energy cost of information capture is less than the energy source (food) that is acquired. With the additional constraint that there is an energy surplus in order to avoid predators and to propagate the system's genetic information.

The diversity of nature readily shows that large scale sophisticated solutions are achieved by progressive and incremental change.

To be discussed

This years technical articles have attempted to introduce, highlight and contextualize a bunch of the ideas that underpin ROC. I've been trying to gradually show that you can use NK with a "framework-mindset", but you can also, step away from the details and think about the first principals of the information engineering architecture.

You can just get things done with NK if that's all you need - but I think having an ROC perpsective is the way to get to the next level.

After last week's article I received some really detailed feedback from Jeff Rogers which I want to incorporate in an article soon. There is still a ton of things I want to discuss and I'm aware there are several topics that are still only partially covered...

• Thinking and modeling in terms of sets of resources.
• Choosing Representation models - thinking of representations as sets.
• Constraint and confidence boundaries.

Our possible futures

This state of the ROC-union address is at a point in time where we're between the past (defining ROC, making it real with a hard-core implementation) and the future (the consequences and opportunities engendered).

What are our possible futures? I think you can see a number of orthogonal axes...

1. End-user solutions
   • I showed that the Code state transfer pattern has a trend line: going from Turing complete languages to configuration state to constrained resource sets. This line doesn't stop. It can be projected forward to a point where it is safe to give an end-user (a call center operator, an account manager etc etc) a palette of click-fit tools and they can do their own "programming".
   • This isn't a vision, we've been progressively putting in place the core infrastructure for this over the last year and a bit. (Remember all those - "we added metadata", updates).
   • We are bringing this together now and plan to have some really exciting stuff for you by the time of the NetKernel conference in April.
2. Space Runtimes
   • You learn the runtime pattern by coming at it from a classical scripting point of view. But you saw that the pattern holds even if the code-state transfer isn't "Turing code".
   • The next step for this is to have a "Space Runtime" - that is a runtime which receives a declarative spacial architecture definition and instantiates and "executes" the spaces.
   • Surprisingly perhaps, we're actually doing this pattern inside a number of the existing tools that come with NK; but we're doing it down on the metal. The next step is to abstract it and to provide a declarative specification language.
3. Multi-dimensional "General Web".
   • This year we snook out the NetKernel Protocol (NKP). It allows NetKernel's general ROC abstraction to span servers.
   • As you saw last week, ROC is a generalisation of the Web.
   • With NKP it is valid to think of a multi-dimensional distributed Web address space. The "General Web". (No-one can accuse me of not thinking big).
4. Extrinsic Data
   • I've been showing how ROC is innately about inverting the intrinsic nature of classical monolithic software. To have an extrinsic view of functions and services and the computational state.
   • This trend doesn't have to stop. A natural progression is for the internal resource state to be exposed externally to the outside world.
   • This is where the "Semantic Web" is coming from.
   • I don't necessarily see RDF as the only representation model. Extrinsic data can be expressed in any suitable form: CSV, XML, HTML, JSON, SQL...
   • However I do recognise that the principal of extrinsic data is inevitable. (I think the next topic also makes it more likely too...)
5. Contextual Confidence - Trust Boundaries
   • With ROC you have direct control of the context within which information is accessed.
   • Control of context is what traditionally gets lumped under the label "security".
   • I don't think we've tapped the surface of the implications of ROC on defining trust and confidence boundaries in information systems.
   • This is stuff that really does need you to step away from the code and think about the information resources and context.
6. Malleable Discoverable Comptutional Systems
   • ROC is a dynamic computational system.
   • You can see, in for example, the dynamic import pattern, dynamic state may be used to define spacial architecture
   • The natural progression of this is to have state that enables higher order system capabilities to be emergently created.
   • I can see that the core pieces are already in place for patterns of emergent, "self-generated" software...

Thanks for taking the time to follow my ramblings this year. The future is out there - all we have to do is reify it...

Have a great holiday, see you in the New Year.

NetKernel West 2011: Call for Papers

The NetKernel West 2011 conference will take place 13-14th April 2011 in Fort Collins, Colorado, USA. The conference will be preceded by a one-day NetKernel bootcamp on the 12th where you can get a fast track immersion in NK and ROC before the main event.

We want to make this an open opportunity for the NetKernel ROC community. We invite you to please let us know if you would like a slot for a talk/presentation/demo etc.

NetKernel West 2011
Location:	Fort Collins, Colorado, USA
Conference:	13-14th April 2011
NetKernel BootCamp:	12th April 2011

We've now confirmed the conference venue and will be opening registration very soon.

Looking forward to meeting many of you face to face in April!

Happy Christmas and a Successful New Year