NetKernel News Volume 4 Issue 11

April 5th 2013

• Repository Updates
• NEW: Web Mapper
• On Potential Reality
• On Potential
• ROC Engineering Potential Reality
• Space Runtime

Repository Updates

Today we are releasing a new enterprise architectural component called the Web Mapper. The Web Mapper acts as a smart rewriting proxy for web content that allows web content to be easily translated from it's origin path, optionally on a different host, to a new path.

This functionality has two purposes. Firstly it allows content and applications to be designed and implemented without concern for their eventual hosting location and without a mechanism for translation built into the functionality. Traditionally this could be done with XRL/TRL substitutions into the appropriate content but this adds to the complexity. One specific use case that we have for this functionality is to host multiple versions of the same application and serve them on paths specified by the versions, i.e. /v1/* /v2/* /v3/*

Secondly the Web Mapper can act as a component in a general purpose web proxy allowing content to be seamlessly served or included from another host hiding the original host completely. This has uses in implementing mashups and other composite applications.

The Web Mapper is implemented as an overlay which takes web content from inside the overlay and maps it to a different path on the outside of the overlay. This overlay not only maps the resource identifiers exposed but actively translates the responses from various common web representations - XHTML, HTML, Javascript and CSS - by looking for embedded URIs and translating them as necessary to preserve the integrity of the web content. Only responses from SOURCE requests are translated and all other verbs are passed through transparently.

More details of the operation are included in the shipped documentation:

http://localhost:1060/book/view/book:mod:architecture/doc:mod:architecture:webMapper

Also there is a demo module available at:

http://resources.1060research.com/packages/2013/4/urn.org.netkernel.demo.webmapper-2013-4-5.jar

which shows how to host multiple versions of an app at different locations and a web proxy for the BBC News website - which works mostly bar a some javascript issues on a small number of pages.

Over the last few months we (really practically-speaking Tony) have been undergoing a technological acceleration. Our motivation has not been technology for technology's sake, but rather to objectively assess NetKernel's capabilities and to round out the tooling in order to reach another (perhaps ultimate?) level of ROC systems and solutions.

One of the fruits of this effort has already been released - the new L1 cache and the cache visualisation tools - which fundamentally enables vastly larger ROC solutions.

A powerful consequence is that the new cache infrastructure enables the cache heatmap view in the space explorer. A very simple to use tool that offers a new human-scale perspective of the emergent operational computational state of your ROC systems.

The cache release highlights another critical objective of our recent work - the technological push creates and/or completes the set of technical capabilities, but at the same time, is allowing higher-level simplifications to be introduced, either to the NK tools or to the structure and architecture of your ROC systems. However, there is an innate tension here.

From an engineering perspective "simplifying" something means achieving an objective with the least possible complexity: simplicity and elegance are the same thing.

The tension is that in ROC, we are pushing a new frontier, and therefore making things simpler (more elegant) doesn't necessarily mean that they are going to get closer to an existing developer's world-view or perception. This is often stated as: simplicity is not the same as easy.

Eventually in this article I want to introduce a new technology. It may be the simplest and most elegant endpoint we've ever produced - however, it is absolutely not easy. But it has vast, extraordinary, enormous potential.

So before I talk about it I want to set some context.

I've come to realise that there are a class of ROC adopters who are pre-wired to embrace the abstraction - sometimes they intuitively get it, more often they have a background in disciplines in which they've been exposed to concepts that prepare the ground (physical sciences, mechanical/electrical/chemical/nuclear engineering, process design, industrial systems). Sometimes there is a cultural framework that seems to allow a level of comfort with new concepts - for example, did you know there is a strong cluster of adopters in Netherlands and Belgium (but also Afrikaans speakers too) - something about the Dutch language perhaps, makes ROC a natural fit?

Sometimes, we see people who really want to get get it, but who have been taught a world view that is highly-tuned to the classical software world but has conceptual gaps that actively inhibit the transition. These people have the toughest ask - since they are presented with a new world and to make progress they have to relax their pre-learned expectations and either unlearn or create a mental-partition between empirical classical coding and lose coupled emergent composite systems.

I think the new tool I want to reveal is at the extreme end of this experience spectrum.

The key word I used two paragraphs ago is potential. If you've been exposed to the concept of potential in other disciplines I think you'll just get it. If not... I'm not so sure...

So to ease the transition, lets take a moment to talk about potential...

On Potential

What follows is somewhat allegorical. If you're a careful literal scientist you will see that at points I am deliberately stretching the classical use of potential to encompass non-deterministic probability distributions too. The key point is that potential, in all its forms, only leads to real effects when an extrinsic context is introduced.

When you wire a small light bulb to the positive and negative terminals of a battery the bulb lights up. The chemical potential of the separated electrolytes provides an electrical potential difference which causes a current to flow which causes the bulb element to heat up which causes light to be emitted. This simple electrical circuit is a system that turns potential into reality.

Our senses are so innate to our awareness, that we feel that they are providing direct contact to the "real world". We think of the parts that make up the flashlight as being an indivisible thing. The torch and the light it provides are real.

Its hard for us to get behind the curtain and envisage that our flashlight is a composition of parts, each with prepared properties, and a billion electrons all of which came together in just the right way to provide us with the real effect of the light going on when a certain event (a switch being thrown) occurred.

The things we can touch, smell, taste, see and hear are the most real to us. But the strange thing is, the more science has investigated natural phenomena the more it has discovered that the tangible "real" sensory world is really just a very small part of the overall picture.

The tangible physical phenomena we see all around us are in a way just the consequences of the hidden world of potential that surrounds us. Potential is the reality behind reality.

Before you glaze over with philosophical apathy - remember this is an important discussion about software - this might seem like metaphysical clap-trap but if you can keep with it I think I can show you that its not so difficult and is in fact fundamental to deeply understanding what we are doing when we create information systems (software).

In the physical sciences we get introduced to the concept of "potential". Often this is done indirectly and often it is without considering the implications. I want us to think about the implications - but first lets see how you might have been taught about one form of potential...

Potential Energy

You will probably have heard of Potential Energy. You might even remember that it is the energy of a mass (weight) when you lift it up off the floor. You might even remember that the Potential Energy of something is proportional to how high you lift it.

If you can't remember about Potential Energy, you still know all about it from your sensory experiences. If you've ever dropped a log on your toe - you'll know that it hurts. But from experience you will know it hurts much less if you drop it from one centimetre than from one metre. The dropped log had potential energy.

Like our flashlight system, the potential energy in the log gets converted to kinetic energy as it falls, which when it strikes your toe gets converted into deformation of bone, crunching sound, heat and a little bit of electrical potential energy as nerves fire to tell your brain not to be so bloody stupid next time. Potential got converted into a reality of pain.

In Physics class you might have been told an equation for Potential Energy (PE) like this:

PE = m g h

Where m is the mass, g is the constant acceleration due to gravity (9.8 m/s/s) and h is the height you raise the mass.

You might also have been told that Kinetic Energy (the energy of motion) (KE) of a mass is

KE = 1/2 m v^2

So we can work out how fast a log falling from 1 meter will hit our toe, by noting that when it hits our toe then all of its potential energy must have been converted to kinetic energy (assuming you're in the habit of carrying logs in a vacuum). So all of the kinetic energy is now equal to the potential energy before the log was dropped. Which we can say like this...

KE = PE

or

1/2 m v^2 = m g h

which (fiddling around with some basic algebra to equate to v) gives us...

v = ( 2 g h) ^ 0.5

Which is as Galileo said it should be - the speed that something falls is independent of its mass (there's no m in the velocity equation). The log you dropped at 1m will hit your toe at 4.4 m/s (or about 10 miles per hour) the one from 1cm will be a much slower 0.4 m/s (1 mph).

So now lets think about the stuff they don't tell you...

Potential is Abstract

In science classes they get us to practice with the equations and pretty soon we get used to dealing with the results of Potential Energy. We can work out useless stuff like how fast a log will hit our toes - we get to use the concept but only need to consider the effects it has on reality. We get to stick with our senses.

But meanwhile the elephant is left holding on to the balloon in a lift in a sky scraper (come on, its an elaborate elephant in the room metaphor).

What they don't tell you is this: You cannot measure potential energy.

Imagine you're in the lift with the elephant. Imagine the floor level indicator is not working. Imagine you have a bench full of state of the art science kit and a pair of scales. Imagine you get the elephant to stand on the scales...

So you know how much the elephant weighs (which is one property that affects its potential energy).

The lift goes up and stops. You don't know what floor you're on because the level-indicator is not working. So what is the potential energy of the elephant?

The simple fact is no experiment you can try will tell you the potential energy of the elephant in the lift. In the closed system the Potential Energy cannot be measured. It is an abstract property. It becomes real (is reified) only when the system is coupled to some other frame of reference - like the lift cable snaps because some idiot let an elephant in and you, the lift and the elephant start to convert your potential to kinetic energy.

So Potential Energy is abstract but it doesn't mean its not there. Its behind the curtain. Its reality behind the reality.

Its a shame they don't teach this stuff in science classes. Potential Energy is the energy of a mass due to its relative position in a gravitational field. It's abstract yes, and relative too. It cannot be locally measured but it can be inferred when it is converted to a physical effect.

Even so - we get familiar with it - we can work stuff out with it. We can understand it and we can build useful systems from it. For example for a long period of civilized human history we used to keep track of time with it. Water, and then pendulum clocks run on potential energy.

Even though its beyond our senses, we get used to it, we don't fret about it. But its actually not at all different to other concepts that seem to cause all sorts of difficulties...

A short list of immeasurable abstract properties that lie behind the curtain

Here's a list of other things you might or might not have heard of. They are behind the curtain. They are all a form of potential...

1. Electrostatic potential energy
2. Magnetic vector potential
3. Quantum mechanical wave function

They are useful things, you are using all of them right now just to read this...

I'll bet you have an LCD display? Electrostatic potential energy is what keeps the liquid crystal in place so that you can read these words. Its the electrostatic equivalent of gravitational potential energy. Imagine the elephant in a lift is a charged nylon elephant between some capacitor plates.

Magnetic vector potential is the fundamental reality behind magnetic fields and was required in the generation and distribution of the electricity you're using to read this. Its a key property of Maxwell's equations of electro-magnetism but for some reason doesn't get the attention of the next one on the list...

Quantum mechanical wavefunctions are what you and your laptop are made from. No-one has ever seen or measured a wavefunction because they are abstract probability distribution functions. A wavefunction describes the potential of where something might be. You can't measure a wavefunction - you can only make tangible measurements of a quantum system when it is coupled to something else.

I bet you've heard of Schrodinger's Cat and been told its a big and freaky mystery. But really there's no mystery - its conceptually no harder than the elephant in the lift. You can't measure abstract potential so in a closed system you cannot know anything about the state of the cat - you can only get information when you convert potential to a measurable effect.

OK I think we're ready now...

ROC Engineering Potential Reality

In ROC we create information solutions by constructing contextual arrangements of address spaces and endpoints. One of the reasons some people take to it like ducks to water and others like ducks to a microwave is that ROC's primary concern is not to execute code but to reify state.

When we arrange spaces and endpoints we are working in the domain of potential. We are structuring potential in such a way as when a real event occurs (in ROC we call this a request) the set of possible state collapses to a point of actual state (we call this a representation).

It turns out, this is an extremely elegant and efficient way to build and operate information systems.

One of the skills that working in the potential domain requires is an ability to live with and imagine the abstract relations of the structured potential.

This sounds rather grand - but it is not actually a particularly new way of thinking. It is, for example, exactly the same mindset that an electrical engineer has when devising a circuit, or a chemical process engineer has in devising an industrial chemical plant, or nuclear power plant engineer has when balancing the control rods...

In ROC, as with these other disciplines, we take hold of an infinity of potential and we constrain it and where appropriate use engineering approximation in order to yield a useful reality of operating information. We are engineering Potential Reality.

History

About 5 years ago we understood that the underlying implementation of NetKernel (the NKF API, the kernel abstraction and the primary elements of space, endpoint and request) had reached a level of "sufficient refinement". That is, we felt like the ROC abstraction had an implementation that was essentially complete.

You get confirmation of this feeling when you realise that certain aspects of the implementation of the abstraction are using the abstraction to achieve an efficient implementation (really!). For example the resolution process treats the result of resolution as resource representations. The bootstrap process uses operational representation state to leap over computationally recursive landmines (such as the dynamic import) so that the system organically anneals rather than mechanistically boots.

We also knew that there were classes of endpoint in which we were internally using the full power of the abstraction in order to create elegant engineering black boxes. One example is the virtual host endpoint we discussed recently. Others include the various space injecting bridge endpoints such as the HTTPBridge.

We also knew that the way that the Standard Module constructs the ROC domain from the declarative module.xml is in no-way a special case. It is just using the generalized low-level space elements to commission the logical ROC spaces from the declarative structure.

We recognized that we could generalize these instance examples - but we held back from doing this because, frankly, we were worried that it might be a step too far. Not in terms of technical challenge - but in terms of comprehensibility and familiarity to the classical developer.

I don't know maybe we're getting impatient. Maybe we're ready to go to the next level and we need to be brave enough not to be afraid to leave some people behind.

Anyway whatever the motivation, we have implemented the ultimate endpoint, we call it a "Space Runtime".

Space Runtime

A space runtime is similar to a language runtime but instead of executing code it "reifies space". That is it makes a resource request for a "space resource" declaration and instantiates it. It is extremely simple (as in elegant). Here's an example...

When it is commissioned the DynamicSpaceAccessor requests the resource specified by its <space> parameter. In this case it SOURCEs res:/resources/mySpace.xml and instantiates the space.

In the host rootspace the space runtime acts exactly equivalently to an an <import> - only it is not an import but a gateway endpoint into the newly reified space.

For example imagine that res:/resources/mySpace.xml has a very simple literal endpoint declared...

Notice it is using the Standard Module declarative syntax - so anything you can do in a module.xml you can do in a dynamic space.

Then if we make a request in the host space for "res:/helloWorld" - it would be resolved via the DynamicSpaceRuntime to the literal endpoint, just like an <import> would behave.

So far so good. Gently does it. Are you ready for the mind warp?.

The dynamic space is a representation that depends on the state of the declarative resource it was instantiated from. If you change the declaration then the space runtime will tear down the existing space and reinstantiate a new space.

The space is a resource. Everything is a resource.

What would happen if the space resource contained the complete structure of your application? Its a <space> like any other it can contain <mapper>, <accessor>, <fileset>, <overlay>, <import> endpoints... anything goes.

What would happen if instead of resolving the space resource to a static file, you configured it to a dynamically generated declaration - perhaps one generated from a DB query?

What would happen if the space resource declared a transport endpoint? What would happen if it was changed and the transport endpoint was removed?

What would happen if the <space> contained an endpoint and the system endpoint monitor told us it was becoming overloaded with requests? What about if we rewrote the spacial context of the endpoint so that it became wrapped with a throttle?

What would happen if you resolved the space resource over NKP to another NetKernel instance? Or over HTTP (if you're prepared the tolerate the temporal latency innate to HTTP caching)?

Are you ready for the ultimate mind warp?

What would happen if the dynamic space declared another dynamic space endpoint?

Yes, the dynamic space runtime is recursive...

Do you see where we've been leading to with the all the talk of potential? Welcome to the world of potential reality.