NetKernel News Volume 2 Issue 7

December 3rd 2010

What's new this week?

• Repository Updates
• NetKernel West 2011: Call for Papers
• The Butterfield Chronicles: Modelling Injected Spaces
• ROC and Languages: Arguments

Catch up on last week's news here

Repository Updates

The following updates are available on both the NKEE and NKSE repositories...

• demo-websockets 1.1.1
  • A new demo module showing how to use WebSockets (see last weeks news)
• http-server 2.1.1
  • HTTP Server updated to Jetty7 with support for WebSockets
• layer0 1.49.1
  • Support for exposing injected spaces (see below)
• layer1 1.22.1
  • Expose injected space for scratchpad
• module-standard 1.36.1
  • Update to branch-merge to support overlays that expose injected spaces
• nkse-control-panel 1.17.1
  • Fix to stop duplicate healthchecks in NKEE
• nkse-dev-tools 1.27.1
  • Better handling of malformed metadata from endpoints
• nkse-http-fulcrum-backend 2.1.1
  • Updated to use Jetty 7 configuration and WebSockets
• nkse-http-fulcrum-frontend 2.1.1
  • Updated to use Jetty 7 configuration and WebSockets
• nkse-visualizer 1.9.1
  • Support for links into space explorer for injected spaces
• system-core 0.21.1
  • Better handling of malformed metadata from endpoints

The following updates are available in the NKEE repository...

• nkee-dev-tools 0.17.1
  • Fix to login gatekeeper after internal space name changes
• nkee-http-server 2.1.1
  • Updated to Jetty7 with updated NetKernelAsyncHandler for asynchronous HTTP server patterns

NetKernel West 2011: Call for Papers

The NetKernel West 2011 conference will take place 13-14th April 2011 in Fort Collins, Colorado, USA. The conference will be preceded by a one-day NetKernel bootcamp on the 12th where you can get a fast track immersion in NK and ROC before the main event.

We want to make this an open opportunity for the NetKernel ROC community. We invite you to please let us know if you would like a slot for a talk/presentation/demo etc.

NetKernel West 2011
Location:	Fort Collins, Colorado, USA
Conference:	13-14th April 2011
NetKernel BootCamp:	12th April 2011

We're closing in on final details and will be announcing venue, accommodation and travel arrangements shortly.

Looking forward to meeting many of you face to face in April!

The Butterfield Chronicles : Modelling Injected Spaces

Tony has taken on the unenviable task of continuing to introduce metadata tweaks so that the dark matter of the ROC abstraction is illuminated in the new Space Explorer. Hence you'll find in this week's repository-updates numerous tweaks to modules to surface metadata.

Recently he's turned his attention to those endpoints which construct and inject dynamic spaces.

He's written a blog entry which discusses where this pattern occurs and also introduces an ROC visual language to represent these injected spaces...

http://durablescope.blogspot.com/2010/11/modelling-injected-spaces.html

You'll see the results of all this effort when you next use the Space Explorer.

ROC and Languages: Arguments

In the first article in this series I covered the basics of the ROC language runtime pattern and discussed how, by treating code as a resource, we get something we've called code state transfer.

In the second article I discussed the general nature of imports and surveyed a variety of languages detailing some typical models for accessing code resources referenced via internal import statements.

We saw that languages running in the ROC domain can have internal imports that, like XSLT's URIResolver, can transparently exploit the external ROC resource space and thereby gain dynamic modularity.

But further, we saw that thinking "outside the box" of the language we can make requests from within our code out to extrinsic functions. We discovered that with extrinsic functions the concept of an "import" also becomes extrinsic and is a property of the spacial architecture.

We found that by extending our view from thinking in terms of running code, to requesting normalized functions, we gain the ultimate in loose coupling - allowing the implementation of the function to be abstracted, to be scaled out across local physical computation resources and even to be externalized into the cloud.

Arguments

ROC allows us to adopt an extrinsic view of functions. But a function must be told what it is to operate upon. For that, a function needs arguments. As you'd expect given the trend we've established, arguments in ROC are also extrinsic...

Lets take these last few sentences at face value. Further along with the story we can return to this and explore the implicit underpinnings in more depth. For the moment, we'll examine the specifics of how NetKernel allows arguments to be supplied to a function.

First off, what follows is a discussion of how an NK/ROC endpoint embodying some function may receive arguments. We have stepped out of the plane of a specific language. In particular we will consider pass-by-reference and pass-by-value and these are specifically constrained to the ROC meaning.

As you know, NetKernel is implemented on the Java virtual machine, but in what follows we are not concerned with the specifics of the Java language. (For a quick recap of the Java model see this article.)

One more bit of preamble. We'll use pseudo code examples using the NKF API however everything discussed also holds for the declarative request and where necessary we'll illustrate the equivalent construction. This is an important point, since it means that the ideas discussed here are equally applicable in any of the higher-order declarative technologies of NK such as the mapper, XRL, DPML etc.

Scenario Recap

Recalling the configuration we've used in the earlier articles. Lets assume we have a requesting endpoint EP1. This endpoint wishes to invoke a language runtime active:lang with some code Code1. Lets also assume that we are going to supply an additional argument which will be used by the code. Lets call this argument operand.

So the active URI structure of EP1's requests will look something like this...

active:lang + operator@Code1 + operand@[....to be discussed below.....]

Arguments Have Names

The first observation is that NK arguments are named. This is different to Java where they are positional.

In our scenario we have two named arguments: operator and operand

The order in which you specify named arguments is not significant to the requested endpoint. However, the NKF core will always normalize a constructed resource identifier by sorting the arguments by name.

For example to the active:lang endpoint these two requests are equal...

1. active:lang + operator@Code1 + operand@xxx
2. active:lang + operand@xxx + operator@Code1

However, when you programmatically construct a request (NKF or declarative) it will always be sorted before it is issued. So, in the visualizer for example, you'd only ever see the second one (with operand ahead of operator).

By sorting by argument the infrastructure is taking care of you to ensure that you always have normalized identifiers which a) minimizes the representation cache size, or vice-versa, b) maximizes your cache hit rate.

Again, to make progress we will take named arguments as a given. We may get around to showing later that this is not a hard coded constraint but purely a convention of the higher-order Standard Module. To be absolutely ROC purist, an argument is just a unique subset of an opaque resource identifier token and, in fact arguments are a higher-order construct. The NetKernel kernel has no concept of arguments!

Pass By Reference

Lets imagine that EP1 knows the identifier of some resource res:/X. EP1 wants Code1 to process that resource. It could construct and issue a request like this...

//Inside EP1
req=context.createRequest("active:lang")
req.addArgument("operator", "Code1")
req.addArgument("operand","res:/X")

result=context.issueRequest(req)

Which would construct and issue the following SOURCE request (notice that we've accepted the default SOURCE verb for the createRequest(). Also note the sorting)...

SOURCE: active:lang + operand@res:/X + operator@Code1

You can see that the request construction has added the named arguments and each argument is a resource identifier. Therefore these arguments are pass-by-reference.

Evaluation

The request is resolved to the active:lang endpoint which sources and executes Code1 (as discussed at length in the first article).

So now lets assume that Code1 is written to perform some work for us on the operand argument. Code1 can obtain a representation of the operand argument by making a SOURCE request for it.

One way it can do this is to inspect the invoking request and to obtain from it the value of the named operand argument...

//Inside Code1
receivedRequest=context.getThisRequest()

operandIdentifier=receivedRequest.getArgumentValue("operand")
//At this point: operandIdentifier == "res:/X"

operandRepresentation=context.source(operandIdentifier)

When this code is executed in Code1, it SOURCEs a representation of the state of res:/X and can do whatever it is that Code1 does.

The constraint is that, just like a pointer in C or a reference in Java, the representation held by Code1 is not intrinsic to it. It is extrinsic and any number of other threads could also be accessing it.

Therefore you must take care to treat the representation as an immutable value object. Immutability ensures safe and scalable concurrency and is now regarded as a requirement for multi-core concurrency in the new generation of languages such as Clojure etc.

Representation Immutability is a strong rule. However there are circumstances where you as an architect are completely sure of the extrinsic context and can bend the immutability rule - but you are taking responsibility - if you change a representation value you have stepped beyond the range of NetKernel's safety net.

Dereferencing

By explicitly inspecting the request, the invoked endpoint gets comprehensive access to everything that is known about the incoming request. However this is generally not something that concerns us.

Mostly all we really care about is dereferencing the argument reference to get the representation.

The NKF API and declarative requests, allow us to do this by using the local arg: identifier scheme. This is easiest to see with an example. Here's some code that does exactly the same as in the previous explicit form...

//Inside Code1
operandRepresentation=context.source("arg:operand")

SOURCEing arg:operand dereferences the "operand" pass-by-reference argument and all of the baggage goes away.

Transreption: Type Decoupling

Of course as soon as we drop down inside the execution of the language we've chosen for Code1 we are intrinsically constrained to a given API. It follows that Code1's use of the the received argument will also conform to a given internally chosen API. Otherwise we'd have a type mismatch.

So, are we saying the origin endpoint for res:/X has to share the same type model as Code1? If so, then our journey to explore extrinsic decoupling becomes a lot more tightly constrained!

Fortunately, that's not the case...

EP1 doesn't need to care what the representational form (type) of res:/X is. The endpoint providing res:/X doesn't need to care what language or representational resource model someone requesting res:/X might use. And to complete the triangle, Code1 can be coded to its own internal representational model too.

Alchemy? Well yes, kind of. In ROC we call it transreption.

Let's imagine a concrete example, suppose that res:/X is a binary stream (maybe its really coming from a remote HTTP GET request or a file resource. Within Code1 we don't know about the mapping - its extrinsic!).

Here's all that Code1 needs to do if it wants to work with that resource as a String...

//Inside Code1
operandRepresentation=context.source("arg:operand", String.class)

We are explicitly stating that we require the argument "operand" as a String. We're also introducing an extrinsic type constraint. We're saying, if the extrinsic context can't get us the argument as a String then the architect needs to sort it out!

Usually an architect sorts it out by importing a resource model library that provides transreptors that impedance match representational types that are expected to be used. So SpaceA where EP1 lives, would probably import layer1 which has a bunch of basic type Transreptors for convenience.

Type conversion is extrinsic and not a concern of Code1. Nor is it a concern of the language runtime or it's library Space either.

So we now find that extrinsic functions are logically decoupled and type decoupled. Loooooooooooose coupling!

Pass By Value

Often you will have a situation where you have a local value object in the execution state of your code and it is this intrinsic state that you want to process with an extrinsic function.

For example suppose we have some Stuff in EP1 and we want to have Code1 process it? Here's how we'd do it...

//Inside EP1
localState=new Stuff();

req=context.createRequest("active:lang")
req.addArgument("operator", "Code1")
req.addArgumentByValue("operand",localState)

result=context.issueRequest(req)

Notice that we now explicitly indicate that the localState object is to be passed-by-value by using the addArgmentByValue() construct. The named operand argument is now pass-by-value.

If you're familiar with REST, you might be thinking this is analogous to a POST - there is some local state and we require it to be PUSH-State-Transfered to the remote endpoint.

Notice that we've not changed the verb, we're still constructing a SOURCE request. Surely this is a violation of all that is good and true. We are spitting in the face of Resource Orientation. We are defiling the holy temple... We are....

Not so fast REST-boy. Stand-back this calls for ROC-man...

If, like a POST, we were to place the pass-by-value state representation inside the request then, just like a POST, that state would leave the address space and become dark-state.

As any good REST-afarien can tell you, you know what happens with dark-state don't you? The representation that is computed by the extrinsic function is no longer unambiguous, it cannot be given to the next person that asks for it since we don't know what was in the hidden layer.

Writ large on the tablets of stone in every REST tutorial will be: You cannot cache a POST.

Bzzzzzzzzzzzzzzz. Wrong.

Or at least right, if like the Web, you only have a partially resource oriented abstraction. With ROC we have a general resource oriented abstraction of which REST and the Web are a subset - this changes everything.

In ROC we have space as a degree of freedom and, as we have seen throughout this series, we have extrinsic contextual relativity.

Let's see how this gives us a Get Out of Jail free card for pass-by-value arguments...

Space and Pull-State-Transfer

When you addArgumentByValue() NetKernel doesn't shove the value object inside the request like a POST body.

It uses ROC's new dimension.

It constructs a transient space and places the value-object as a resource in that pass-by-value space. It gives the resource an identifier too, a made-up unique identity, typically with the scheme pbv:.

The request construction is exactly the same as in the pass-by-reference, only now the pass-by-value argument has the resource identifier to the resource in the pbv: space.

When the request is issued, NetKernel injects the pass-by-value space into the request scope. You can see the spacial structure in the diagram on the left.

This code ...

//Inside EP1
localState=new Stuff();

req=context.createRequest("active:lang")
req.addArgument("operator", "Code1")
req.addArgumentByValue("operand",localState)

result=context.issueRequest(req)

...therefore causes the following SOURCE request to be issued...

SOURCE: active:lang + operand@pbv:XYZ + operator@Code1

Where, lets assume the localState object was placed into a PBV-space under the resource identifier pbv:XYZ.

Requested Endpoint

So what are the consequences for the requested endpoint? Answer: absolutely nothing, the endpoint has no interest in the nature of by-reference or by-value arguments.

With PBV-arguments the examples we showed of Code1 obtaining the operand argument are exactly the same. Just for detail, here's the first example if it was requested with the pbv:XYZ argument ...

//Inside Code1
receivedRequest=context.getThisRequest()

operandIdentifier=receivedRequest.getArgumentValue("operand")
//At this point: operandIdentifier == "pbv:XYZ"

operandRepresentation=context.source(operandIdentifier) //Resolved in the PBV-space

PBV Consequences

It should be clear that, to the ROC abstraction, pass-by-reference and pass-by-value are the same!

It follows that all the other properties hold true also. Transreption of the PBV state occurs automatically and as required by the needs of the requested endpoint. Type decoupling holds.

Furthermore, since the PBV is a true resource in a true resource space. Any resource computed based upon its value is cacheable. The extrinsic memoization of functions holds.

It may be that the PBV state has limited commonality for future requests. For example EP1 might compute new localState for every request it receives. In which case the result of EP1 will have no cacheable value.

However what if in the request below EP1, Code1 is a long lived process, making many requests, all of which rely on the state of the localState received from EP1? All of these micro-processes generate micro-resources all of which can be cached since the localState is static in the PBV space and extrinsic to them!

It might come as a surprise but there are many many real world problems where outer (apparently transient) state can be made extrinsic and allow for micro-caching and dramatic computational efficiency. What about dealing with external HTTP POST requests? NK/ROC will internally cache the intermediate state while we're processing them. You can cache inside a POST!

Pass By Request

There's one more trick up ROC's argument passing sleeve. We can also pass-by-request. Here's an example...

//Inside EP1
requestForResX=context.createRequest("res:/X")


req=context.createRequest("active:lang")
req.addArgument("operator", "Code1")
req.addArgumentByRequest("operand",requestForResX)

result=context.issueRequest(req)

Just as with addArgumentByValue(), NetKernel constructs a PBV-space and places the request object as the representational resource.

Now when Code1 sources its argument...

//Inside Code1
operandRepresentation=context.source("arg:operand", String.class)

..it causes the request to be resolved in the PBV-space. But this time because it was Pass-by-Request, then that request is itself issued and the result of that PBR-request is returned to Code1 !! This is functional lazy evaluation.

You can see that the benefit of PB-Request is that the requested resource is not reified until the very moment it is actually required: JIT state transfer.

Declarative Functional Programming

You may find its not so common that you need to use PB-request from the NKF API, but you might like to know that the declarative request supports this too.

So you can readily construct declarative functional programs like this...

Which is your classic f(g(x)) construction.

By-Request arguments can be nested to any depth. This can be very powerful in the mapper, XRL, DPML, XUnit tests etc etc etc.

State Transfer: Does anything move?

Yet again we've covered a lot of ground. I'll leave you with one last thought.

I'm sure you know that Einstein conceived of relativity theory by posing the child-like thought experiment. What would happen if I sat on a beam of light?.

I recall that when we were finding our feet with NetKernel and ROC, we asked ourselves a similar question...

What would happen if I sat on a request?

Its interesting that, like Einstein found, when you follow this question to its end you discover that its a relative opinion of whether state is transferred or whether state is where it is and space bends!

In real-world systems you invariably find that a large part of the time you hit the cache. In such situations you can confidently say that state is not moving (its in the cache), only the contextual ROC space was bent for the given request.

Part 4: What is Code? What is a Function?

Have a great weekend.