NetKernel News Volume 2 Issue 8

December 10th 2010

What's new this week?

• Repository Updates
• *New* Image Utilities
• ROC and Languages: What is Code? What is a Function?

Catch up on last week's news here

Repository Updates

The following new package is available in the multiverse set of both the NKEE and NKSE repositories...

• image-util 1.1.1
  • An image processing library (see below)

*New* Image Utilities

Those of you who've been with us on the journey will remember that NK3 had a set of image processing tools.

We've ported these to NK4 and made them available in the repositories as the package image-util. The tools it contains are based on Java AWT and include

• active:imageCrop
• active:imageRotate
• active:imageScale
• active:imageQuantize
• active:imageSerialize

Image processing is a natural resource model and the tooling lends itself to declarative pipeline processes. (As you'll see below, images can also be quite a good domain for demonstrating ROC concepts).

Parsing

The library provides a binary stream image parser (transreptor) which will accept PNG, JPEG and GIF binary image formats and transrepts to ImageAspect representation - which is the generic representation used by all the tools in the library and is based on a java.awt.Image

Image Processing Pipelines

The active:imageXXXX tools all accept and produce ImageAspect representations and may be seamlessly sequenced to form image processing pipelines. Here's an example of a functional crop-rotate pipeline written in declarative request syntax...

Default Serialization

A default PNG serializer (transreptor) is provided for ImageAspect to BinaryStream. If explicit serialization to JPEG or GIF is required the active:imageSerialize tool is provided.

ROC and Languages: What is Code? What is a Function?

I'm pleased to say that this series of articles has reached a point where we can think about some interesting stuff and consider the nature of code and functions in Resource Oriented Computing.

As a quick recap:

In the first article in this series I covered the basics of the ROC language runtime pattern and discussed how, by treating code as a resource, we get something we've called code state transfer.

In the second article I discussed the general nature of imports and surveyed a variety of languages detailing some typical models for accessing code resources referenced via internal import statements.

In last week's third article I discussed arguments and showed how, with the added dimension of spacial context, the difference between pass-by-reference and pass-by-value is only meaningful to the intrinsic state of the requestor, in the extrinsic ROC domain of a requested endpoint, PBR and PBV are exactly the same.

The story so far...

My hope is that these articles will have shown that ROC provides us with a computation environment where we are able to extrinsically apply computation to resources. And since functions are extrinsic, their computed resources are themselves extrinsic resources.

Caching and Performance

ROC provides an environment in which computation is normalized and extrinsic (relative to the address space). Furthermore being extrinsic, we can store all computed state. We can cache representations. The cache is extrinsic and is completely independent of language/code and even representational form.

ROC therefore provides an environment in which we can consistently and very efficiently determine when a given requestor may be served with the same stored state instead of executing code - we call this extrinsic memoization and I talked about it in depth here. This property of the ROC abstraction has dramatic implications to the overall computational efficiency which I discussed here.

So ROC is the definitive loose coupling in which architecture and code are independent degrees of engineering freedom. But this freedom and flexibility doesn't come at a computational cost. Quite the opposite. Strange as it at first appears, making computation extrinsic actually makes it more computationally efficient.

In a forthcoming article I want to talk about our motivations. Why we are doing what we're doing? What was the basis for even considering ROC?

To pre-empt that a little, when we started working on ROC ten years ago, my overwhelming concern was the brittleness and economic cost of software. I was an outsider provided with the best in conventional state-of-the-art software and it just didn't add up. The cost of building even modestly complex systems was dwarfed by the innate limits on scale and the mind-blowing cost of change.

With this perspective, for our first forays into ROC, to get a grip on the economics, I was prepared to trade performance for freedom and flexibility. It came as a bonus when we started to realize that the freedom of ROC's extrinsic computation was complemented by more efficient systems. Talk about a double whammy.

Today's Installment

So, lets get to the point, I don't want to spend any more time on the basics or the justification for this etc etc. If you've read the previous articles you'll have got the picture. Today I want to go to the next level. To consider what we mean by code and functions and to show how with a simple mental adjustment to our point-of-view we can see whole new ways of doing things...

What is Code?

We've seen that language runtimes accept code as transfered resource state. So far, we've discussed Turing complete languages and we've assumed that the code is written in a general Turing complete language. In this section I want to show that the language runtime pattern applies more generally and to show that it embodies a continuum which spans general code through to basic configuration.

I'll also show that there is a natural pressure which inclines us to travel from general purpose languages to domain specific languages and, in the limit, constrained resource sets.

But first, some pictures...

Illustrated Lecture

Its not a coincidence that we released the image processing utilities this week. It was a trade-off for me: spend an hour implementing the library so that I can use it for the discussion that follows, or spend longer using words that would be less effective at telling the story.

Let's consider the active:imageRotate tool contained in that library. Its interface has this form...

active:imageRotate + operand@[...image to be rotated...] + operator@[...transformation...]

Here's a PNG image (lets assume its resolvable with the identifier res:/nkimage.png)

The active:imageXXXX tools take an operator argument which should be transreptable to declarative form like this...

Internally the tools use an HDS tree structure to interrogate the declarative transformation.

HDS is our "house-red" representation model. It is flexible and means that we can have a range of transreptable representations all be valid formats. XML transrepts to HDS, but we can also use HDS forests, we could even provide, in the requesting context, a JSON to HDS transreptor.

We really don't care much about the data type of the declarative transform specification. All that the imageRotate tool internally cares about is that it has a node with a name of angle. It uses HDS's XPath selection syntax to find this node with "//angle". It therefore doesn't matter whether the angle tag is deeply nested in an XML document, a JSON element named angle, or just a single node HDS with name angle. They're all good.

You might ask, is the transform resource pass-by-reference or pass-by-value? Well, from last week, you know that the imageRotate endpoint doesn't care. Just so long as its argument can be transrepted to HDS and has the angle in it.

To illustrate, here's an example of a single identifier using a data: URI for the operator to specify the resource that is the 320-degree rotation of nkimage.png...

active:imageRotate + operand@res:/nkimage.png + operator@data:text/xml,<angle>320</angle>

Which gives us the representation...

Configuration or Code?

The imageRotate tools is a runtime that takes a declarative operator and applies the specified transformation to whatever image resource is specified in the operand argument.

For this brain-dead simple tool you wouldn't really describe the operator resource as code, you might be more comfortable describing it as configuration state.

Of course I'm setting you up. What's the difference between this tool and the general Turing complete language runtimes, such as active:groovy, we've assumed previously? Here's a table comparing the two..

The details of the internal representation form or the syntactic constraints of the transreptable representations may be different. But the key point is the last line. The effect is exactly equivalent - the presence of the operator state in the runtime determines the computation it performs.

So from the ROC abstraction's extrinsic point of view there is no difference. The two tools are instances of the same pattern.

And yet... I bet you feel that they're really not same sort of thing at all. How can they be?

Set Up

You've been set up, and as we'll see in this section, quite literally.

Does the base identifier of a runtime, with no arguments, such as active:imageRotate have any meaningful sense on its own?

Put another way, does active:imageRotate identify anything on its own, without the presence of the specific arguments?

These are peculiar questions since you could never resolve that identifier (the endpoint has a grammar that requires both the operator and operand arguments).

So its not a technical question. Its a question about our understanding of Resources and the nature of computability!

Hold on tight. This is the bit where we do the mental leap.

My answer is yes the parameterless base identifier of a runtime has a meaning and expresses something valid about the nature of ROC. And we get there if we step away from the "tree" of code/configuration and step back to look at the "wood", the resources.

To get concrete, active:imageRotate is the identifier of the set of all possible image rotations. Specifically since we have an internal java.awt.Image representation constraint - it is the set of all possible JPG, GIF, PNG rotated images.

Bounded infinite sets are weird and the root of many many paradoxes. So you don't need to go too deep into making meaning from these set identifiers. My advice is to just glance sideways at them every now and again and let them sit in the corner of the room. But always keep in mind that they're really there.

With this advice given, lets just say active:imageRotate is the set of all rotated images and leave it alone.

Things get more comfortable if we progressively introduce and pin down the arguments and consider what their more specific identifiers mean. Things get real so fast that we can even draw pictures to help us think about them.

Here's a table that progressively gives meaning to the sets of resources that are expressed by active:imageRotate with the operand and operator arguments progressively introduced...

Identifier	Representation	Description
active:imageRotate	∞ no representation possible	The set of all possible rotations of all possible JPG, GIF, PNG bitmap images.
active:imageRotate + operand@res:/nkimage.png		The set of all possible rotations of res:/nkimage.png. Note the representation could not actually be computed by the tool - but for illustration it can be represented as the black circle of diameter 121 pixels. The convolved view of all the rotations.
active:imageRotate + operand@res:/nkimage.png + operator@data:text/xml,<angle>320</angle>		res:/nkimage.png rotated by 320 degrees.
active:imageRotate + operator@data:text/xml,<angle>320</angle>	∞ no representation possible	The set of all possible JPG, GIF, PNG bitmap images rotated by 320-degrees Look I warned you - stop doing this. You'll break your head if you start thinking about infinite sets inside infinite sets. Put it in the corner out of sight.
active:imageRotate + operator@data:text/xml,<angle>320</angle> + operand@http://www.flickr.com/...	Finite representation not shown due to space limitations	The set of all images on Flickr rotated by 320-degrees That's better. We're back on concrete ground - even though this is a very large set it's computable *and* safe to think about. They can all be cached too!

So now you've warmed up. We can do the same for the active:groovy runtime and its Turing complete friends...

Hopefully at this point you'll recognise that the stuff I said right at the beginning in the introduction to the series...

"Ultimately I hope to persuade you to look at the world as sets of information resources. Like Michelangelo releasing David from the marble block, information engineering is the art of applying context to release information."

...wasn't just grandiloquent claptrap. This is very analagous to what you do. The art of information engineering (aka software or programming) is to define the resource sets (get some granite - the set of all possible sculptures within that block) and introduce the constraints to reveal the representation state (sculpt it).

Cardinality

The reason you might feel different about active:imageRotate and the Turing complete languages is that it feels like there must be more resources in the language runtime set. After all I could write a script that does the same job as active:imageRotate - so the set active:groovy *must* be bigger than active:imageRotate right?

I said set theory is weird but they're the same size. The cardinality (a fancy set-theory way of saying "size") of the sets of resources computable by the two endpoints is the same: aleph-0 - countably infinite.

If you followed the reasoning that said pass-by-reference and pass-by-value are the same. Then we can say that every possible representation we supply as state to either runtime can be written "in-line" as an identifier. The set of characters in that identifier is expressible as a very large integer. The integers have a cardinality of aleph-0. Therefore the two sets have the same cardinality.

To the ROC abstraction they are just instances of a pattern and produce representation state that is extrinsically managed as efficiently as possible for you. We are language neutral - configuration and code are truly and fundamentally the same. NK treats the computed resource representations just the same.

Open Runtime Problem

Right we're all running out of steam but we just have to briefly cover one more topic. I've called it the open runtime problem. Actually this came up only a couple of weeks ago when I was discussing the Security considerations of WebSockets.

You must never put a language runtime out on the wild west frontier of an open web server. Here's just one example of why..

active:groovy + operator@data:text/plain,while(true){}

Turing called it the Halting Problem. Godel called it the Incompleteness Theorem. In ROC you might call them "Unreifiable Resources".

At an absolute fundamental level there are a set of unreifiable resources for any given language runtime. The problem is that you can't tell ahead of time if you might be attempting to reify one by inspecting arbitrary code in the runtime (that's Turing's halting problem).

The upshot of this is that you cannot receive external state and issue that state to a language runtime for execution. (Believe it or not that's what the recent CREST thesis proposed and did not even mention these implications!).

Your common sense already meant you wouldn't even have considered this as an application pattern. But I just wanted to make it absolutely explicit.

And Relax

Before you get all concerned I'm not saying that the language runtime pattern has a fundamental security hole. If the code resource is in your local spacial context you're as safe as houses. Just keep it in mind that for a Turing complete runtime, external code-state transfer is a no-no.

But I didn't say that about runtimes in general did I. As I intimated in the preamble, the runtime pattern is powerful and in the limit of considering really simple configuration state (such as the imageRotate tool) you can very readily introduce constraint boundaries that can inspect this state and determine if the resource that is being requested is reifiable (and sensibly within the context of the application).

So, this is an interesting observation, as you constrain down the set of resources accessible by a runtime - you actually can determine if the resource is reifiable. The Halting problem is actually just a limit on a particular class of runtime.

So my warning is not necessarily that you should never use the runtime pattern on an open web server. Just make sure that the set of resource it exposes are bounded and consistent with your application model.

Oh, and of course, validation and constraint of state can be architected to be an extrinsic property of the address space too!

Part 5: Functions, Sets and ROC

NetKernel West 2011: Call for Papers

The NetKernel West 2011 conference will take place 13-14th April 2011 in Fort Collins, Colorado, USA. The conference will be preceded by a one-day NetKernel bootcamp on the 12th where you can get a fast track immersion in NK and ROC before the main event.

We want to make this an open opportunity for the NetKernel ROC community. We invite you to please let us know if you would like a slot for a talk/presentation/demo etc.

NetKernel West 2011
Location:	Fort Collins, Colorado, USA
Conference:	13-14th April 2011
NetKernel BootCamp:	12th April 2011

We've now confirmed the conference venue and will be opening registration very soon.

Looking forward to meeting many of you face to face in April!

Have a great weekend.