NetKernel News Volume 3 Issue 37

September 14th 2012

What's new this week?

• Repository Updates
• Resource Oriented Analysis and Design - Part 4
• Resetting the State
• Implementing active:cellResetState
• Implementing active:cells
• Board Yet?
• What's with the Code Aversion?

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repositories...

• lang-trl 1.5.1
  • Added *NEW* active:sed endpoint as a complementary tool to active:trl. Provides unix-like sed runtime. Thanks to Steve Banks at BestBuy for use-case and previewing.

Resource Oriented Analysis and Design - Part 4

Fourth in a series of articles taking a well understood domain problem and casting it as an ROC solution. Read Part 1 Here., Read Part 2 Here., Read Part 3 Here.

Recap

Last week we implemented the stateful active:cell endpoint (and its shorthand alias c:x:y). We avoided writing any code by taking a compositional approach and mapping our resources to the pds: address space. We also discovered the potential offered by allowing the address space to implement "fallback" resources and used the fpds: variant of pds:.

Now we have to look at the composite resources and in particular how to implement active:cells and its alias cells{...}. But first we need to do a little groundwork...

Reseting the State

Last time we saw that the <setup> and <teardown> requests in Xunit allow us to establish a known state for the context of the test request. With our focus on individual cells it was sufficient for us to setup by SINKing to a cell, SOURCE it in our test and DELETE its value in the teardown. But obviously this gets pretty tedious when we want to start working on the composite active:cells resource.

So our first job is to be as lazy as possible and find a way to make our life easy. In order to develop the active:cells resource we're going to need to try it out as we go - therefore before each attempt we need to be able to wipe out all cell state and restore all cells back to the default (platonic cell resource - remember last time?). So we need a state reset service.

A quick and easy first idea to solve this problem would be to implement an endpoint like this...

Notice I'm doing the groovy code inline since its so short (I have the code in a CDATA section to make it readable but the XML serialization above has escaped the < and > characters).

...but as it happens I had bigger ideas and I didn't take this option. As it turns out I paid the price... here's the warts and all steps I took...

Implementing active:cellResetState

The reason I didn't go with the simple solution above, is that I wanted to be able to show you an ROC pattern that's pretty handy. I wanted to show how sometimes its useful to adopt a convention for our identifiers which allows us to talk about a collected set of resources, rather than single items.

I was pretty sure that pds: implemented this convention so I quickly implemented my answer to the reset with this endpoint declaration...

My expectation is that a DELETE request for active:cellResetState is mapped to a DELETE request to pds:/ttt/cell/ which ought to allow us to delete multiple resources sharing the same root path. This is an instance of the naming convention I touched on above.

The convention is that an identifier ending with a trailing slash (/) is referring to all of the individual member resources below that path.

Deleteing everything in pds:/ttt/cell/ obviously covers the atomic set of cells we'd established last time when we implemented the atomic resources.

To check my new service I added this test ...

And naively, I expected that I had shown off another neat trick and avoided any code... Oh such hubris, Oh such over-confidence; I was ripe for a fall...

Aaaaagh! My test failed with a horrible exception. I clicked execute on the test to see the detailed stack trace above. NullPointerException - ouch. I'd found an edge in the ROC universe.

Since this was an NPE, I went to the mod:pds module, opened it up and looked at the source code for org.ten60.netkernel.mod.pds.PDSInMemoryImpl.delete().

Ooops - on inspection I discovered that the pds in-memory impl is just a very simple in-memory key-value map - it doesn't implement the "trailing slash pattern" I was aiming to show off. Bugger!

[Note to the "not to be named author" of the in-memory impl - the NPE is not a nice way to behave - we should fix this!]

OK a quick face saving recovery is needed. I'm damned sure that pds: does implement this pattern as we use it all the time in the system tools. Its just we use the truly persistent version that is backed by a local RDBMS implementation.

What's it called? I typed "pds" into the search tool in the control panel and found the pds docs...

http://localhost:1060/book/view/book:urn:org:ten60:netkernel:mod:pds/doc:mod:pds:title

Ah ha that's what I want, I quickly switched my module's import to import the rdbms:local version rather than the in-memory version...

That should sort it. Such conceit! My second fall was coming...

I ran the unit tests to see if my new active:resetCellState endpoint was now fixed. At this point all hell broke loose...

At first glance bad. But relax and look again. Actually not too bad - since the tests are executing without errors its just that several assertions are failing...

Look at one of the assertions and think a moment...

Ah look at that! Its the <stringEquals> assert that's failing. Hmmmm, because I've switched the pds: implementation it's now returning a ByteArrayRepresentation instead of a java.lang.String.

Oh of course - the in-memory pds impl is a POJO-bucket that will hold any old POJO representation, but the RDBMS-backed version of pds is a bit-bucket and necessarily has to store its state as Blobs. So it always requires its argument resources as binary streams. When we SOURCE from it without expressing any representation type preference we get serialized binary streams back!

In order to be equivalent to what I had before (String POJOs as my basic representation model) I needed to ensure that my request to pds: automatically transrepted the representation to a java.lang.String. So looking at my atomMapperConfig.xml I modified the request to pds: and added a <representation> declaration to say "Don't come back if you're not a String"...

That should sort it! Wrong again. This time when I ran the tests I saw exceptions! Clicking on a failing test I saw...

Request Resolution Failure
TRANSREPT from ByteArrayRepresentation to String

Primitive type transreption errors like this happen when your address space doesn't have any primitive type transreptors (not a very helpful sentence - but its a truism). Oh yeah of course, to allow my endpoints to automatically discover a transreption pipeline I'd forgotten to import layer1 into my mapper's space so that it had access to all the low-level transreptors that layer1 offers...

OK all back in the green again - I had fully reimplemented my persistence layer (now with true long-term persistence not transient in-memory) and I was confident that the typing was all correct and that my high level tictactoe atomic resource model was behaving exactly the same as before...

All except my damned active:cellResetState test I'd added! I was asserting <true>, expecting the DELETE on pds:/ttt/cell/ to succeed - but it was coming back false.

Damn Damn Damn... should I give up and just implement the simple groovy cell deletion script after all? I could just pretend like I never made this misstep - dear reader, you would be none-the-wiser.

No way. That would be a lie and a defeat and... and... and... I'd have had to write some code!!

So with a cry of "No surrender" - which startled the others in the office. I dug deep and looked closely at the atomic active:cell mapping to fpds:. The mapped request looked like this...

Ah ha, a little subtle, but look at that. I hadn't given any thought to the actual identity of the pds: path I'd used when I set this up. Could it be the slash delimited path notation fpds:/ttt/cell/x/y that is causing the problem?

We are trying to delete set pds:/ttt/cell/ but I seem to remember that the pds: model doesn't differentiate super-sets - its a bit like the unix-path and expects the trailing set convention to only apply to members after the last slash. (Err its in the name "trailing slash convention").

So, for efficiency of implementation, the pds-local-RDMBS impl only thinks of the last slash as being the identity of a collection of resources, not the higher level paths. So to succeed what I really needed was to ensure that all my mapped cells went into a single pds: "trailing slash collection set" like this: pds:/ttt/cell/x-y.

Oh thats easy! I can just change the mapping. Lets get rid of the slash delimiter between x and y and instead lets use a dash (-)...

I also remembered that because we're using fpds: I also had to change the fallback resource too..

I make the changes. Run the tests... and ... we're all green! Yes! By the skin of my teeth I'd avoided writing any code.

And who knows - this stupid little detour might have cast some light on the trailing-slash pattern and also revealed some implementation detail on pds:. Maybe not a wasted journey. Anyway this had taken about 5 minutes - so not too bad.

OK lets make sure we're actually reseting the state by adding a test with some sequences of stateful requests. Here's a sequence of tests that first SINKs, then resets and then SOURCEs to show that our active:cellResetState endpoint is working...

This sequence of tests are all green. active:cellResetState works! Now we can reset the state easily. Now we can actually start on the real story this week...

Implementing active:cells

You will recall that the composite resource active:cells was left as a simple mapping to the dummy resource which when requested in our tests is returning "ReplaceMe!"...

It follows that all of the other composite resource mappings such as row:y and column:x are producing the same state (since they're all ultimately just aliases to active:cells).

Time to make active:cells do something real. Dare I say it, time for some code!

I changed the request mapping from res:/dummy to active:groovy like this...

I also, having had my fingers burned above by forgetting to provide imports when introducing new services, added an import of the groovy module urn:org:netkernel:lang:groovy into my mapper space so that active:groovy would be resolvable.

Finally I created a script cellsImpl.gy in the atom/ directory. To start with, I made it implement a code-based equivalent to the dummy mapping - as you can see, I returned a string telling me that I hadn't done anything yet...

context.createResponseFrom("Not Finished Yet")

I ran my tests to make sure that the mapping was correct and that my code was running instead of the dummy resource...

This time the assert failure is giving me some good news. Its saying that my test is calling the code since its now returning "Not Finished Yet" but unfortunately all my asserts for the composite resources are expecting the old dummy resource.

So, using the power of search-replace, I did a global search for "Replace Me!" and changed it to "Not Finished Yet"... now, all the composite resource asserts looked like...

Rerun the tests and all is good. In fact all is exceptionally good. Something rather pleasing just happened. I changed all the asserts including all the asserts on the row:y, column:x and diagonal:z tests. They're all passing too.

What this is telling me is that all of the composite services are all correctly mapping to the active:cells endpoint and its single line of implementation code. My new dynamic implementation of active:cells is now being used by all of the composite resources that map to it. I have a single normalised focal point to concentrate my efforts on and I know they will be reflected into all the other services...

OK we're ready to implement a real active:cells solution. But I need to focus and check my progress so I choose the first active:cells test and execute it (test 10 in the list)...

http://localhost:1060/test/exec/html/urn:test:org:netkernel:demo:tictactoe?index=10

I see "Not Finished Yet" in my browser. Now I can change the code and refresh this link to iteratively solve the problem.

First off I add another line of code to echo back the operand argument (you'll recall this is the composite identifier containing the list of cell identifiers)...

operand=context.getThisRequest().getArgumentValue("operand")

context.createResponseFrom(operand)

When I F5-refresh my browser pane I get back...

c:0:0,c:0:1

OK I can see the cells. Next we need to split this on the comma separator "," to get a list of identifiers...

operand=context.getThisRequest().getArgumentValue("operand")
cells=operand.split(",");
for(i=0; i<cells.size(); i++)
{   println(cells[i])   
}
context.createResponseFrom(operand)

I println to stdout and glance at the console to see that I have the identifiers being spat out. OK nearly done. My final job is to build an HDS tree representation...

import org.netkernel.layer0.representation.*
import org.netkernel.layer0.representation.impl.*;

operand=context.getThisRequest().getArgumentValue("operand")
cells=operand.split(",");
b=new HDSBuilder();
b.pushNode("cells")
for(i=0; i<cells.size(); i++)
{   b.addNode("cell",cells[i]);
}
context.createResponseFrom(b.getRoot())

Now in my browser refresh I get...

Which is just a structured list of cell identifiers! I need to dereference these identifiers. By which I mean I need the representation state of the identified cell. So I need to SOURCE each identifier. Here's the final result...

import org.netkernel.layer0.representation.*
import org.netkernel.layer0.representation.impl.*;

operand=context.getThisRequest().getArgumentValue("operand")
cells=operand.split(",");
b=new HDSBuilder();
b.pushNode("cells")
for(i=0; i<cells.size(); i++)
{   b.pushNode("cell",context.source(cells[i]));
    b.addNode("@id", cells[i])
    b.popNode();
}
context.createResponseFrom(b.getRoot())

And in my browser I see...

The important detail is that I added context.source(cell[i]) which in this example means I am making requests for c:0:0 and c:0:1 and incorporating the representation state into the HDS tree structure. Notice also that since I know the identity of the cell resource - I might as well retain it for future reference so I add it as an attribute on the cell node.

Progress. But this is now not a good test since it is just pointing to a set of empty cells. I need to modify my test to setup some intial state and check that my active:cells implementation is really doing what I think...

The test now gives me...

I have a real solution. active:cells is finished in 12 lines of code.

One thing to point out in my updated test. Notice that the assert has changed - its now using an <xpath> assert. Xpath is a great way to give a rich assertion on tree structured data. In this case I'm saying: the first cell should have the value "X".

But to be able to use the <xpath> assert I need to edit my Xunit testlist.xml and import the xml assertion library. I can never remember what its called so I search "xml assert" in the docs. The page comes up and so I add the import to my testlist.xml...

Again I remember that my test space (in my test module) knows nothing about the xml:core library (which provides the xml assertion services) so therefore I need to add an import for urn:org:netkernel:xml:core.

My first test of active:cells is now complete and showing that I have a full working implementation. But of course, since my composite resource tests are live and hitting the active:cells endpoint they are all showing assert failures since they are still using <stringEquals> asserts. I quickly do a search-replace...

<stringEquals>Not Finished Yet</stringEquals>

is replaced with...

<xpath>/cells</xpath>

Hey presto all my row:y, column:x and diagonal:z resources start passing their tests and so must be producing real <cells> resources!

Finally and for good measure I <setup> some inital state for each of those tests so that I can prove they're all for real.

I've finished the problem. Total time (including the stupid state reset detour) 15 minutes.

Board Yet?

I was still highly caffeinated from my earlier relaxation breaks. So decided to press on a little further. I knew that Tom had implemented a way to get back a representation of the state of the complete board. What would it take for me to do the same? Perhaps more importantly, could I do it with declarative composition and no more lines of code? (The twelve lines in the active:cells implementation was already pushing my limit for a working day).

Of course I already knew the answer... I quickly created a file called board-hrl.xml in the composite/ directory. With a bit of cut and paste it looked like this...

I created the basic structure and one <row> then copied and pasted the others then I changed the identifier in the row <request> to point to the correct resource I wanted.

Next I added a new active:board endpoint using a mapping in the compositeMapperConfig.xml which maps to active:hrl and tells it to evaluate the board-hrl.xml resource...

active:hrl is a recursive HDS composition language - it is a member of the family of recursive composition languages which includes XRL and TRL. It is very simple. It recursively evaluates embedded requests and adds the HDS representation state to that location in the tree structure. It supports declarative requests (used above) or full NKF request objects. It is also trivial to make the evaluated requests asynchronous by adding @async attribute to the declarative request.

I knew that HRL is in a library so I added an import to my mapper space for urn:org:netkernel:lang:hrl (if its not installed on your machine look for lang-hrl in apposite). You can see how simple it is from the docs here http://docs.netkernel.org/book/view/book:lang:hrl:book/doc:lang:hrl:title

Finally I added a test...

I ran the test and got back...

Job done. Literally this took one minute.

I wanted to show one more thing. I cut and pasted a copy of my active:board test. So I now had two of them back to back. I tweaked the first to have the setup and the second to do the teardown - since I wanted my requests to active:board to run on exactly the same cell state...

Take a close look at the asserts. In the first I am saying "I expect this to take at least 5ms". In the second I am saying "I expect this to take 0ms". In both cases I expect the representation to be <notExpired>.

Aside from the ease of being able to create composite resources, as we see in the row:y, column:x and active:board solutions. Here at last, is the evidence of a further massive payback for decomposing the state into atomic and composite resources.

The composite resource active:board has requests for row:x which maps to active:cells which has requests for active:cell. But what we're seeing is that every single resource starts to roll up into the high-level composite.

So long as nothing changes nothing needs to be recomputed.

If something does change (like one cell when we start playing the game) only one active:cells request (and therefore its alias row:x) are affected. Therefore when the active:board is requested again we automatically recompute only the affected row and, it follows, active:cells only requires that the one affected cell is actually requested directly. Everything else is the same as it was last time.

We have achieved a completely normalized computation for the state of the tictactoe game.

And the visualizer confirms this. Here's a filtered view of my tests showing just the back-to-back active:board requests...

Alternative Board with Code

Just for the hell of it I implemented a code-based solution to the same problem. I added a mapping for active:boardWithCode to active:groovy and the following script...

import org.netkernel.layer0.representation.*
import org.netkernel.layer0.representation.impl.*;

b=new HDSBuilder();
b.pushNode("board")
for(i=0; i<3; i++)
{   b.pushNode("row")
    row=context.source("row:"+i, IHDSNode.class);
    b.importNode(row.getFirstNode("/cells"));
    b.popNode();
}
context.createResponseFrom(b.getRoot())

Not much code but just compare it with the compositional approach. It has so many moving parts. So many ways it could break or be misinterpreted or become brittle when required to add new features in response to change.

The HDSBuilder is convenient but it separates me from thinking in the domain of the solution. While the HRL composition actually lets me inspect and play with the structure of the result in the same domain (just by copying and pasting). I literally didn't need to consciously think how to solve the problem I just "painted the structure I wanted". But the composite approach goes much further...

It makes it trivially simple for me to augment it. What if, as we get into developing the actual game, I need to return the history in the same resource as the board's current state? Simple, I would just embed a request to active:history (which we haven't started on yet). What if we're scaling this service up to run as a stupid facebook app playing a 100 million simultaneous games. We would surely want to parallelize the requests to the persistence engine to ammortize the network transfer time? No problem, just add an @async attribute - much much easier than dealing with callbacks or iterating over asynchronous NKF request handles.

What's with the Code Aversion?

What I hope I'm starting to show, is that our instinctual tendency "to start with code" is not always a good idea. I hope that its starting to become apparent that declarative composition can produce extremely concise solutions and yet offer scale-free evolvability.

Of course, and as Tony keeps pointing out, when I say I did it with "no code". What I really mean is that I have minimised my need for "imperative code". I have, as much as possible, stayed in the declarative compositional world.

And you might ask yourself: Do I know a wildly successful technology which adopts this same declarative compositional model?

And then look at what you're using to view this story? Yep, all I'm doing inside my ROC solution is using the self-same patterns and proven economics that your web-browser uses.

Ask yourself: Would web-page development be easier with imperative code or with declarative HTML?

So why do we feel the need to start with code to solve general information (software) problems?

Next time we'll look at implementing the game and find there's not a lot to it...

Checkpoint

You can download a snapshot of the modules at this point here...

• urn.org.netkernel.demo.tictactoe-1.1.1-r4.jar
• urn.test.org.netkernel.demo.tictactoe-1.1.1-r4.jar

Note: You will need to install lang-hrl from apposite if you don't already have it.

[Read part 5]

NetKernel West 2013

Several people have recently asked if we're planning a conference. To which the answer is: "planning" would be too strong a word - but heck, yeah, why not...

So here's the "plan"...

Date: lets say spring 2013 to give everyone some lead time.

Location: USA. Those who've been to any of the previous three conferences will know we have exacting requirements. It has to be somewhere young, cool and hip (college towns have been good to us so far), within reasonable travel from a decent airport, *not* be a corporate hotel and (purely coincidentally) have a thriving local brew-scene. Any ideas?

Format: Shall we stick with the one or two day pre-conference bootcamp followed by two-day main conference?

Content: Now you could come along and listen to me go on and on about the esoteric nature of reality and the relation between resource spaces and chrono-synclastic infundibulae*... But its time to get real. To share experiences. To establish common practice. To get a shared perspective of how ROC fits in the IT landscape....

What we as a community all need are your presentations describing your experience, your practices, your patterns, your stories...

So get thinking and send me a short proposal. All selected speakers will be rewarded with some shameless give-away (can we do your own weight in beer?)...

Sounds like a plan... see you in 2013...

*Courtesy of Kurt Vonnegut and pointed out by B. Sletten Esq.