NetKernel News Volume 3 Issue 38

September 21st 2012

What's new this week?

• Repository Updates
• Update_Notes
• Resource Oriented Analysis and Design - Part 5
• Working from Both Edges towards the Middle
• WWW Surface
• Giving it some style
• Adding Structure
• Restart at the Beginning
• Putting down some moves
• Reset Shame
• Tidying up - a touch of constraint
• 1060 Research Ten Years Old

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repositories...

• cache-se-1.7.1.nkp.jar
  • Resolution cache key enhancement (see below)
• http-client 2.12.1
  • Eliminated potential for connection pool starvation in very heavy load if a connection fails but is not aborted. Thanks to Steve Banks and the BestBuy team for discovering and suggesting the fix for this.
• intray-transport-1.4.1.nkp.jar
  • Fixed to use correct context during postCommission() configuration
• layer0-1.91.1.nkp.jar
  • Thread safe endpoint change and ClassLoaderResourceRepresentation file handle close (see below)
• layer1-1.40.1.nkp.jar
  • Throttle configuration visibility (see below)
• nkse-control-panel-1.26.1.nkp.jar
  • Updated for new infobox (see below)
• nkse-cron-1.12.1.nkp.jar
  • Added option to prevent automatic reissue of misfires on restart (see below)
• nkse-dev-tools-1.48.1.nkp.jar
  • Updated for new infobox
• nkse-docs-1.20.1.nkp.jar
  • Updated for new infobox
• pds-core 1.9.1
  • Associate a golden thread on the fpds: SOURCE if it is a member of a "trailing slash set".
• system-core-0.33.1.nkp.jar
  • (see below)
• xml-saxon-1.10.1.nkp.jar
  • active:xpath2 now treats "/" the same as "/*"

The following updates were also made available in the NKEE repository...

• nkee-apposite-1.42.1.nkp.jar
  • Updated for new infobox
• nkee-architecture-1.7.1.nkp.jar
  • Updated for new infobox
• nkee-layer0-1.3.1.nkp.jar
  • Eliminated rare potential for a HashMap.getEntry() spin in the EndpointProfiler

See the detailed update notes below...

Update Notes

Infobox visibility (nkse-style, control-panel)

Additional information boxes that explain usage have been available on many of the control panel and developer tools in the management console however a lot of people didn't know they existed. By default they where hidden and a little blue (i) icon displayed them. This change makes the infoboxes initially displayed on a fresh installation until dismissed by the user. Once dismissed they will not be re-displayed unless re-opened manually again.

This new feature required small changes to several modules to remove unneeded imports of javascript libraries that are now in the main control panel template.

ClassloaderResourceRepresentation file handle left open (layer0,ext-introspect)

We've had two independent reports of problems with file handles being left open. Firstly on Windows this meant that some source files were locked for editing and on Linux deployments with very large heaps, so many representations were in the cache that available operating system file handles could be exhausted.

The previous implementation left file handles open where possible (this usually happens between construction of the representation, reading file metadata and first streaming of the contents) to optimise file access. This change makes the default behaviour to close file handles between accesses but allows for a kernel property netkernel.filehandle.optimize to be set to true to change to legacy behaviour.

Thread-safe Endpoint default (layer0, ext-introspect)

Previously all endpoints that were not explicitly marked as being thread-safe (using a declareThreadSafe() call in their constructor) would not be allowed to receive multiple concurrent requests by the kernel. This change makes the default configurable with a kernel property netkernel.threadsafe.endpoint.default. If the default is not specified then the behaviour remains as it was, to assume non-thread-safe code (false). The property is editable in the kernel configuration tool on the control panel.

Cron issueMisfiresOnRestart (mod-cron)

Previous behaviour of Cron was that any job that failed to fire because of downtime of a NetKernel instance would fire immediately on restart. Sometimes this is not desirable so an additional configuration property has been added into the job specification <issueMisfiresOnRestart>. The default behaviour remains unchanged but may now be specified with

in a <job> declaration.

Throttle config in explorer (layer1, ext-introspect)

The throttle overlay now reports both concurrency and maximum queue length in it's metadata which is exposed to the space explorer. This allows you to view the live configuration of a throttle in a production environment.

Resolution Cache verb check (cache-se, ext-introspect)

This fixes a small corner case that could cause problems when different endpoints are used to implement different verbs for the same identifier. The resolution cache now keys on the request verb and this stops resolution for one verb being used to determine resolution for another.

Tom's Blog

This week Tom gets excited about <literal>...

http://practical-netkernel.blogspot.be/2012/09/short-and-sweet.html

Resource Oriented Analysis and Design - Part 5

Fifth in a series of articles taking a well understood domain problem and casting it as an ROC solution. Read Part 1 Here, Read Part 2 Here, Read Part 3 Here, Part 4 Here

Recap

Last week we implemented persistence of the cell state. We also added an implementation of the higher level composite resource active:cells endpoint and used a recursive "linked data" composition pattern.

Having established the core state of the TicTacToe resource model, today we need start to think about the game itself...

Working from Both Edges towards the Middle

When building a solution I like to follow a very simple mental model. It goes like this...

1. What have I promised that I will provide?
2. What do I need in order deliver my promise?
3. Add value (the thing that makes this special)
4. Return a response

It turns out this sequence of steps is scale invariant. That is, the same steps apply when you are locally implementing the internals of an accessor as when you're implementing a complete channel through a top-to-bottom multi-tier architecture.

Essentially this is a recipe for local success and it implicitly suggests a "directed spacial relationship" to the contextual architecture at each step.

• Step 1 is focused on looking upwards to any layer above.
• Step 2 is saying what arguments have I been given but, also, what further resources should be requested - ie what should I expect from promises others have made to me (from peers at the same level or often from other layers below)
• Step 3 is what makes this endpoint unique.
• Step 4 just says, you should always keep your promise.

It follows that these steps suggest that a good working practice is to try to work from the edges inwards - since then you'll know what you've promised and what's been promised to you.

This contrasts with Object Oriented software, in which the focus on object design and relationships tends to start in the middle and move to both edges. Hence the term "middleware".

By working at the edges first - our hope, and the ideal, is that there is no middle. I sometimes think of this as "no-ware".

So lets find out how much middle we really need...

WWW Surface

Our unit test fixtures gave us an easy way to probe the resources of the core model. But now we're working on the application itself we want a more rich and dynamic interactive development model.

We need to see instant results to our changes. We want a short virtuous circle of feedback.

Iterative feedback provides the environment for creativity and inspiration. Imagine trying to paint a picture if you only looked at it after you'd covered all the canvas? Creativity is a process. It requires you to adapt and change course when a new idea surfaces.

So the first job is to create a new module. Somewhere we can work on a live web rendering layer. Which is another way of saying - we need a top-edge to complement the bottom-edge of the TicTacToe resource model.

Using cut and paste of the boiler plate, like we did in part 2, I quickly knocked up a new module called urn:org:netkernel:demo:tictactoe:www.

I knew that I'd need a mapper so set one up pointing to an empty configuration for the mappings (wwwMapperConfig.xml). I added a SimpeDynamicImportHook.xml resource to make sure this module would get imported into the HTTPFulcrum and hence be exposed to Web HTTP requests. Finally I added an import of the tictactoe resource model urn:org:netkernel:demo:tictactoe (the bottom edge to which we'll work towards) ...

I added the new module to my etc/modules.d/tictactoe.xml file and it was discovered and commissioned.

In order that I could start to see something happening, I quickly added a mapping from res:/demo/tictactoe/ to active:board in the file wwwMapperConfig.xml...

In my browser I clicked this link...

http://localhost:8080/demo/tictactoe/

And lo and behold I saw an empty <board> representation - just like we saw in our active:board unit test (see last week if you don't remember it).

So I now had a top-edge connected to the bottom-edge and no middle. Here's what my spacial architecture looks like...

Click image for full-size SVG

Giving it some style

So far this raw representation rendering was not very inspiring. So my next step was to make it look like a minimal tictactoe game. I changed the mapping to this...

And wrote a hacky stylesheet to turn the data into an HTML table representation of the board - don't worry about the ugly approach - I just cut and paste the structure and changed the paths - at this point we're not after perfection - we just want to see something...

When it failed first time - I remembered that yet again I'd forgotten to add an import to my mapper space to provide active:xslt - so I added an import for unr:org:netkernel:xml:core. When I clicked the link again I saw this...

(From the stylesheet you can see that I am not trying to make this a well formed HTML page - its just a <div> fragment. But I'm getting my browser to render it nicely by ensuring that I return a mimetype of "text/html" - look at the mapper definition above, notice that I'm adding an explicit "mime" <header>).

Progress - its starting to look like the game. But of course the cells are empty as the model has no state.

Checkpoint

You can download a snapshot of the www module at this point here...

• urn.org.netkernel.demo.tictactoe.www-1.1.1-r1.jar

Adding Structure

So while I had a cheap and dirty representation of the board, it was, not to mince words - a heap of crap. A context free div fragment is not a web application. I needed a contextual page. No problem I'll use the recursive composition pattern again to create a recursive template structure.

I copied and pasted my endpoint. I renamed the grammar for the active:xslt request to be res:/demo/tictactoe/board - which is a more specific name. I then mapped the original res:/demo/tictactoe/ to a request to active:xrl2.

XRL is a recursive XML resource composition language - in the same family as active:hds. It requires a template argument which will be recursed. In this case I also provided a content argument (my name) and provided a reference to the XSLT generated board resource...

I created the necessary directory structure and implemented main.xml like this...

Notice the embedded <xrl:include> - this is a declarative request that is evaluated by the XRL2 runtime. Here it will source the content argument - which as we saw in the declaration will be dereferenced to res:/demo/tictactoe/board - which in turn will invoke the active:xslt styling of the active:board resource.

While I was at it, do you notice I imported into the page the Jquery library (I know this is in the backend-fulcrum - so for now I'll just reference its URL on the other transport - we can sort this out later).

So now I can animate my solution. I added the following snippet of javascript. I actually placed this inside the XSLT template since its where my focus of attention is and I need to be able remember the structure of the HTML table for the board. (I can always move it later)...

If you don't know JQuery the gist of it is this - when this div fragment is rendered the function bindCell() will be invoked. This function finds all elements with a class of "cell" (see my table stylesheet). For each discovered element bind the "click" event to invoke the inline anonymous function. For now I show that things are working by displaying an alert box with a message.

I go back to my browser and refresh. The table looks the same - but now when I click a cell I see this...

Result. We're pretty much there already.

Checkpoint

You can download a snapshot of the www module at this point here...

• urn.org.netkernel.demo.tictactoe.www-1.1.1-r2.jar

Restart at the Beginning

Now I remembered from my unit testing of the core resource model that its important to be able to cleanly reset the state. So I added a link to my main.xml template like this..

Its going to request the relative URL "restart". So now I have to map that to something. I quickly copy and paste an endpoint definition and write an inline groovy script like this...

Just like in our unit tests cleanup, it simply makes a DELETE request on active:cellResetState. I'm doing a funky trick known to all HTML-jockeys. The browser will request "http://localhost:8080/demo/tictactoe/restart" - but I don't wan't it stay there so my response includes a relative redirect back to the original /demo/titactoe/ path. Result I have an invisible reset/redirect/re-render of the page.

In my browser I try the Reset link and use the visualizer to see the correct series of requests going down from the top-edge to the bottom. I have a reset.

Putting down some moves

OK we're now ready to take first cut at implementing a real game. I can't quite remember the exact order that I did the next steps but here's a retrospective view...

The first rule of "no-ware" is to delegate state management to the edges. That is you don't want state in the middle since it is very brittle and is a scaling bottleneck.

I followed this rule and added a variable called "player" to the javascript of the main template. This will hold the token (X or O) of the currently active player. I also add a function togglePlayer to toggle the token value.

The script in main.xml looks like this...

Notice that I also added a function to do an in-situ AJAX refresh of the "board" element. Notice it calls the relative URL "board". This is just a request for the active:xslt generated board fragment. My original endpoint declaration was already providing a suitable AJAX micro-channel.

Now I need to change the bindCell() function so that it actually does something when I click a cell...

Notice I do a JQuery $.get() of the relative URL "move" and provide some query parameters for the "state" (the player token value) and the "cell". This latter is a neat trick - I just find and pass the "id" attribute from the clicked cell (this happens to the be the c:x:y identifier of the cell in our resource model - another example of linked data references).

I associate an anonymous callback function to the GET request. If the received data is "OK" then I togglePlayer() and refreshBoard(). So the successful move causes the board to automatically update and the player to change turns.

All I need to do now is to add a mapping for the "move" REST path. So I cut and paste the "restart" endpoint declaration and modify it to look like this...

Notice I source the cell and state parameters from the httpRequest:/param/ space. Then I do a SINK request to the cell with the received state.

I refresh my browser and click a cell...

Nothing happens! I turn on the visualizer and try again. All my requests are looking good. I look at the response for the AJAX request to load the "board" micro-channel. Whoops, look at that its not rendering the operand correctly - I can see that it has the X value in the cell but the HTML is borked.

I look at the XSLT and realise that the xpath I used to select the cell state is missing the "cells/" part. A quick change like this does the trick...

I can play the game and the board starts to play like a traditional piece of paper version..

Reset Shame

Having filled up a board. I hit reset. Crap. Its not reseting. The board seems to be cached. I turn on the visualizer. Everything looks ok but when I reset the cells state the active:board resource is not expired....

Now, the quick fix, is to look at the application level and to associate a golden thread with active:board and to cut the golden thread when a move is made. But that wouldn't be the normalized solution! So I dug deep.

After half an hour of digging around with the visualizer I am red-faced. I discovered that whilst the DELETE to fpds: is clearing state and cutting golden threads - there's a problem with the implementation of the trailing slash pattern in the RDBMS impl of the pds: tool.

It turns out that whilst its correctly deleting its internal golden threads - the SOURCE request for fpds: resource is only associating a golden thread for the identified resource but *not* the "trailing slash set".

So when I delete a set - any SOURCEd individual items are not expired since they don't know about their membership of the set. The SOURCE needs to also associate a golden thread for the set.

Its a one line - additional argument to the call to active:attachGoldenThread in the pds impl.

With this in place. All of sudden reset works and the game is done. (I've built and shipped an update to pds-core in the apposite updates - see above).

Tidying up - a touch of constraint

So far you'll have noticed that there is no logic in our solution. There are also no constraints to prevent, for example, setting the state of a cell that has already been played. Don't worry - the constraints are still not our focus. We're still in the composition stage.

However, because I was publicly humiliated by the mod-pds issue - I went back to my hacky XSLT stylesheet and quickly reworked it so that it automatically renders and generates the cell "id" value. Showing that I can do things other than cut and paste(!) and also its a very small pre-cursor to the application of constraints that we'll do later...

Next time we'll try to make a better game than the one you can play on a piece of paper...

Checkpoint

You can download a snapshot of the www module at this point here...

• urn.org.netkernel.demo.tictactoe.www-1.1.1-r3.jar

Note: You will need to accept the update to pds-core from apposite if you don't already have it.

[Read part 6]

1060 Research Ten Years Old

Somewhere either today or tomorrow, 1060 Research will start its second decade. When we started I knew that we were committed to playing a long game. Step by step we have patiently been laying the foundations.

There are many many people who have helped and supported us along the way. Thank you. You know who you are.

One thing that we have constantly revisited is an attempt to succinctly answer the eternal question "What is NetKernel?". It so happens that yesterday, off the top of his head, Tony might just have put his finger on it...