NetKernel News Volume 5 Issue 3 - RESTOverlay with Transform, HDS2 Full XPath with Builder

NetKernel News Volume 5 Issue 3

February 21st 2014

Catch up on last week's news here, or see full volume index.

Repository Updates

The following updates are available in the NKEE and NKSE repositories

  • http-server-2.25.1
    • RESTOverlay supports multiple <produces> and adds support for @withTransform - see below
  • module-standard-1.62.1
    • enhancement to allow literals to be declared private <literal uri=“res:/literal” type=“string” private=“true”>string</literal>
  • nkse-control-panel-1.36.1
    • fix to stop debug message when module is unlicensed
  • nkse-dev-tools-1.60.1
    • fix to refresh button on log view
  • restoverlay-test-1.2.1
    • New tests of @withTransform feature

The following updates are available in the NKEE repository

  • nkee-deadlock-detector-0.8.1
    • stop debug message when module is unlicensed
  • nkee-license-manager-1.13.1
    • added improved reporting of license status in license management tool fixed warning dialogs on control panel when any license fails to verify

Please note these updates are published in the 5.2.1 repository.

HDS2 Preview: Full XPath functions, Mutable Builder

Tony has been scratching an itch and has taken HDS to the next level. Whilst useful as an API in its own right, this work also provides the foundation for resurfacing the active:stm runtime (Simple Tree Manipulation) with implementations for both XML and HDS...

We’ve known that the usability of HDS has needed to be improved for a while now. Working on Comb9 with HDS as the primary representation within the app has made me see how much brittle code I needed painstakingly write to manipulate HDS. So there are a few motivations behind this new HDS.

Firstly, and primarily, the aim was to gain support for full XPath. Additionally I wanted to get a mutation interface (not sure - is there an OO pattern name for this?) similar to XDA that we have for XML.

Also it’s been a bugbear for a long time that HDS has a parent reference in the data-structure. This was necessary for navigation but causes big inefficiencies because it is always necessary to clone the whole tree when making modifications.

I have used Apache JXPath to provide XPath support. With a bit of customisation it has provided a seamless implementation of XPath. It is a 300k jar and licensed, obviously, with an Apache2 license. Because of the capabilities of JXPath it has been possible to eliminate the need for the parent reference whilst still being able to perform full XPath including support for .. and ancestor etc axes.

To provide full backward compatibility with existing HDS I have created an IHDSContext and IHDSMutableContext interfaces which can be created by a factory and wrap or import and existing IHDSNode or a new lighter weight IHDS2Node. Also you can create an empty IHDSMutableContext to create a document from scratch.

The IHDSContext interface provides methods to perform queries to gain a single or multiple values and single or multiple nodes in response.

The IHDSMutableContext provides similar methods to XDA such as insertBefore, rename, delete etc but supports the concept of a cursor. This allows a developer to seamlessly use the builder pattern functionality such as pushNode, popNode with macro manipulation operations. When all manipulation is complete you can extract an IHDSNode or IHDS2Node from the context using toHDS() or toHDS2() methods.

Here is some sample code:

IHDSMutableContext b=HDSContextFactory.getNewHDSContext();
b.pushNode("b").addNode("id", 1).addNode("v", "first").popNode();
b.pushNode("b").addNode("id", 2).addNode("v", "second").popNode();
String v=(String)b.getFirstValue("/a/b[id='2']/v”
IHDSNode hds=b.toHDS();

Details to download the module and a set of unit tests are provided below. Have a play and let’s discuss.

I don’t plan to release this right away as I need to use it anger to check it really works as well as I hope it will!


You can download the module here. At the moment this module is just for testing. When we've had some feedback we'll decide whether to make this a standalone module or just incorporate it as an update to layer1.

mod.hds-2014-02-20.jar - mod.hds temporary module
test.layer1-2014-02-20.jar - tests for layer1 including hds2 tests

RESTOverlay Update: Multiple <produces>, @withTransform mimetype-based representation transforms

While implementing a Collections+JSON set of REST services Brian Sletten proposed an enhancement for the RESTOverlay. His idea was to support multiple <produces> tags on a target endpoint and to allow for late-bound transformation services to be specified.

A representation from the endpoint could then be transformed when the matched type is not the same as the endpoint's default response representation. A sort of mimetype-based transreption.

Its one of those things that's simple to understand when you see it, but hard to describe in words. Here's the gist of how it works in practice.

Here's an endpoint with simple grammar acn1. It has three possible mimetype's declared by three <produces> tags...

    <produces withTransform="active:acnXML2JSON">application/json</produces>
    <produces withTransform="active:acnXML2HTML">text/html</produces>

The default is to provide text/xml. However if your HTTP request's Accept header specifies a preference for text/html then this endpoint will still be matched (based on the 3rd <produces> tag saying it can do text/html). But when the endpoint returns its default text/xml, then the RESTOverlay will call the transform service declared with the @withTransform attribute to "transrept" the response to text/html.

By convention, the initial response is passed as the operand to the withTransform service. Obviously you have to implement an endpoint that does this tranform and has the appropriate grammar.

Typically you're likely to have some common transformation services that can be used by many of the REST API's endpoints.

The most immediate usecase for this new feature is to have a REST API that responds nicely to browser requests (without the need to duplicate endpoints to deal with text/html) but there are potentially many ways that content negotiated transreption based on mimetype can be useful.

Great stuff Brian!

Here's the formal documentation for the new feature, like I said its easier to understand than it is to formally explain (you can also install the restoverlay unit tests from Apposite to see examples of the feature)...

One more thing, since both representations have unique identities (the withTransform request being the identity of the mimetype transrepted representation) then they both get cached and both have chance to be reused by subsequent requests! Efficient no?



The RESTOverlay can perform dynamic content negotiation in order to locate a target endpoint that satisfies the expressed content requirements of the HTTP client (via the incoming HTTP Accept header).

If the Accept header preference is not matched by any <produces> tags and the RESTOverlay is not in strict mode then the request will be resolved to the first endpoint matching the REST path's simple grammar.

The value of the produces tag must be the mimetype which this endpoint produces.

An endpoint should only produce one mimetype.

Tag Value

A single valued mimetype.

Multiple <produces> tags and @withTransform

If multiple produces tags are specified then the Accept header preferences will be resolved against each of the <produces> mimetypes. The required mimetype resolved during this process is subsequently checked against the mimetype returned in the endpoint's response.

In the event that the endpoint returns a mimetype that differs from the required mimetype then the response will be passed as the operand argument to the service specified by the withTransform attribute on the resolved <produces>.


Say an HTTP request has an Accept header requiring text/html...

HTTP Request:
Accept text/html

and your RESTOverlay metadata for the target endpoint has multiple <produces> tags...

<rest> ...
  <produces withTransform="active:styleMyXMLResponse">text/html</produces>

Then the request will be issued to the endpoint but by default it will return a text/xml representation.

The RESTOverlay will detect that the required mimetype is text/html. It will also determine that the <produces> tag that specified text/html has an @withTransform attribute.

To convert the received response to the required mimetype the RESTOverlay will make a request to the service specified by the withTransform attribute and provide the endpoint's response as the operand argument.

In this example it will issue a request for

active:styleMyXMLResponse+operand@response from the endpoint

The withTransform service must be able to transform the mimetype of the endpoint response to the mimetype declared by the matched <produces> tag.

Any number of <produces withTransform=""> tags may be added. The single requirement is that the withTransform service is implemented and has an active grammar that accepts the operand argument with the original response.

Data and Reality 3rd Edition: An Apology

If, after my recommendation last time, you ordered the 3rd edition of Data and Reality then I think I owe you an apology.

Whilst the editor has made well intentioned efforts to "modernize" the story with commentary, I'm afraid its a bit like trying to read Pride and Prejudice while someone constantly interrupts to tell you Lizzy Bennett is just like Miley Cyrus and Mr Darcy is Justin Bieber! Harsh, but it doesn't need a commentary - its a timeless work and stands on its own.

I strongly suggest that if you can, you try to find the 2nd edition - which used to be available as a PDF e-book. Hopefully its still available somewhere.

Have a great weekend.


Please feel free to comment on the NetKernel Forum

Follow on Twitter:

@pjr1060 for day-to-day NK/ROC updates
@netkernel for announcements
@tab1060 for the hard-core stuff

To subscribe for news and alerts

Join the NetKernel Portal to get news, announcements and extra features.

NetKernel will ROC your world

Download now
NetKernel, ROC, Resource Oriented Computing are registered trademarks of 1060 Research

© 2008-2011, 1060 Research Limited