NetKernel News Volume 5 Issue 11

September 19th 2014

• Repository Updates
• Asynchronous Request Detection and Delegation
• NEW: Abbreviated Declarative Request Syntax
• httpRequest:__updates
• RESTAssured - REST endpoint testing DSL
• Java 6- End of Life Planning
• New - Evernote Resource Model

The following updates are available in the NKEE and NKSE repositories

• apposite
  • Compatibility updates to work with Groovy 2.3.x
• database-relational-1.17.1
  • Update to driver classloading (deals with API change between Java 6/7).
• evernote-1.1.1
  • NEW - a resource model for interacting with Evernote (see below)
• hds-g2-1.3.1
  • Improved error reporting on XPathNotFound
  • prepend() added to Mutator
  • Fixed serialization to correctly roundtrip nulls vs empty strings.
• http-client-2.20.1
  • Fixed potential leak in SINK/DELETE of HTTPState endpoint
• http-server-2.26.1
  • Ehancements to httpRequest: space
• intray-transport-1.5.1
  • Fixed configuration bug of not accepting wildcard file extension if @ext was missing on the config.
• kernel-1.35.1
  • Added support for user-side detection of asynchronous requests (see layer0)
• lang-groovy-1.15.1
  • updated to use Groovy 2.3.6
  • added source code view in toString() on the compiled groovy representation. Helps give more clarity in visualizer.
• lang-trl-1.7.1
  • added support for abbreviated declarative request syntax.
• lang-xrl-1.14.1
  • added support for abbreviated declarative request syntax.
  • added support for children="true" attribute to xrl:include to permit insertion of mutliple child fragments in one step.
• layer0-1.107.1
  • added isThisRequestAsync() method to the low-level INKFKernelContext
  • added abbreviated declarative request support (see below)
• layer1-1.50.1
  • Added detection of asynchronous requests and asynchronous delegation of requests to the DynamicImport endpoint.
• nkse-control-panel-1.38.1
• nkse-doc-content-1.49.1
  • added documentation for abbreviated declarative request
• module-standard-1.65.1
  • Added detection of asynchronous requests and asynchronous delegation of requests for standard overlays (including the mapper).
• restassured-1.2.1
  • NEW - new REST service testing framework that can be used with XUnit or as a standalone runtime.
• system-core-0.39.1
  • Improved the readability of resolution exceptions including URI decoding identifiers to be human readable.
• xml-core-2.7.1
  • Fix to use JDK TransformerFactory by default if available.
• xml-saxon-1.12.1
  • Support for returning POJO representations from XQuery
• wink-slinki-3.1.1
  • Updated to the latest io2012 HTML5 engine

The following update is available in the NKEE repository

• nkee-apposite-1.51.1
  • Compatibility updates to work with Groovy 2.3.x

Please note updates are published in the 5.2.1 repository.

Updates Summary

The summer is always a good time to keep your head down and get stuff done. We've been busy working on many areas and so there are a lot of updates today.

The details of the key new features are discussed in depth below. The headlines include:

• a comprehensive update of infrastructure endpoints/overlays to detect and delegate asynchronous requests - resulting in significant performance gains for very wide parallel processes.
• a new abbreviated syntax for declarative requests making them faster to write and clearer to read.
• a corresponding update to the validation within tools and language runtimes that use declarative requests to ensure they seamlessly detect and accept the simpler abbreviated syntax.
• a set of enhancements and tooling for the HTTP stack including full release of the RESTAssured test framework.
• update to lang-groovy to use the latest 2.3.x stable release of the Groovy language. In addition, when the system is running with debug metadata (the default) the compiled groovy representation now provides the source code as output on its toString() - ensuring that when viewed in the visualizer, a compiled representaion can be understood without changing context.

We have a number of other significant items in the works which we'll be announcing this Autumn...

In the spring we discussed Asynchronous Request Patterns. That conversation was prompted by enhancements to the HTTP-client library to support asynchronous parallelisation (with, for example, the ability to perform >20,000 concurrent requests using only the core set of ten's of kernel threads).

These highly parallelised fan-out patterns gave us reason to review the handling of asynchronous requests within the standard module's endpoints - not least because our fan-out tests were re-entrant and calling the same NetKernel instance with a corresponding 20,000 asynchronous fan-in requests.

As ever with re-entrant callbacks, you push the system to an extreme limit, and we discovered the potential for thread starvation deadlocks if extensive use of throttles were not used.

While throttling solves the fan-out-fan-in challenge - it was not an optimally engineered solution.

Being perfectionists, we've taken the time to do a detailed review and have released a set of changes that make a big step in the efficiency of massively parallel concurrent architectures.

The primary enhancement has been to the standard module overlay endpoints to detect asynchronously issued requests and correspondingly to delegate them as asynchronous sub-requests and to provide asynchronous handling/relay of the subsequent responses.

So, for example, arguably the most important architectural component, the <mapper>, now seamlessly adapts to asynchronous requests and the handling of the asynchronous response of mapped requests. The same is now also true for <import> and the dynamic import endpoint (from layer1).

Collectively this review means that widely parallel fan-out patterns, whether at the top, middle or bottom of an architecture, can be performed without needing to increase the core kernel thread pool and thereby incur operating-system-level context switching overhead.

One important enhancement to support these new capabilities has been to provide the necessary user-side ability to detect if a request has been issued to the kernel asynchronously. We have added a new method, isThisRequestAsync() to the INKFKernelContext so that advanced user-side endpoints can detect asynchronous requests if they anticipate being used in very highly parallel architectures, and so can also adapt themselves to use an asynchronous handling pattern as necessary.

Aside from this one new method - these changes make absolutely no difference to existing systems! But they make large scale parallel processing even more seamless!!

The Declarative Request is one of the most frequently used aspects of NetKernel. It is used in the mapper, the XUnit tests, the request trace tool etc, and is also the technology behind recursive language runtimes such as XRL and TRL.

I know writing XML is no-one's favourite syntax so we took on the job of seeing if we could preserve the full richness and capability of the declarative request whilst removing as much of the boilerplate as possible to provide an abbreviated syntax.

The result is a new abbreviated syntax declarative request whilst retaining 100% backwards compatibility to existing "full-form" declarative request.

Here's an example of what's now offered. Say you have a mapper endpoint that maps a request to groovy and relays some arguments like this...

can be written using the new abbreviated syntax like...

Whitespace is not significant and so newlines, tabs and indentation can be liberally mixed. For example this is equally valid...

Importantly, all of the powerful features of declarative requests are still preserved. Notably literal arguments. Here's an example of calling groovy with a literal script...

<request>
active:groovy
operator	<literal type="string">context.createResponseFrom("Hello Literals!")</literal>
</request>

To support literals and to seamlessly work with existing tools, this is still XML. Its just now the boilerplate is no longer needed and so solves 80% of problems in 10% of the time, with the benefit of higher "readability".

The declarative request's ability to specify verb, representation, to relay varargs, to provide identifier argument substitutions, add request headers etc are supported. As are the method= attributes on arguments to support the advanced representation/identity translations.

Reference

The full details of the abbreviated syntax are documented here (and locally in NetKernel here).

Please note that by necessity there are some small trade-offs to the abbreviated form - such as, if your request has significant whitespace then you must URI encode it in the declaration.

Please note, you cannot mix abbreviated and full-form syntaxes. The presence/absence of the full-form <identifier> tag is used to determine which form of syntax you are using.

Of course, the full-form declarative request remains 100% backwards compatible.

Examples

Declaring a SOURCE request to groovy with multiple arguments, varargs, representation and a request header...

<request>
   active:groovy
       operator
           res:/resources/my-script.gy
       operand	
           arg:foo
       varargs
           true
       representation
           java.lang.String
       header;name=forget-dependencies
           <literal type="string">true</literal>
</request>

Declaring a SINK request to pds...

<request>
pds:/foo/baa/baz
   primary
       httpRequest:/param/foo
   verb
       SINK	
</request>

Declaring a SOURCE request to xrl with a literal template argument and varargs (notice that XRL supports abbreviated syntax now too!)...

<request>
   active:xrl2
       template
           <literal type="xml">
           	<document>
           		<xrl:include>
				active:widget
				foo res:/resources/foo.xml
           		</xrl:include>
           	</document>
           </literal>
       varargs
           true
</request>

Note these examples use tree-like indentation and line-separators to show how argument pairs may be split across lines and indented for readability - whitespace is not significant other than as structural separators.

Tools

The new abbreviated syntax can be used anywhere you'd use a regular declarative request including:

• mapper
• pluggable-overlay
• XUnit tests
• Request trace tool
• active:xrl2
• active:trl
• etc etc...

We've released several enhancements to the HTTP tool chain...

httpRequest: updates

The httpRequest: address space is dynamically injected by the HTTPBridge. HTTP disperses its state into lots of different denormalized locations - headers, URL, hostname, cookies etc etc. The httpRequest: address space provides a clean normalized abstraction for treating all of this state as resources in the ROC domain.

Prompted by Ron Hitchens from Overstory, we've increased the coverage of this normalisation and added several new resources to the httpRequest: set including support for matrix URL parameters, URL path, query and relative URL.

We've also added support for httpRequest:/collection - which provides all the readable resources in the request in a single HDS tree-structure - enabling you to make one request to grab the lot and then select from it as needed using HDS xpath.

After accepting the updates to http-server, the full details are documented here.

RESTAssured - REST endpoint testing DSL

Ron Hitchens has also helped complete the development of the RESTAssured module. REST Assured is a groovy-based DSL that allows rich testing and assertion of requests to REST HTTP-endpoints.

The library is incorporated as a runtime active:RESTAssured and so can be used as a tool within an ROC application or can be called in an XUnit test set. Here's an example of the syntax...

bogusContentType = "application/this-is-a-bad-content-type+xml"
given()
   .body ("<content>dummy</content>")
   .contentType (bogusContentType)
.when()
   .post ("/content")
.then()
   .statusCode (415)
   .contentType ("application/vnd.errors+xml")
   .body (not (empty()))
   .body (hasXPath ("/errors/unrecognized-content-type", null, equalTo (bogusContentType)));

To use RESTAssured, install the restassured package from Apposite. After installation the documentation can be found here.

For the last three years we've specified Java 6 as the minimum base-line Java version for NetKernel. Oracle has stated that Java 6 is now end-of-life and no-longer supported. Furthermore, its becoming increasingly challenging to maintain modules that use 3rd party libraries without stepping into Java 7 dependency issues.

Inevitably we will have to transition to Java 7 as the minimum Java version at some point and its getting close. In order to prepare please note...

This is a formal official announcement that we will transition to Java 7 as the official minimum Java version from the start of 2015.

In the period up to the end of this year we will continue to release all our updates compiled to Java 6. From 1st January 2015 new features and updates will be compiled for Java 7.

As a benefit, requiring Java 7 will allow several important 3rd party libraries to be moved forward to use newer versions, which we will introduce early 2015

Our experience is that most people are now running at least Java 7 in production but please get in touch if you require assistance or help in preparing a transition plan.

Please also note this is the minimum Java version requirement - NetKernel is perfectly happy running on Java 8.

Finally, we've got a new toy for you. A comprehensive resource model for interacting with Evernote.

This library can be installed from Apposite. After installation the documentation is available here.

The module provides accessors for SOURCEing, SINKing and DELETEing notes. Notes can be created in specific notebooks with full Evernote tagging support. Note attachments can be supplied and, with transreption of resources from the ROC domain, means arbitrary ROC resources can be seamlessly attached to notes.

Full Evernote search capability is provided and discovered notes can be SOURCEd (and so cached) in the ROC domain.

Finally Evernote supports its own HTML-derived syntax for notes called ENML. We have provided a set of tools for validating ENML and a powerful tool to transform XHTML to ENML.

Therefore, as an application example, it is trivially simple to create a solution that would support creating wiki content, use the existing wiki parser tools to produce XHTML and store this as ENML notes in Evernote.

Hope you find this a fun/useful new capability!

---

Now summer is behind us, the plan is to return to a more regular newsletter schedule. Have a great weekend!

Comments

Please feel free to comment on the NetKernel Forum

Follow on Twitter:

@pjr1060 for day-to-day NK/ROC updates
@netkernel for announcements
@tab1060 for the hard-core stuff

To subscribe for news and alerts

Join the NetKernel Portal to get news, announcements and extra features.

NetKernel will ROC your world

Download now

NetKernel, ROC, Resource Oriented Computing are registered trademarks of 1060 Research