NetKernel News Volume 2 Issue 47

September 30th 2011

What's new this week?

• Repository Updates
• Announce: NetKernel Standard Edition 5.1.1 Preview 1
• Tony Blog - Enterprise Visualizer Tools
• active:trl - Text Recursion Language

Catch up on last week's news here

Repository Updates

The following update is available in the NKSE and NKEE in version 4.1.1 and 5.1.1 repositories.

• lang-trl 1.1.1
  • New active:trl the Text Recursion Language, related to XRL it allows declarative resource requests to be recursively evaluated in textual resources (see below).

Announce: NetKernel Standard Edition 5.1.1 Preview 1

We are very pleased to announce the immediate availability of 1060 NetKernel Standard Edition v5.1.1 Preview 1. Downloadable now from...

http://download.netkernel.org/nkse/

Features

This release consists of a collection of refreshes and enhancements to various packages that comprise the distribution. It is a progressive evolution and at a fundamental level it is 100% compatible with the NK 4.1.1 system.

The key areas of enhancement of the core infrastructure include a new kernel build, new cache build and layer0 with the following significant enhancements:

• Optimization of the kernel cost and expiration aggregation data structures optimised for larger request trees
• Optimization of the kernel resolution algorithm
• Optimized cache keying
• Enhanced request resolution debug tracing
• New resolution trace infrastructure to support more consistent and detailed resolution traces
  • logical endpoint resolutions shown
  • reason for no match shown

As well as enabling some enhanced tooling, these core updates have yielded some quite significant performance benefits. We think you'll notice that the system just feels "even snappier" but it's also born out in the nkperf benchmark numbers. Here's the NKMark10 score for my laptop on 4.1.1 and 5.1.1 respectively...

Yes, you read that correctly. The new kernel/cache combo has improved my score by nearly a factor of two!

Of course the NKmark is an artificial benchmark. Also with realworld applications often making lots of use of the cache its not easy to measure a representative system. However our experience is that the net effect of the changes is a very considerable performance gain.

System Enhancements

Above the core infrastructure several system tools and libraries have been updated:

• Brand new visualizer with many more tools and views
  • integration with space explorer
  • New "view callstack" view gives traditional point-in-time call stack for any request
  • New "view trace rooted here" view gives a way to manage big traces by just viewing a fragment
  • cachability traffic light indicator (with reason in request detail view)
  • request trace rendered with tree view with visible branches aiding navigation
  • faster performance with large traces
  • async/sync request icon
  • richer cross-linking with space explorer
  • convenience "clear cache" button
  • use of jQuery datatables for better layout
• Improved request trace tool using new resolution debugging
• Updates to the unit test framework
• HTTP transport update to use latest Jetty 7.4.5 with latest WebSockets support
• xml-core drops legacy Java 5 support of XMLStream (see below) lang-dpml is updated due to this transition.

Java 5 Officially End-of-Life: Please note this release marks the end of support for Java 5. Notably the XMLStream API has been removed from the xml-core library and while still supported the implementation relies on the built-in Java 6 APIs.

Documentation Refresh

This release also provides a general refresh and review of the documentation with the following updates:

• Enhanced documentation covering caching
• New documentation on understanding and resolving memory leaks
• New ROC Patterns book

Library Compatibility

Your existing applications and libraries will run on NK5 with no changes. Indeed, the majority of the existing NK 4.1.1 user-level library packages are unchanged and are now provided in the NKSE 5.1.1 Preview repository.

The detailed list of specific packages that have changed between 4.1.1 and 5.1.1 is as follows:

• apposite-1.25.1.nkp.jar
• bootloader-1.18.1.nkp.jar
• cache-se-1.5.1.nkp.jar
  • OPTIMIZE: representation cache (lower CPU use and better cache recall in some corner cases and cost threshold to discard cheap representations)
• database-relational-1.11.1.nkp.jar
  • FIX: relational database pool will incorrectly create duplicate pools under some situations
• demo-websockets-1.2.1.nkp.jar
• http-server-2.7.1.nkp.jar
  • Jetty 7.4.5 with latest Websockets
  • New RememberMe cookie implementation
• json-core-1.5.1.nkp.jar
• kernel-1.19.1.nkp.jar
  • OPTIMIZE: resolution failure exceptions only formatted on demand (over 100 RFE during out-of-box boot)
  • OPTIMIZE: kernel cost and expiration aggregation data structures optimised for larger request trees
• lang-dpml-1.16.1.nkp.jar
  • FIX: Non evaluated SOURCE of non existent arguments in DPML script would incorrectly throw exception
  • Now uses Java 6 compatible XMLStream parser
• lang-ncode-1.4.1.nkp.jar
• layer0-1.71.1.nkp.jar
  • Declarative request support for relaying requested representation by default
  • FIX: logging with null message would break log file formatting
  • FIX: mapper now relays requested representation unless overridden in declarative request
  • OPTIMIZE: validation cache (used by all system configurations with schematron validation) persisted for increased boot speed
• layer1-1.33.1.nkp.jar
  • Layer1 fls (filesystem list) accessor to support returning empty directories
• module-standard-1.50.1.nkp.jar
  • New <literal> tag in standard module to support literal resource declaration
• nkse-control-panel-1.20.1.nkp.jar
• nkse-cron-1.11.1.nkp.jar
  • FIX: CRON management view pause/resume buttons not working
• nkse-dev-tools-1.36.1.nkp.jar
  • Updated Configure NetKernel tool with additional kernel.property values and defaulting of new values
  • Logging view now shows "debug" icon for debug level log messages
  • Asynchronous news feed with new styling on management console homepage
  • Deployment editor now supports direct adding of a filesystem module (no need to edit modules.xml)
  • FIX: Double newlines appearing in scripting playpen on windows
• nkse-doc-content-1.36.1.nkp.jar
• nkse-doc-patterns-1.2.1.nkp.jar
• nkse-docs-1.16.1.nkp.jar
• nkse-http-fulcrum-backend-2.3.1.nkp.jar
• nkse-http-fulcrum-frontend-2.2.1.nkp.jar
• nkse-license-1.6.1.nkp.jar
• nkse-visualizer-1.13.1.nkp.jar
  • New complete refresh
• nkse-xunit-1.7.1.nkp.jar
  • Now clears cache before and after running a test-set but does not expire requests within a test set. This change allows for the testing of distributed caching patterns.
• photonk-0.5.1.nkp.jar
• security-core-1.4.1.nkp.jar
  • New GeneratePasswordHash/CheckPassword Hash accessors
• system-core-0.29.1.nkp.jar
• xml-core-2.1.1.nkp.jar
  • Removed legacy Java 5 impl of XMLStream API.

Migration from NK4 to NK5

Those familiar with NK's modular architecture will see that these packages could easily be deployed to an existing NK4 system to bring it up to NK5 spec. However, we have decided that whilst it would be possible to provide an in-situ migration tool to switch repositories and update an existing NK4 system, we think that a clean download and install is the better migration strategy and guarantees that you start with a clean and known system.

Preview Programme

There is very little risk in moving to NK5 since it consists of a set of independent changes all of which are logically decoupled to the ROC domain. We have been running our systems on the combined changes for several months now and all our existing systems run as before. However we believe that its prudent to have a review phase in which we hope you'll play with the Preview-1 build and let us know how you get on.

We expect this to be a very short evaluation/review period and in two weeks we'll ship a final NKSE 5.1.1 and the associated NKEE 5.1.1 release.

Tony Blog - Enterprise Visualizer Tools

Tony has been like a dog with a bone over the summer - the ongoing development and enhancement of the new Visualizer has proved to be a spur to find optimisations in the kernel and the cache - a perfect positive feedback loop.

The visualizer is an extraordinarily powerful tool. It considers that the operational state of the entire system is a set of resources and then provides a collection of tools to present uniform and cross-linked representations of orthogonal subsets of the state. In short - there is nowhere to hide - everything is revealed by this tool - everything that the kernel knew and did, is shown to you.

Some of the NK5 visualizer enhancements are in today's preview release, some others will be introduced soon via the NKEE build. But in his blog this week Tony discusses the killer new tools you can use to optimize and tune your computational state space...

http://durablescope.blogspot.com/2011/09/enterprise-visualizer-tools.html

active:trl - Text Recursion Language

Over the last week I had a very useful discussion with Joe Devon. Joe is based in LA and is the consulting architect for some seriously hard core, very high traffic web properties.

Joe's been exploring NetKernel and has become engaged with ROC's potential for systemmic caching, amongst other things. One suggestion that came up was that while Joe loved the potential to build composite web solutions with the XRL composition, he had misgivings about raising the bar for his developers to be required to provide well-formed XML fragments to combine into the site tree structure. There were also small niggles with things like setting values of attributes etc.

I took this feedback on board and, being pragmatic, decided that what I thought Joe was saying was that he wanted a textual recursive composition tool. Similar in design to XRL, but unconstrained from XML, to allow the arbitrary and recursive composition of textual resources. In short something he could use to composite the resources created by a diverse set of developer skill levels.

So there's now a new language runtime called active:trl (available in the repositories as lang-trl). Here's its documentation...

Hello ${res:/some/path/to/world.txt}

 Hello ${
 	<request>
		<identifier>active:groovy</identifier>
		<argument name="operator">res:/some/script</argument>
	</request>
 } 

As you can see it supports simple embedded tags containing resource requests (both simple and full declarative request syntax). Each sourced resource is than recursively evaluated for more tags in the same manner as XRL recursion. Therefore each referenced resource is independently cacheable. An active:trl recursion will minimize system cost by only dynamically generating those parts of the composite that need recomputing.

Currently this tool is in the "multiverse" set of apposite - indicating its preview status. However after some review and feedback we'll declare it final and make it an official tool.

Please let us know what you think and how you get on. (And thanks Joe for the pragmatic reality check).

NetKernel Europe Bootcamp - Brussels, Belgium, Thursday 27th October 2011

Sponsored by Steria Benelux

http://www.1060research.com/conference/NKEU2011/

The event takes place on Thursday 27th October. If you've registered already, thank you! We will soon be getting in touch with the details so you can plan your travel.

If you want to take advantage of face-to-face opportunity we will also be around on both Wednesday 26th and Friday 28th for meetings etc.

We're looking forward to seeing you in Brussels.