NetKernel News Volume 7 Issue 5

November 4th 2016

• Repository Updates
• Announce: NetKernel 6.1.1 Release
• NetKernel 6.1.1 - What's New
• NetKernel: General Resource Engine - The Elevator Pitch
• NetKernel - A General Resource Engine
• NetKernel: What's in a name?

Repository Updates

This release sees an extensive set of updates and enhancements to the NetKernel system but be assured that the core: kernel, layer0 are unchanged so it is business as usual for your ROC solutions.

A detailed description of what's new is given below. Generally we have refreshed library modules such as language runtimes, resource models including new HDS Schema etc. The NetKernel tools have been enhanced with improvements to Visualizer, XUnit, Docs etc. In addition, a substantial set of new engineering overlays and features have been introduced. Jetty is updated to v9.3.12 which means HTTP/2 is now supported.

In Enterprise Edition the all new fully revised NetKernel Protocol v2 is available which supports pluggable transports including Netty, MQTT, ZeroMQ and HTTP!!

As a special bonus a brand-new Javascript language module is provided in the core distribution, based on the Nashorn Engine provided with Java 8. As well as a near-native performance boost, this offers the prospect to harness Node.js NPM libraries in modular evolvable ROC solutions with the engineering performance of the NetKernel foundation.

More details of the key features of 6.1.1 are given below. Over the next few months there will be newsletter articles with in-depth discussion of the new capabilities.

You have two paths to getting 6.1.1: You can grab a fresh distribution and install a clean instance either by direct download or by using the Gradle download task. If you're running NetKernel Enterprise Edition you can just use Apposite to seamlessly update your existing instance.

IMPORTANT NOTICE: Java 8 Requirement

As previously announced. NetKernel 6.1.1 now requires Java 8 as the minimum JVM version. Please make sure your current deployment is running Java 8 before perfoming a system update through Apposite.

Download

• NetKernel Standard Edition v6.1.1 is available here
• NetKernel Enterprise Edition v6.1.1 is available through the portal here

Update Process: NKEE 5.2.1

The following steps will seamlessly update your existing NKEE 5.2.1 instance...

1. IMPORTANT: Verify that your current NKEE 5.2.1 is running on Java 8 (check here: http://localhost:1060/tools/sysinfo )
2. Go to Apposite Admin and synchronize with repository.
3. Go to Apposite and apply all available updates.
4. Go back to Admin and sync with repo again (this detects the new distribution, verifies signatures and switches Apposite to point to the 6.1.1 repository)
5. Sync one more time (this now makes Apposite discover the updated modules available in the 6.1.1 repository)
6. Go to Apposite and apply all available updates again (this installs all the modules that have been updated in NKEE 6.1.1)
7. You are now running NKEE 6.1.1
8. Optionally you can now install the lang-javascript-nashorn and nkp2 packages to add these new capabilities which weren't part of 5.2.1

Here's a brief set of highlights of what's new with 6.1.1

Pass-by-Value Optimisation

In NetKernel pass-by-value is inverted from push to pull by constructing a value-space and making the pushed state a first class resource referenced by identity in the resource request (this article provides a detailed discussion).

You will see PBV resources in the visualizer with the URI scheme pbv:

We have been able to significantly improve the efficiency of pass-by-value caching by making the pbv: identity of the PBV resource incorporate a hash of the PBV representation. This has two effects it leads to much better key dispersal in the cache maps which makes retrieval more efficient but it also ensures that when two or more requests are passing exactly the same value, an application will leverage cached copies and any transrepted resources since they will always share exactly the same identifier.

This change is completely transparent to applications.

HDS2

Schemas can be attached to HDS2 documents or may be applied stand alone.

A schema can be attached to an IHDSDocument when it is constructed as a representation from an IHDSMutator using toDocument(). This serves the purpose to validate the document against the schema. Any client then using that document then knows that the document must be conforming to its schema.

All IHDSDocument objects have a schema, any document not specifically assigned one will get an instance of org.netkernel.mod.hds.impl.HDSNullSchema. The null schema places no constraint on the document which ensures this update has no effect on existing use of the HDS2 representations.

mod:db

By default mod:db still returns HDS1 representations which can be transrepted to HDS2. However it now also natively supports HDS2. All you need to do is set the required representation to IHDSDocument.

json:core

Now supports HDS2 to JSON bidirectional transreption.

jena:rdf

The RDF module has been updated to the latest Jena v3.1. Extensive work has been done to make Jena aware of resources in the ROC domain - so for example SPARQL queries can be applied to active resources referenced in the ROC domain, combined with web resources...

SELECT ?s ?o ?p
FROM <active:myDynamicRDFResource>, <http://host.com/sparql?foo>
WHERE ?s ?o ?p

Groovy, Javascript, Python Language Runtimes

• Groovy is updated to the latest v2.4.7 release

• Javascript Nashorn - a brand new Nashorn-based Javascript language runtime is now shipped as standard. It is faster and offers tighter integration with Java together with the ability to load() libraries with the potential to utilise the node.js NPM libraries. A runtime endpoint prototype is provided which enables any type of NetKernel endpoint to be written in Javascript. Transports, Transreptors, Accessors may all be Javascript.

• Javascript - the old Rhino based lang:javascript has been updated to the very latest (and probably last) Rhino release and is retained as an installable option since it still has a fuller feature set of the ECMA standard (eg E4X support) if required.

• Python has been updated to the very latest Jython release and now provides the full Python 2.7.0 class library.

Golden Thread Listeners

The Golden Thread endpoints in layer1 now support the ability to register Golden Thread Listeners. A listener registers a declarative request to be issued to their contextual space when a Golden Thread expires. When a Golden Thread is cut, the registered listeners on that thread will receive the asynchronous request.

This pattern enables cut/notify patterns in which the agent causing the expiration is decoupled from the system that requires notification. It also enables the a pattern for instantaneous decoupled resource recalculation upon expiration - the "Recaculating Golden Thread" pattern.

Piggyback Throttle

The NKEE architecture tools now provide a new Piggyback Throttle. This overlay can be used to wrap a space with two lines of configuration and acts as a concurrency optimisation for resources with very high demand.

For example imagine a resource that is being updated every 0.1ms which triggers a set of Golden Thread listeners to be notified. Each listener event is a new request for the same resource. By introducing the Piggyback throttle this concurrent set of resource requests for the same resource is throttled. A single request is admitted to calculate the representation, when it comes back into the throttle the representation is then given to every other pending request. The high demand concurrent requests piggyback on just one request. Another name for this overlay is the One-for-all pattern.

NetKernel Protocol - NKP2

NKP2 is a complete rewrite of NetKernel Protocol. As before it allows the ROC address space to be seamlessly distributed and conceptually looks to the ROC domain identical to an import endpoint.

The semantics of the client and server NKP endpoints has been retained and they behave exactly the same as the NKP1 generation. However most significantly the ROC endpoints are decoupled from the underlying transport mechanism. This means that your NetKernel ROC architectures can be deployed across arbitrary network protocols.

The protocol choice is declared in the configuration parameters of the client/server endpoints by declaring the required tunnel. Available tunnels include:

• Netty - highest performance and scaling and the ability to scale with NIO TLS security
• MQTT - internet of things reliable messaging
• ZeroMQ - a popular configurable MQ protocol
• HTTP - the most widely used protocol with the least operational politics for deployment

Perhaps the most powerful of these is that NKP2 now enables NK requests to be tunnelled seamlessly over HTTP. NetKernel doesn't care that we're using HTTP and HTTP doesn't care that the REST endpoints are actually talking using full NetKernel ROC requests!

We will be providing several articles on how to take advantage of the new NKP2 and how you can even write your own custom tunnels. It is, for example, even possible to send NKP over carrier pigeon. (Which given the current state of the world will appeal to the clock-turner-backers)

Visualizer

The visualizer has been restyled to be more real-estate efficient. More significantly, browser performance has been improved when viewing large traces. It used to be that from the NK side a visualizer request might be provided in a couple of 100ms. However the browser rendering of a deep request stack might take the browser up to a minute to display the trace. The browser side code has been changed so that rendering is now progressive - which means that a trace should be viewable almost instantly no matter how large it might be.

XUnit

It used to be that a set of XUnit test requests were grouped under on single request to the XUnit engine. This meant that if you wanted to debug a failing test, you had to hunt inside the visualizer trace to find the one test request that you cared about. XUnit execution now decouples each individual test request and issues them as root requests to the target test space. What this means is that every test request will be captured in the visualizer on its own. This makes a very big improvement to usefulness of the visualizer while executing development tests.

Install Portability

Finally, we have normalized the install dependencies so that all references are relative to the install directory. There are no hard coded paths anywhere in NetKernel. This means you can copy a NetKernel install from one place to another and it will just run.

This is very useful for quick and dirty deployment. Zip up NK. Copy to server. Unzip. Run.

But probably most importantly it means that if you have a problem and need support - you can just send us a zip of your full local install and we can fire it up and work through with you. No more guessing.

Hope you like 6.1.1. Its been a labour of love. Thanks for your continued support. We think the world is nearly ready for ROC - what do you think? Please spread the word...

I've been giving some hard thought to why its always seemed challenging to come up with a short explanation of NetKernel.

Over the summer it dawned on me that I had always assumed that it was self evident what a web browser does. But in fact it is not commonly understood that, far from being a rendering engine for HTML, the Web browser is an application specific operating system for obtaining, managing and compositing web resources.

So if you understand that the browser is a "Special Resource Engine" then wouldn't it be possible to understand the value and immense potential of a "General Resource Engine". Which is what NetKernel is.

What I realised is that in general we do not acknowledge that a browser is a Special Resource Engine - we see the rendering surface and forget about all the action going on under the hood.

So with these thoughts in mind I rewrote the introductory section in the documentation (see below).

The elevator pitch: NetKernel its the browser unchained...

Brief History

NetKernel is different. To give you a picture of how it is different here's a short story...

A computer program defines functions, invokes functions, receives values, assigns those values to variables (state) and invokes more functions with that state. There are dozens of programming languages, you choose your language, you write your program, you run your program. That's how it is, that's how it has always been.

Step forward to the early 1990's. The World Wide Web was developed as an efficient distributed information system. To begin with it was static, serving files, but soon it became dynamic - when your browser requested a URL the Web server would dynamically compute the state and return it to you.

The Web is a software system that has completely decoupled the state from the implementing functions. Nobody cares if you write your web application in PHP or Java or ... etc etc etc. The only thing that matters is the name of the resource (its URL identifier) and the state of the resource when you request it.

So what?

So what? So, the Web can be instantly updated, the Web scales, the Web absorbs change, the Web is malleable and flexible. These are properties that every well run business would like to share.

So in the early 2000's people started to think this was "a good thing". People started to talk about "Web-Services". There were some absolutely abominable technologies that got labelled with the "Web" label. Fortunately many of them died and over time people have started to realize that these weren't very web-like at all. Today there is growing interest in REST.

REST is just another name for "the properties of the Web".

End of story. Or is it...

So we can expose dynamic business information and business processes (business state) using the Web. But the dynamic information is generated with computer programs that call functions, receive values, invoke more functions etc. Because that's a "computer program" and that's how it has always been.

Not necessarily...

The Browser - A Special Resource Engine

What is a web browser? If we look at the web pages that a browser displays we might see it is an application for rendering HTML, images, styles and additionally the execution engine for associated javascript.

But look further, what if we take off the top rendering surface and look under the hood.

• The browser is an engine for accessing resource state over HTTP.
• Using DNS it resolves each URL to locate the physical network address of an endpoint.
• It constructs a request for the resource and issues that request to the physical endpoint.
• It receives a response to the request from the endpoint.
• It uses the associated response Header metadata to determine how to manage the state it receives (the resource representation).
• Based upon link resource references in the representation it manages a cascade of further resource requests for associated resources (styling, images, javacript...)
• When sufficient state has been transfered (REST) it starts the process of compositing the resource state so that it may be rendered on the top surface.
• As we interact with the render surface we trigger further requests - every time we attempt to access the state of a resource the browser looks in its local managed resource state (known as a cache) and if it finds a representation and that representation is valid (not expired) it uses that local copy instead of invoking the remote endpoint.

The web browser looks like an application for rendering web applications. In fact, under the hood, it is a Resource Engine for acquiring and managing resource state.

NetKernel - A General Resource Engine

What if we had a General Resource Engine (GRE)? What if we harnessed such an engine in any business solution? Wouldn't we start to get the economic properties of the Web appearing in any business problem?

Solutions that scale, that absorb change, that are malleable and flexible...

If you want a very simple analogy for how NetKernel works, it is a browser with the top taken off.

But NetKernel is not tied to HTTP. Its Resource Oriented Computing abstraction is a general abstraction independent of network or operating system.

It generalizes the core principles of the Web and applies them internally or externally to the fundamentals of software and software systems.

NetKernel is a General Resource Engine and enables Resource Oriented Computing.

Why is it called NetKernel? Its not "Net" the noun, as in Internet. It's "Net" the adjective, as in Net-weight or Net-profit.

Its the distillation of the Resource Oriented Abstraction into a core General Resource Engine. The NetKernel.

Look do I really have to say anything?