NetKernel News Volume 2 Issue 32

June 10th 2011

What's new this week?

• Repository Updates
• Kernel Update / Efficiency Enhancements
• World IPv6 Day
• nCoDE in Practice - DropBox Demo
• nCoDE ≠ Code Generation, nCoDE ≡ Resource Composition
• Brian Sletten - Semantic Architecture
• Euro-Bootcamp

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repository...

• json-core 1.4.1
  • JSONFromHDS now returns an empty json object {} in the degenerate case rather than a NPE
• kernel 1.15.1
  • Enhancements: Implements EXPIRY_POLL_DEPENDENT and dependency rollups (see below)
• layer0 1.63.1
  • Added EXPIRY_POLL_DEPENDENT to NKF
  • Grammar bug fixed. Parts are now being matched against a correctly encoded input rather than unencoded as they were.
• layer1 1.29.1
  • Dynamic Import will now react instantly to changes to configuration
  • SpaceHDSAggregator uses optimised expiry function EXPIRY_POLL_DEPENDENT
• module-standard 1.46.1
  • Standard space will not re-evaluate META data every time if it expires too many times in too short a period. This stops slow startup in some pathological cases.
  • Standard space drops binding expiration functions once it is fully bound
  • Optimised resolution of requests for endpoint metadata in standard space.
  • Transreptor base class now detects un-constrained transreptor and gives warning and makes them unresolvable to prevent inadvertent application failures.
• nkse-dev-tools 1.32.1
  • New kernel properties added to kernel configuration editor
• nkse-doc-content 1.29.1
  • Documented EXPIRY_POLL_DEPENDENT
• nkse-xunit 1.6.1
  • XUnit now catches Throwable in its Exception handler so that stack recursion tests can be implemented. Serious system level Throwables are detected and thrown as before.
• rdf-rdfa 1.3.1
  • Update to use 0.4.1 rdfa parser library, fix to more intelligently select a default base URI if no argument provided.
• system-core 0.24.1
  • ModuleListAccessor uses optimised expiry function EXPIRY_POLL_DEPENDENT
  • Prevent an insignificant and ignorable potential NPE during binding triggered if an irrelevant space change event is received. Thanks to Carl Conradie of Momentum Insurance for reporting this.

Kernel Update / Efficiency Enhancements

Today's update to the kernel implementation is the first change for over a year. The new kernel incorporates a couple of relatively small optimisations that yield quite significant system-wide benefits.

The first is an enhancement which optimises the rollup of dependencies for MaxConstantDependent and MinConstantDependent expiries. In short it introduces a small and simple change to eliminate holding on to unnecessary dependencies that can have no bearing on expiry.

Small enhancements like this can result in broad returns - but not necessarily at the application level, after-all these particular expiry functions are somewhat obscure for application uses. However, as I've discussed before, the entire spacial architecture of a running NK system is a Resource Oriented solution.

NetKernel uses ROC to boot ROC.

For example from a classical perspective, patterns like the Dynamic Import are, practically speaking, not possible (Order Exponential complexity) since they require an exponentially expanding set of indeterminate transient boot state for every space in the system to be recursively and repeatedly visited in order to discover a stable steady-state arrangement.

However, by treating the spacial state as resources, it is possible to solve the spacial relationship algorithm in linear time by caching transient boot state as reusable representations during the boot/reconfiguration binding phase. So NetKernel boots the higher order ROC domain by progressively annealing the state of the spacial resources.

Why am I explaining this? Well during the spacial binding cycle, we use MaxConstantDependent expiry functions, to transiently hold "good enough" representations of the state of the spacial configuration. But once the system has transitioned to steady-state, much of the spacial resource dependency model that was necessary for boot is no longer relevant for the steady-state stable spacial relationships. So the update to the timed-dependent expiries is able to work out the unreachable dependencies and roll them up, effectively normalising the dependency relations.

Previously the full transient dependency model was retained and, since it was part of the ROC spacial state, became part of the dependency relations for application-level representations. Hence if you looked at a deeply tiered application in the visualizer you might have seen some very large and complex dependency relations in the request view.

The kernel optimisation is able to rollup (drop) extraneous dependencies for this type of expiry. Which, due to the ROC nature of the whole system has the following benefits:

• Architectural dependency relations are no longer mixed into application-level dependency models, which means...
  • Expiry determination is faster on most application representations.
  • Smaller memory footprint since unnecessary object references are eliminated for transient boot state.
• Summary: Solutions run even faster with even less memory.

New Expiry Method: EXPIRY_POLLED_DEPENDENT

In addition to the changes outlined above, the second kernel change introduces a new expiry function EXPIRY_POLLED_DEPENDENT. Again the motivation was driven by optimisation patterns in the bootstrapping and reconfiguration of the ROC address space - but it may also prove useful in application level patterns.

The essence of this expiry function is that it provides a regular dependent expiry function (valid so long as all sub-requests are also valid) but it has a transient poll hysteresis whereby the dependency tree is only checked for validity periodically.

The point of this is that there are situations in which you may have very large dependency trees, but which you only require approximate consistency with and, for the sake of efficiency and for transient stability you want to smooth over the potential noise of instantaneous expiration.

By default the poll period is taken from the kernel property netkernel.poll.internal - this can also be set in the Kernel config tool in the Internal Poll field (default is 500ms). Also there's another field you can tweak Initialisation Stability Threshold which defaults to 10 - this just allows you to control chaff of "false warnings" during the transient binding process.

The net effect of these optimisations are that the system is faster, smaller footprint and visualizations of applications will be less cluttered by "irrelevant" transient dependencies and so be more easily inspected and explored.

Additional Updates

The change to layer0 was a complementary change in the same area. Standard Spaces now detect and drop unnecessary dependencies once the initial boot/reconfigure binding phase has completed. Which further reduces any application level mixing in of the architectural resource model. Also changes in layer1 and to the system tools were to switch to use the new polled dependent expiry and enhance their performance.

Other Updates

• A corner case in the Dynamic Import endpoint in which a change of state of the import configuration might potentially have led to inconsistency in the step between resolution and execution has been resolved. The Dynamic Import now quickly and consistently reconfigures based on expiration of its configuration resource.
• As with the recent consistency warning change to StandardAccessorImpl, the StandardTransreptorImpl now checks that either To and/or From representation has been declared to prevent situations where if you forget to declare these you accidentally promise to transrept everything into everything! (I shall mention no names - but this came up recently and led to some bizarre behaviour).

World IPv6 Day

Wednesday was World IPv6 Day and saw the main web service companies switch on universal IPv6 network access to begin to test the global IPv6 readiness. Given nobody noticed anything unusual it must apparently have been a success.

You'll recall that I first mentioned this event in the February 4th Newsletter. Which also explained how you can do your own local IPv6 readiness tests on your layer-7 application stack. If you didn't catch it back then, the headline is that NetKernel is IPv6 ready.

One small question remained in the testing last time, and that was how to use raw IPv6 addresses in URLs. The problem being that IPv6 makes use of colons instead of dots and that makes the separation of host and port ambiguous. So in honour of World IPv6 Day, I decided to find out the answer.

It turns out that way back in 1999 an extension to the URL specification was published as IETF RFC 2732. Which states that raw IPv6 host parts should be delimited by using square brackets.

For example, here is how to request the NK admin tools with a raw IPv6 localhost address of ::1

http://[::1]:1060/

Which proves that the server-side of NK has no issue with IPv6 initiated client requests. Also, in the last article, we showed that the http client and, for cloud ROC architectures, the NetKernel Protocol NKP-client/server are IPv6 ready too.

There was one small missing corner of our unit tests - since we didn't know how to do raw IPv6 URLs we'd used a DNS entry for ipv6-localhost (mapped to ::1). Which meant we had tests like this...

active:httpGet+url@http://ipv6-localhost:8080/test/blah

But, as you know, identifiers are important in the ROC domain. So it was pretty important to add tests for raw IPv6 addresses too - just in case the internal implementation of the HTTP client (for instance) did not implement RFC 2732. Fortunately this test passes and shows that we're all clear...

active:httpGet+url@http://::1:1060/

So I am happy to announce that even for raw IPv6 addressing, NetKernel is IPv6 ready on both server and client-sides.

nCoDE in Practice - DropBox Demo

As you know, Tony has resurfaced from the intense rebuild of DPML. He's stepped-over to the user-side and has been playing with a pure nCoDE implementation of a web-application-front-end to provide a file-browser over a Dropbox store.

The demo shows how apparently complex interactions, like OAuth delegated authentication, can be readily implemented in a simple-to-understand nCoDE process model. The first instalment is here...

http://durablescope.blogspot.com/2011/06/dropbox-with-ncode.html

The second instalment which gets down to details is here...

http://durablescope.blogspot.com/2011/06/dropbox-with-ncode-part-2.html

nCoDE ≠ Code Generation, nCoDE ≡ Resource Composition

Incidentally, I had a moment of clarity while talking over an nCoDE demo with Brian (see next item). Brian observed that graphical composition tools are not new. But that nCoDE is completely different in that it is reflecting and sequencing the spacial context which was already present in the ROC address space.

Previous GUI tools have only provided a tool-box which when executed would generate and run classical code. It follows that such tools have been limited and required conformity to a strict API in order to extend and add new capabilities.

nCoDE provides a live visual representation of the addressable services in your address space and very simply allows you to orchestrate those services. If you add new services to a space they instantly appear as components for use in nCoDE. Indeed assemblies you build in nCoDE become services in the address space, are therefore reflected back into nCoDE.

This is a resource oriented development environment.

Brian Sletten - Semantic Architecture

On Monday, Brian gave a full-morning session on ROC and Semantic Architecture at the Semantic Technology 2011 conference in San Francisco. It started with REST, introduced ROC, showed details of NetKernel and then finally demonstrated how to progressively implement a fully dynamic architecture.

The elegance of the composite architecture demo is that all of the architectural features (throttle, access control and actual service interfaces) are dynamically driven from the underlying resources and in-particular RDF models.

I hear that to many in the audience this seemed to be like "advanced magic". But since you know NetKernel, you'll appreciate that the services were instantaneously conjured by dynamically generating a mapper configuration resource. So changes to the resource state led to the instantaneous appearance of a logical endpoint with a service interface grammar and mapped resource request.

Equally the access control layer was just implemented with the pluggable-overlay and the preProcess request was handled by a dynamic router which determined access control rules based upon the requested resource and its access policy - a derivative resource driven off the data model.

You'll also understand why, last week, I needed to tweak the throttle so that it really did respond to a dynamic throttle configuration resource.

Slides

I know when he gets chance, Brian plans to tidy up his implementation and will make it available. In the mean time here's his presentation...

http://bosatsu.net/talks/sletten-semantic-architecture.pdf

It includes some excellent summary slides on REST and very clearly shows how ROC is a progressive generalisation.

Tweets

Being on another continent, it was great to see the stir of excitement the talk/demo generated. Here's a sample of the tweets that it elicited...

@sortelli: #netkernel is the most exciting thing I have learned about in ages. Thanks to @bsletten for his great presentation. @cjmconnors: "When you guys are right, you are RIGHT. The past 5 minutes of @bsletten's talk is worth the whole trip for me." #netkernel @joedevon: .@bsletten just blew my mind again. I can see so many uses for Netkernel. Damn this is good. #semtech @shangz: should spend some time on netkernel, generating quite a bit buzz during #semtech

Incidentally, I know several new people signed-up for the NK portal and hence will have received this newsletter for the first time today. In case it occurred to you that Brian is some kind of NK astroturf jockey, its on record that he is entirely independent. It follows that we really really appreciate his public advocacy and evangelism of ROC and NetKernel.

Euro-Bootcamp

The Bootcamp that preceded the NK West conference was well attended and proved to be an effective way for many people to get their feet wet with NetKernel and ROC. We also know that due to the travel logistics many people outside of the US couldn't make the conference. So we're wondering if there'd be interest in us doing a one or two day bootcamp over on the East-side of the Atlantic?

This is a tentative idea at the moment and we're trying to gauge interest. Off the top of my head it'd likely be in the October time frame and somewhere easily accessible in Europe. (Following the great success of the beer-related distractions of Fort Collins - Munich and the Oktoberfest would provide a suitable equivalent ;-)

Drop me a note if this interests you and, all things being equal, we'll arrange things accordingly.