NetKernel News Volume 1 Issue 26

April 30th 2010

What's new this week?

• xml-saxon update.
• NKP no show.
• 'Redirects' what's all that about?

Repository Updates

• xml-saxon: Chris Cormack, the maintainer of xml-saxon, has published some tweaks. Updated to use Saxon HE 9.2.0.6, fixed a possible NullPointer in the catalog code and removed a println in the catalog code.

NKP Status Report

We've been pushing the envelope with NKP. We had a candidate release prepared and to push it to the limits have implemented a general pubsub framework with it. This shows that NKP is able to offer bi-directional channels without the need for a callback server interface on the subscribers end. (contrast with pubsubhubbub)

Implementing pubusb was outside of the original design goals and core use cases that we wanted to cover. But it helped to highlight that, whilst from the application level's point of view we can cleanly implement the pattern, at the low-level we needed a more sophisticated infrastructure.

In particular, each subscriber required a waiting thread in the server-side hub. For small scale scenarios this is sort of ok, but we aim for perfection, so we're close to completing a second generation implementation with a true threadless asynchronous architecture. This will mean that NKP will handle very large numbers of concurrent requests, including very long-lived requests, with a small impedance matched set of threads corresponding to available cores.

So, sorry to say, nothing to play with yet.

[Incidentally, did you know that the NKEE enterprise http server provides com.ten60.netkernel.tpt.http.NetKernelHandlerAsync. If you switch to this handler in your etc/HTTPServerConfig.xml and combine it with a throttle overlay after the HTTP transport, you have a threadless async frontend for HTTP. Talk to us if you want to know more about how to set this up to handle extremely large HTTP concurrency.]

Redirects in ROC - demo-redirect

We recently had a discussion about whether it made sense to implement a redirect capability into the NK protocol. A la HTTP's 30x response codes and, at the Web-application level, the browser's detection of redirects in the <meta> Refresh header of HTML representations.

It turns out that given that NKP transparently projects the ROC address space abstraction between systems, and, ROC can treat requests/identifiers as resources, then it doesn't make any sense to make special cases in the protocol of what are really general ROC patterns.

To see what I mean. I knocked up a demo/tutorial "demo-redirect" (available in the 4.1.0 apposite repos) which shows 4 different patterns for implementing redirects in the ROC domain (and hence can be used for NKP architectures too). [Note these are ROC redirect patterns - nothing at all to do with HTTP]

A static copy of the documentation for the demo is available here...

http://docs.netkernel.org/book/view/book:urn:org:netkernel:demo:redirect:guide/

(Note, the demo's don't work in the static copy - you'll need to install 'demo-redirect' with apposite to try it for real)

The demo is fairly mechanistic, in that, it shows the different physical ways that you could implement redirection. What's not addressed is the philosophical perspective. For example why doesn't NetKernel implement redirect handling of requests at the kernel/protocol level? It surely could.

The answer to this lies in the old adage: one man's metadata is another man's data. A redirect is a resource representation, the abstraction's job is to reify resource representations. Things start to get very muddy very quickly if we start to pollute layers and introduce hard-coded context and/or semantic interpretation of resource values at the kernel level.

As you'll see from the demo, its really very simple to introduce architectural patterns that allow the application layer to perform redirects. That being so, its cleaner, and ultimately more consistent and efficient (in terms of the re-use of computed state etc) to make it an application architecture design decision.

Next question is, what purpose do redirects play in the ROC domain?

Probably the most important thing that redirects offer is the ability for an application architecture to be decentralized (see search example below).

But, as I mentioned previously, identifiers are relative, therefore being able to redirect allows stable names to be used for transient contextually defined identifiers.

Very interestingly, redirection allows loose, "approximate identifiers", to be progressively tightened into more and more definitive and authoritative identifiers. (That could be a semi-formal definition of what we mean by "search".)

In fact we've used approximate identifier patterns in a number of parts of NetKernel. For example did you know that when you search, all the results are actually redirection resources that only point to a resolver for the specific category of documents?

When you click the search result you request the resolver which knows about the rendering interface and redirects you to the identifier of the rendered resource. With this approach, the search sub-system's index generation is focused souly on the textual source to index and has no connection with the (completely unrelated) reification of HTML representation of the content.

Another example in the doc system is the {luckysearch} wiki macro engine we provide. For example this...

"The {luckysearch}declarative request{/luckysearch} can be used to implement redirect patterns"

Would replace the {luckysearch} with a link to the lucky search engine with the specified term as its argument. The lucky search engine is just the standard search tool which selects the first search hit and returns a redirect resource to take you to it. The net effect of this is that we have hypertext linkage capability within the system that is entirely approximate! But, if the approximate identifier (search term), is sufficiently well defined, it's as good as a definitive hard-coded link!

As you can see, using approximate identifiers, context and redirection leads to incredibly robust long-lived systems. (Dare I say "loosely coupled" - hell no, they're "decoupled").

Long story, but hopefully yet more grist to the mill of what we mean when we say that identifiers are relative.

Now something for you to worry about: if you extrapolate the philosophical implications, you'll quickly realise that relative identity leads to an infinite contextual regression. (I said its weird).

Now to stop worrying: While this is true, its no concern as the basis for a real system since you can always introduce constraint boundaries that "absolutize" the context (ie isolate it into a bounded set - that's what DNS and the Web does. We don't get hung up on the parallel intranet address spaces). The beauty of NK and ROC is that unlike the web, you can place the constraint where you want and allow address spaces and context to flexibly and dynamically converge.

Next week, I'll show you some patterns with ROC (using NKP) that show how the context established by a client associated with two (independent) downstream NK servers allows each party to acquire the aggregate address space of all three. Makes pubsub look positively primitive. (BJ are you watching ;-).

Have a great weekend.