NetKernel News Volume 2 Issue 6

November 26th 2010

What's new this week?

Catch up on last week's news here

Repository Updates

No updates this week. Steady as you are.

Services: Scheduled Maintenance - Sunday 28th November

Please be aware that we will be undertaking scheduled maintenance updates to our cloud server infrastructure this coming Sunday 28th November.

We very occasionally see segfaults of the JVM when GCing. In staging we've isolated this to being caused by the interaction between Linux's KVM virtualized memory management and the Java GC.

We will be updating our production Linux server kernels and will therefore be needing to reboot numerous server instances.

We'll try to keep disruption to an absolute minimum but possible affected services will include: NK downloads, CSP support/licensing, the NetKernel forum, and the NetKernel apposite repositories.

Announce NetKernel West 2011: Fort Collins, Colorado, 13-14 April 2011

We can now announce that the NetKernel West 2011 conference will take place 13-14th April 2011 in Fort Collins, Colorado, USA. The conference will be preceded by a one-day NetKernel bootcamp on the 12th where you can get a fast track immersion in NK and ROC before the main event.

NetKernel West 2011
Location:Fort Collins, Colorado, USA
Conference:13-14th April 2011
NetKernel BootCamp:12th April 2011

We will be announcing venue details, accommodation and travel arrangements shortly.

Looking forward to meeting many of you face to face in April!

HTML5: Something is Happening

Something is happening, to quote his Bobness...

You hand in your ticket and you go watch the geek
who immediately walks up to you when he hears you speak
and says 'How does it feel to be such a freak?'
and you say 'impossible' as he hands you a bone
and something is happening here but you don't know what it is
do you, Mr. Jones?

So what is it that is happening?

Some call it the Web-Browser.

I know funny name. You might prefer "universal user-interface above a common runtime with network resource loading over HTTP" or UUIAACRNRLOHTTP for short.

Ridiculous as this sounds, I am prepared to go out on a limb and state publicly that "the UUIAACRNRLOHTTP could catch on".

What? You've heard of it already?...

The Future catches up with the Past

Scene 2 Fade up from black. Pete's alarm clock wakes him. Twenty years have gone by. It was all a dream. Now read on... [Legal Notice: Any allusion between Mr Jones and Microsoft in the text that follow is purely in your head.]

Just as Bob revisited Highway 61. Mr Jones has listened to the Geek. He's revisited the Web-Browser.

You know its serious when Microsoft throws Silverlight under the bus and positions HTML5 as the future of the GUI application.

This isn't just another round of the 90's browser war. When IE9 ships we'll finally be in a position where all the market leading browsers really do support a sufficiently-common and sufficiently-capable set of operations.

In short, I'm prepared to say that after 20 years marred by obfuscation, deliberate insufficiency-of-capability and deliberate subversion-of-commonality, we are at the dawn of the true Web-Browser.

Indeed for the first time ever in the Web client space, we are witnessing all the main browser providers actually competing to cover the HTML5 standard feature set! Even though HTML5 is a work-in-progress.

I know, the IT industry hyper-ventilates over nearly every line of code that gets written.

But this time there really is something to be excited about. A true, universal(-enough) Web Browser holds the prospect of truly exciting progress for the Web and a driver to move to larger, more sophisticated, more radical "applications".

Begin Voice over, Pete's internal monologue: This is great news for me. More sophisticated, radical systems will hit the economic glass ceiling of the complexity and costs of classical server-side software architecture. The driver of "distributed universal resource oriented applications" will finally drop the penny. I can hear them now, "Wouldn't it be cool to take resource orientation inside, to the server-side; we'd have a universal abstraction that spans the cloud..."

Sheesh. Some times over the last ten years its been like being one of those Kaminoan alien engineers preparing the Clone Army. "Your Resource Oriented Computing platform is ready, we have been waiting."


Now don't get me wrong, don't confuse my excitement over the future potential of a universal(-enough) client with excitement over the technical basis for the (approximate-)universality: viz HTML5.

HTML5 is a set of necessary enhancements, the sort of line items that should have you muttering "Yep, check". For a quick and easy tour try this:

If, having toured the HTML5 features, you find yourself whooping "Wow, AWESOME!!!!", you may be a victim of 20-years of "browser-abuse" and should consider joining a class action suit. I found myself going "Yep, check" but inside I was mostly going "about bloody time"

In short HTML5 leaves you feeling that a client implementing the stuff would finally deserve to be considered a universal client.

NetKernel: HTML5-ready?

Having got all high and mighty ('Oh my, we have been waiting')! Its time to come down a peg or two. To consider if we can say "NetKernel is HTML5-ready"?

Well, for the most part HTML5 involves changes to the client-side processing of HTML, CSS and Javascript. Which an old ROC lag, such as myself, can casually dismiss with "No problem. They're just resources. Got that covered".

And so we return to twiddle our thumbs, resting on our laurels and waiting for the glass ceiling to give out...

..until you think a bit and work out that within HTML5 lurks an elephant trap. I am talking about WebSockets.

WebSockets a Primer

WebSockets allow a Browser to initiate an HTTP request to a server and request that the server keep the connection open. The open connection becomes a bi-directional asynchronous message "pipe" through which the browser, or the server, may send the other party messages.

An HTML5 browser provides a Javascript WebSocket object to which handler functions may be bound for "onopen", "onclose", "onmessage" and "onerror" events.

Both the Client and Server-sides of a WebSocket are free to close it at any time.

The messages you can send are not prescribed. Essentially you can consider them as UTF-8 Strings (at least today and for the foreseeable future). You're free to implement any message format and application level message exchange protocol you like. (as we'll see below, it can be quite convenient to use JSON as the message exchange format, but if you've got a hankering to send CSV you're more than welcome).

The initiating HTTP request which is held open can be considered as something like a meta-data envelope and provides all the usual web headers as well as Cookies etc.

Just to make sure you're aware of what you're undertaking, the WebSocket requires that you use the ws: or wss: scheme name, instead of http: However, in order to seamlessly sneak through proxy's etc the browser actually makes the HTTP request to a regular http: URL (replacing the ws: scheme).

Just for your programming pleasure, it turns out that, for the browser's I've tried, they're not smart enough to accept the WebSocket as a relative URI. So you either have to hard code the server name into the Javascript (yueck) or use document.location to generate the server part of the ws: URI manually. (Who was that fool ranting on about the dawn of the Universal-Client?).

Elephant Trap

Let me repeat a couple of things. The client initiates the opening of a WebSocket, but the events are asynchronous and may be independently generated by either the client or server. To the message sender, they are fire-and-forget, not request-response.

Furthermore, unlike the HTTP requests you know and love, there is zero metadata associated with the messages. You receive raw message data. No caching directives, no mimetypes, no authorization headers. Zilch, nothing. Oh yeah, no resource identifier and no verbs.

Cosy RESTafarians take note: WebSockets ain't REST.

This is a radical extension to the Web architecture. This is no longer resource oriented architecture - this is asynchronous message passing.

And Relax

Boeing Dreamliner Flight Deck: Not powered by
NetKernel and HTML5 (yet!)

Fortunately, we of the NetKernel-ilk, are not religious zealots hell-bent on RESTification. In fact, as you know by now, much of the debate in the REST-tribe gets thoroughly bound up with angels on the heads of pins and misses the big picture of ROC.

We ROCers understand that there are some very good use cases where rapid remote server-side state updates must be expedited to the client in near-real-time.

God forbid anyone mentions stock tickers - lets be more creative please. What about an emergency response center coordinating ambulance/police/fire call outs? No? Then what about the Boeing Dreamliner's HTML5 flight-deck system (No! I made that up).

For these scenarios WebSockets make perfect sense and actually considerably improve the scaling and server-side efficiency compared to polling AJAX patterns.

WebSockets On NetKernel

Well I wouldn't have subjected you to all this pontification if I didn't have something real for you. Vapourware is not our style. So here you go, point your WebSocket enabled HTML5 browsers at these demos...

[As of writing this, use the latest versions of one of: Chrome, Safari, Opera, Firefox 4 Beta]

The first demo is a simple helloWorld - the client HTML page opens a WebSocket, sends a JSON message with a count value, NetKernel receives the JSON message, parses (transrepts it) and then sends count messages back, NetKernel then closes the socket.

The second demo is your standard "ChatRoom" (yawn). Loading the HTML framing page automatically opens a WebSocket to the NetKernel application and NetKernel sends back a set of recent messages from its history buffer. Any number of people will be joined to the chatroom by requesting the page (open a second browser window to the same URL to join twice and simulate two people).

Whenever anyone starts to type, a message is sent to NK and NK notifies all the other people that person X is about to say something profound. When X submits their form, a message is sent to NK, which then distributes it to all the other people. NK also makes an ROC request to an Eliza endpoint with the message. NetKernel then tells everyone in the chatroom what "Sigmund" thinks.

Yes friends, in the age of the public stream of consciousness this is Social TherapyTM (when this gets patented by some Zuckerbergian new ager tell them you saw it here first and explain prior art to them. If that upsets them, ask them to tell you about their father).

Early-Access Download

So you can see WebSockets on NK are for real and with them NetKernel is really HTML5-ready. You can start to work with them now. Here's a NetKernel package which you can download...

Install it with Apposite, using the "Upload" package feature. It installs three modules

  1. urn:org:netkernel:tpt:http:II - this is a new version of the HTTP transport, its backwards compatible with the existing transport but now adds WebSocket support (see below)
  2. urn:org:ten60:netkernel:fulcrum:expt - an independent HTTP fulcrum that runs the new HTTP transport on 8088
  3. urn:org:netkernel:demo:websockets - the two websockets demos

Note: The demo uses the json-core library and package upload does not yet support dependencies. So make sure you have json-core installed with apposite first.

After installing you'll be able to use them on your local machine with...

Technical Details

The new HTTP library is architecturally/logically backwards compatible with the older version - so instantiating its HTTPTransport and HTTPBridge gives you exactly the same httpRequest:/ and httpResponse:/ address spaces as you're used to. So applications won't notice any difference.

However, to implement WebSockets it now uses the latest Jetty 7.2.0. There are some consequences in this. Notably the XML configurations in the Front-End and Backend-Fulcrums will need to be modified - hence why we are providing an independent experimental fulcrum for you to play with this new generation of HTTP transport whilst safely leaving your existing fulcrums untouched.

To try out your existing applications against the new transport you can change their dynamic import hook to HTTPFulcrumExpt

Updating the core HTTP server is not done lightly. We need to do some thorough testing and will move our production servers over to it to make sure there are no unforeseen side-effects before we make this the new default tpt-http and make it available through the repositories (together with configuration updates to the front and back-end fulcrums).

Please let us know if you find anything out of the ordinary.

Architecture of ROC WebSockets

In the previous section I discussed how WebSockets are fundamentally not a Resource Oriented web technology. So now I need to explain how NetKernel and ROC has allowed us to make WebSockets look and feel resource oriented. Contradiction? No...

The pattern we've used is to model each server-side WebSocket as an ROC transport in its own transient fulcrum. Every time a new client websocket request comes in the HTTPBridge "spawns" a fulcrum containing a WebSocket transport. The figures below show the pattern in detail

Regular HTTP Request

You'll recall that a regular HTTP request is resolved to the HTTPBridge. The bridge constructs a transient address space which contains the httpRequest: and httpResponse: resource sets. Each and every HTTP request has its own independent bridged address space.

The HTTPBridge constructs a request for the normalized identifier res:/PATH/ (derived from the external URL received by the transport request). The normalized request is issued into the overlayed Application Space - where you can implement an endpoint etc.

This pattern allows the denormalized HTTP state to be treated as a uniform set of resources. Within your application endpoint you can SOURCE from the httpRequest: and SINK to the httpResponse: as required.

Nothing new so far - this is how the NK4 HTTPBridge has always worked.

WebSocket Step 1: Create the raw WebSocket

When the HTTPTransport receives a WebSocket request, it looks like a regular HTTP request and has a URL of http://host/path. (it looks like a regular http request to get through proxy's etc as discussed in the primer section above)

However it includes a header requesting that the protocol be upgraded to a WebSocket. The HTTPTransport now upgrades the http request to a WebSocket.

As in the standard case the HTTPTransport now issues a request into the NK fulcrum address space, only it now includes not only the HTTPServlet request (for the external envelope HTTP headers etc) it also includes a WebSocket object.

As with the Client-side JavaScript WebSocket, this server-side WebSocket has methods for open, message, disconnect events. It also has an Outbound interface through which a message can be sent from the server to the client. The server can also close the socket at any time.

As with a regular HTTP request, if you want direct access to this to-the-metal POJO you're welcome to deal with it directly in an endpoint of your own. However, as with the regular HTTP request, the HTTPBridge will now help out. It can normalize the raw WebSocket and provide a uniform Resource Oriented interaction model.

WebSocket Step 2: HTTPBridge creates a WebSocket Transport and spawns a Fulcrum

In this figure we see the HTTPBridge has received a request with a raw WebSocket associated with it. The bridge creates a new space "WebSocket Fulcrum" and populates that space with a WebSocketTransport.

Raw WebSocket events are translated by the WebSocketTransport into ROC requests into this new WebSocketFulcrum.


Just like with a regular HTTP request, the WebSocketFulcrum space still provides the full httpRequest: resource set. It therefore allows you to access all of the initial HTTP state provided with the initiating HTTP request (eg httpRequest:/header/xxx, or httpRequest:/param/yyy).

Importantly this means that any authentication or correlated Cookie state that has been established for the containing HTTP envelope, is accessible to your endpoint.

So if you have a GateKeeper or Session overlay around your application, they do not need to be changed, they can continue to use the httpRequest:/ to determine authorization headers and correlate cookie session state etc.

In short implementing WebSockets in your secured web application is seamless, the external architecture stays the same.

One thing that is different is that a WebSocket has no httpResponse: space - since at the HTTP level, there is nothing to write a response to. The protocol was switched when the HTTPTranport upgraded the connection to a WebSocket. Therefore the HTTPBridge, having spawned the new WebSocketTransport, now doesn't issue any request into the application space, it just returns a response to the HTTPBridge.

Spawned and Ready: WebSocket Applications Start Here

At this point, you can consider that the HTTPTransport and HTTPBridge's work is done.

If you try out the demos, you'll see that all of this plumbing "just works" - there's nothing to configure, it just gets taken care of and your application is effectively now exposed to two Fulcrums - the regular HTTP transport requests as always, plus the new WebSocketFulcrum.

So what does the WebSocketTransport do?

WebSocket Step 3: WebSocket ROC Requests

This diagram shows the relationship between the new WSFulcrum its WSTransport and your application space.

As we've already discussed a WebSocket is asynchronous and can generate events for "onopen", "onmessage" and "ondisconnect". Plus, the serverside can issue messages and disconnect the socket at any time.

The table below shows how we've modeled these events as ROC requests that you can handle with your application endpoint.

WS Event ROC Verb ROC Request Primary Argument Response
ONOPEN NEW ws:/PATH (where PATH is the same as the PATH in the envelope HTTP request, (ie the same as in the client-side WebSocket URL).

WebSocket.Outbound - a POJO to which you can call sendMessage() or disconnect();

Optional: a String unique identifier for this WebSocket. If a null is returned, the WebSocket transport will create its own unique identifier. (Hereafter we'll refer to this identifier as ''XYZ'')
ONMESSAGE SINK ws:/PATH+socketid@XYZ String. The received message as a unicode String. ---
ONDISCONNECT DELETE ws:/PATH+socketid@XYZ --- A boolean true if the DELETE is successful.


Whenever the WSTransport issues a request your receiving endpoint can deal with it in any way it chooses to support the application model it implements.

You'll see in the demos you can do request-response patterns, pub-sub patterns and server-side event generation patterns. All these are possible and are up to you to implement and boil down to how you choose to deal with the NEW,SINK,DELETE requests that originate from the WSTransport.

But if from the server side, you want to send a message or disconnect the socket, you have two choices.

Firstly, you have contextual access to the WebSocket. Whenever you are handling a NEW,SINK or DELETE request from the transport you can SINK to the wsResponse:/ space

  • wsResponse:/message - SINK a String to this resource to send a message back to the same WebSocket
  • wsResponse:/disconnect - SINK a boolean true, to force the current WebSocket to close. (Note this will also trigger the WSTransport to issue a DELETE ws:/PATH+socketid@XYZ request back to you so you can clean up any state you might have for that socket)

In addition, you can see that WebSockets almost by definition require you to hold server-side state. This state is correlated using the unique socketid@XYZ argument. When the NEW request comes in when the WebSocket first connects, you are free to hold on to a reference to the Outbound POJO (presumably you'd store it under the socketid you create for the NEW response - see the groupTherapy demo for example).

You can therefore implement your endpoint with an ROC interface that allows you to send or disconnect a given socketid based on any application generated state change you choose.

If this all sounds too detailed, don't worry, look at the code in the two demo endpoints and you'll see that its actually really pretty simple stuff to deal with.

WebSockets means many Fulcrums

You're probably comfortable with the idea of the Fulcrum pattern as the host space for a transport. You'll probably also usually have relatively few in your system and set up your imports so that your applications are exposed to those fulcrums (and their transport originated root requests).

This diagram shows that with the WebSocket model, every external WebSocket from a remote client means there is an independent fulcrum with a transport for that WebSocket!

This is probably not how you've thought about ROC spaces before. But essentially this is an example of exposing your application simultaneously to many many fulcrums (as many as you like).

But you don't need to worry about this. Spaces are very cheap. From the Kernel/ROC system point of view a space is as cheap to instantiate as a request!

Message Format: JSON Resource Model

As we discussed above. The WebSocket standard does not prescribe the format of your messages. In the demos you'll see that we've used JSON messages.

JSON is very easy to create and process in the client-side Javsacript code. HTML5 now provides the JSON object which offers stringify() and parse() methods to quickly serialize and parse JSON serializations.

In the demo's you'll see examples like this..

    var data={ type: "foo", message: "bar" }

Where JSON.stringify() serializes the JSON data for the wire like this...

{"type":"foo", "message":"bar"}

Which is the message received by the server-side WebSocket.

With NK's json:core library, its just as easy to deal with on the server-side.

Just import urn:org:netkernel:json:core to your application space and then in your endpoint you can just source the message as a JSONObject like this, which will automatically transrept it...

json.type     //The "type" value
json.message  //The "message" value

WebSockets, ROC and Caching

Wow this has turned into something of an epic! If you're still following just some final things to point out.

You'll see that by placing the WebSocket into a transport, its interaction with the ROC domain is through simple requests. One obvious pattern you'd be likely to want to have is to consider the message exchange as triggering internal requests for ROC resources.

The benefit should be obvious - you can have tens or tens of thousands of open websockets all asynchronously firing into your endpoint. This endpoint can then issue a request for some internal ROC service to fulfil a response or to do a pub-sub dispatch etc.

Whatever way you choose to create outbound messages, they are resources, they can be cached and the cost of creating the message (maybe a DB call?) can be ammortized across all of the outbound sockets!

So this is a really weird situation for us ROCers. With WebSockets the Web is no longer web-like. But with the way we've integrated with NK we're resource oriented on the inside but not on the outside!

Security Considerations

One final word of caution. Keep your wits about you. WebSockets aren't like HTTP requests - they can send you any message they like and there is no authentication on a message - that will always have happened with the initialisation and is external to you.

I don't know if there are any Javascript specific WebSocket security weaknesses - but I can imagine that injecting arbitrary messages to an open WebSocket would be a real high priority vector of attack for the black-hat hackers.

So one thing to consider. You would never want to have your application's message protocol include a resource identifier within it directly.

For example, don't have message formats like this...

{ "request" : "active:sqlQuery..." }

And then have your endpoint code do this...


You're not daft - you get the picture.

As I said above, you can very happily issue resource requests to fulfil a message. Just don't do a pattern where you literally request what the client tells you to do! As your Gran would have said "You wouldn't put your head in the oven if you were asked to!"

I repeat: WebSockets are not REST so don't assume you can treat them like regular external HTTP requests!

With that warning given, go have fun and tell us if there's any issues, or if we can improve the abstraction!

ROC and Languages Status

I was going to continue with the ROC and Languages thread - but you guys have suffered enough of me for one day. Take the rest of the day off and we'll do extra next week.

Have a great weekend.


Please feel free to comment on the NetKernel Forum

Follow on Twitter:

@pjr1060 for day-to-day NK/ROC updates
@netkernel for announcements
@tab1060 for the hard-core stuff

To subscribe for news and alerts

Join the NetKernel Portal to get news, announcements and extra features.

NetKernel will ROC your world

Download now
NetKernel, ROC, Resource Oriented Computing are registered trademarks of 1060 Research

© 2008-2011, 1060 Research Limited