NetKernel News Volume 2 Issue 14

January 28th 2011

What's new this week?

• Repository Updates
• NetKernel West 2011 - Everything you ever wanted to know about caching
• Advanced HTTP Handlers - *New* NetKernelSecurityHandler
• Asynchronous HTTP Handler
• SOAP Client Enhancement
• HTTP Client Review
• XML Serializer
• Resource Stability - Your Rights and Responsibilities

Catch up on last week's news here

Repository Updates

The following updates are available in both the NKSE and NKEE repositories...

• soap-ws 1.5.1
  • Enhancement to support http:client config and state arguments (see below)
• xml-core 1.9.1
  • Workaround for Java regression of default indent size in JAXP serializer (see below)

The following update is available in the NKEE repository...

• nkee-http-server 2.2.1
  • New feature: NetKernelSecurityHandler (see below)

NetKernel West 2011 - Everything you ever wanted to know about caching

Out-the-box you can just use NetKernel and it will start to cache the representations you produce without you giving it any thought. As you get more advanced, or start to consider distributed resources, then it helps to understand some of the properties of the cache and the patterns that will maximize its benefit.

A new talk has been scheduled for the conference "Everything you ever wanted to know about caching". In which Tony will discuss the caching model and show how to optimize your architectures.

This item has been added to conference programme...

http://www.1060research.com/conference/NKWest2011/#Details

Early Bird Deadline

Early bird registration closes Monday at midnight GMT.

Advanced HTTP Handlers

The NKEE repository has an update to the nkee-http-server package which introduces a new NetKernelSecurityHandler.

You might have never looked at the default HTTP transport configuration in your http fulcrum modules. Its the file res:/etc/HTTPServerConfig.xml, and it uses Jetty's declarative XML configuration to set up the Jetty server with a handler chain containing the NetKernelHandler.

In Jetty a Handler is a component which is called by the server when an external HTTP request arrives. The handler is given references to the external HttpServletRequest and HttpServletResponse and can read/write these as required.

Your default HTTP fulcrums set up the HTTP transport with the NetKernelHandler - this component is the guts of the NK http transport and its the thing that knows about the address space and issues the ROC domain request into NetKernel.

Jetty comes with several other Handlers, predominantly to implement the Servlet abstraction. It also provides various protocol-level security handlers (for BASIC, DIGEST etc authentication).

If you know what you're doing you can use the Jetty XML configuration to create a layered handler structure on the front-end which progressively adds features to the transport. One simple example in the fulcrum config is the NCSA logger pass-through handler.

So at this low-level Jetty provides a lot options.

Security Complexity

One thing that's always been possible is to introduce the ConstaintSecurityHandler - which performs low-level protocol-level authentication and takes sets of Realms with associated authorisation methods (BASIC, DIGEST etc).

Which, since it has a bias towards implementing servlet infrastructure, can get out of hand and somewhat complicated to set up and manage.

Now complexity is not a good recipe for security.

This isn't Jetty's fault - its just the ever growing stack of "stuff" that has crept into the servlet API.

Servlet Not Required Thanks

You know that NK doesn't see the world the same way as Servlet. A servlet has a monolithic container model and you implement your stuff for the particular application inside the container (this monolithic view holds even with more modern OSGI modular servlet engines).

In ROC we see a transport as an external event generator whose job is to interpret the outside world and issue a normalized resource request into the ROC domain. The ROC address space is modular, and dynamically composed. ROC has a federation model as opposed to OO-containers monolithic one.

With this view, the NK HTTP transport can cut through Jetty's extensive and somewhat complex servlet layers. We go down to the clean, simple and efficient HTTP request metal.

HTTP Transport Security - ROC Style

So you see it has always bothered me that while you can introduce Jetty's ConstraintSecurityHandler it came with added complexity and brings with it servlet baggage that we just don't want.

I spent a day reverse-engineering the implementation details of Jetty's security handler. I realised that we can cut through to a very simple ROC-centric approach *and* retain the protocol-level implementation stuff which Jetty does so well.

I worked out we could implement an ROC login authenticator that is independent of the chosen protocol-level authentication method (BASIC, DIGEST etc). The NK login authenticator issues an ROC-domain resource request with the username and externally provided credentials. You can implement this endpoint to validate the user anyway you like, and leverage all the ROC infrastructure too.

The other good thing is that ROC naturally caches the authentications as resources - provided that they are cacheable. For example DIGEST auth can never be cached since its uses a nonce for every request, but ROC works this out since this means any two request's credentials are never the same so the authentication resource, which depends on the credentials, can never get cached.

The key take away is there is now a brain-dead simple security "overlay" which can leverage the ROC domain and which due to its simple design means you can have strong confidence in the front-line trust boundary.

There are several details, but the net result is that the nkee-http-server package now provides the NetKernelSecurityHandler. Here's an extract from some of the docs to show what it does...

username=context.source("arg:username")
creds=context.source("arg:credentials")

//BASIC Example
//rep=username.equals("pjr") && creds.equals("test")

//DIGEST Example
rep=username.equals("pjr") && creds.check("test")

//Issue the boolean response representation
context.createResponseFrom(rep)

Usage

There's an extensive example of how to deploy the NetKernelSecurityHandler in your Jetty config. After you've updated nkee-http-transport package you'll find docs here...

http://localhost:1060/book/view/book:tpt:http:com:book/doc:tpt:http:com:title

Asynchronous HTTP Handler

While we're on the subject of Jetty handlers, did you know that the nkee-http-transport has an alternative NetKernelHandler?

Its the class com.ten60.netkernel.http.transport.NetKernelHandlerAsync and it implements a threadless asynchronous ROC transport.

All ROC-requests issued into the ROC address space from this handler are issued asynchronously.

When it issues the internal ROC request, the external http-client request is suspended on a Jetty continuation. When the asynchronous NetKernel request response is received, the continuation is woken-up and the response is streamed to the remote client.

Why is this interesting? Well it gives you the foundation to implement really sophisticated front-end load shaping architectures. For example it seamlessy combines with the layer1 throttle endpoint to provide threadless asynchronous load matching to your hardware's capabilities.

Usage

After updating nkee-http-transport the documentation is available here...

http://localhost:1060/book/view/book:tpt:http:com:book/doc:tpt:http:com:async

You literally just swap out NKSE's NetKernelHander with this NetKernelHandlerAsync - they're fully equivalent in every way other than they're internal operational model.

SOAP Client Enhancement

Kevin Ansberry has provided some interesting use-cases of using legacy SOAP services in a Microsoft-shop. He discovered that some of his company's MS-based platforms (CRM etc) have NTLM protocol-level HTTP authentication.

We've therefore enhanced the ws-soap library to allow you to set up HTTP credentials in a local-client-side state resource and have the soap client pass this through to the active:httpPost endpoint which actually does the HTTP level work.

Update the package ws-soap to get these enhancements. Thanks Kevin for testing these features.

HTTP Client Review

Kevin's experience of being inside a Microsoft corporate infrastructure led me to review the NTLM support in the http client library. Whilst it currently supports NTLM I've not been happy with the implementation since its not baked into the Apache HTTP component but had to be implemented using the JCIFS library.

Since NTLM is proprietary and has no published spec its always worried me just what level of support is covered and using an arbitrary 3rd party library meant that support didn't track with the apache client updates etc.

Fortunately I just discovered that the latest 4.1.x release of Apache client now supports NTLMv1, NTLMv2 and NTLM2 (So many. Why so many Microsoft?)

We don't make radical swaps of HTTP tooling without extensive testing and living with the change in production first. So we won't ship this in the repositories just yet. But if you need/want to use a copy of the http-client module using this new 4.1.x library you can get it here...

http://temp.1060research.com/2011/01/urn.org.netkernel.client.http-trunk.jar

Just comment out the existing http-client-x.x.x reference in etc/modules.xml and point to this jar.

It passes all regression tests and I've used it with OAuth to talk with twitter and on a production server to sync and update with apposite. So I don't anticipate any problems.

XML Serializer

Finally, the xml-core package has an update. Jay Myers discovered that the default DOM serializer (transreptor) was not indenting. So XML that should have been "pretty printed" was just flat and hard to use in human-readable situations.

This was a complete surprise. All these basic JAXP tools are provided in a utility class org.netkernel.xml.util.XMLUtils in the xml-core module. This contains those accumulated nuggets of incomprehensible techniques that you learn once and never want to think about again. Since JAXP is an old API this class is at least six-years old and has not been touched since NK3 and before.

So to find that it wasn't doing what it used to was a real surprise. But how could a bug have appeared like this?

Well a quick search found this...

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6296446

It turns out that there is a regression in the JAXP serializer - someone at Sun decided that OutputKeys.INDENT should default to a zero character indent! Previously it was an indent of four. Better yet, the JAXP transformer API doesn't even let you set this value. So woe betide you if you want long term stability in your XML resource model.

Fortunately someone has published a tip on how to set the indent using a magic incantation by setting an undocumented transformer property.

So we've added this as a workaround in xml-core.

You can run this little groovy script in the scripting playpen to see the problem...

import org.netkernel.xml.util.XMLUtils

sr=new StringReader("<a><b>Indented</b></a>")
dom=XMLUtils.parse(sr)
sw=new StringWriter()
XMLUtils.toXML(sw,dom,true,true, "UTF-8")
context.createResponseFrom(sw.toString())

Which without the update gives...

<a>
<b>Indented</b>
</a>

After the xml-core workaround it gives the correct and expected 4 character indentation...

<a>
    <b>Indented</b>
</a>

Resource Stability - Your Rights and Responsibilities

This sort of Java regression really really pisses me off.

"Steady on Peter. Its only a serializer implementation difference".

Yeah, yeah I know.

The thing that gets me worked up is that in ROC its perfectly reasonable to expect that within a stable horizontal well defined resource model (like XML) a resource request today resolved and evaluated to an endpoint in a library, will produce the same resource representation next time, next year, next decade...

This should be the case even if the underlying platform (NK1,2,3,4 or JVM 1.3, 1.4, 5, 6, 7) changes. This is your right and expectation as an ROC architect. Implementations change, but a given identifier when requested ought to give you the same representation.

Someone somewhere inside Sun didn't respect this contract. Probably because when you have an API mind-set you abrogate responsibility for the first order information and instead leave it to end-user developer to sort it out.

Now, I know that long-term stability is a Platonic ideal. We can only dream of it.

But we, as ROC service creators can bear this lesson in mind. Why shouldn't the user of your service expect resource stability over the long term?

After all in ROC we're independent of type constraint thanks to transreption. Even if I choose to use a new, faster, super-duper internal object model in my implementation, I can always provide a new-to-old legacy transreptor for any resource representation I create.

Do I have an anal retentive regression fear? I don't think so. I really do envisage that with ROC we can demand and expect that information architectures posses natural long-range stability.

I sure as hell know from experience that our internal systems carry on running year after year even though we augment and evolve them.

Isn't this a cornerstone we should expect in information engineering? I do.

Have a great weekend.