NetKernel News Volume 2 Issue 27

May 6th 2011

What's new this week?

• Repository Updates
• Tolerance Enhancements
• Lang Python
• IBM J9 JVM
• Encrypted Modules
• Unattended Install
• Gravity Works as Expected

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repositories...

• lang-beanshell 1.3.1
  • Fixed the beanshell transreptor to use recent layer0 metadata features - thanks to Arun Batchu of Bestbuy.com for reporting this.
• lang-python 1.4.1
  • Update to use latest Jython 2.5.2 release.
• lang-python-lib 2.6.1
  • Update to provide the latest Jython 2.5.2 class libraries.
• standard-module 1.41.1
  • Update to mapper and overlay base classes to add extra fault tolerance.
• nkse-dev-tools 1.31.1
  • Update to new module wizard to provide correct unit tests to a generated nCoDE module (thanks to Carl Conradie of Momentum Insurance for reporting).
• system-core 0.23.1
  • Update to add tolerance for "bad-metadata".

The following update is available in the NKEE repository...

• nkee-dev-tools 0.19.1
  • Added "Update" capability to the RDBMS management tools.

Tolerance Enhancements

One of the enjoyable parts of the conference was the preliminary bootcamp - which was very well attended and in which we gave a very rapid immersion in NK. I tried my best to condense down something that should take 3-5 days into just one intense day.

One of the great things about doing live training is that you get to learn from the students. They invariably do stuff that you don't anticipate. But also, there's an air of high-wire act about doing live examples in front of an audience.

It was during one-such live example during a part of the course introducing the mapper, that I managed to completely screw my module up. I was winging it (as usual) and for some reason trying to explain something without paying attention to the validity of the mapper endpoint definition - I got my module screwed up.

The nature of ROC is that it is an incredibly malleable, completely decoupled model. The job of a developer is to introduce constraints in order to form the information process out of the resource landscape. In my mind, for a "classically" trained developer learning NK is something like being a steel worker used to ship building and transitioning to titanium, carbon-composites and aero-engineering. In NK/ROC software is very light and very strong and information engineering is about composition.

Anyway getting back to the main theme, during the bootcamp I completely buggered up my module and there were "red warning lights" going off all over the place. Fortunately, being a seasoned professional, I noticed it before the audience and managed to revert out my mess. The audience never knowing how perilously close to missing the trapeze I'd got. (Well all apart from Tony - who was sitting at the back - he knew and later he gave me some justifiable stick!).

It was good to have Tony see this though - since having your pants fall down in public is a good incentive to make sure it can't happen again. And it very obviously revealed that the mapper and, as it turns out, the overlay base class both needed to have more tolerance for misconfiguration of parameters and to still be a good ROC citizen even when completely misconfigured (ie to be Pete-proofed).

Hence the updates to standard-module.

In addition, I have for a few months been meaning to sort out another training related conundrum. This one came up with Frédéric Berger at Steria in Belgium. For the life of me I couldn't work out (in the real time environment of training) how Frédéric had managed to create a "completely broken" module. Fortunately I grabbed a snapshot of his system - and its been "one of those jobs" to work out what it was about Frédéric's apparently correct solution that was causing the problem.

Well we finally got the time to investigate (Frédéric, as promised, I didn't forget)...

It turned out that during the class, in our exercise to create a Java Standard Accessor, rather than overriding onSource(), Frédéric had (perfectly reasonably) implemented onRequest(). But the problem with onRequest() is that it's close to the metal and gets called for every request - in the base class it is actually responsible for delegating to onSource() if the request has a SOURCE verb.

It follows, that it also gets invoked for metadata resource requests with the META verb...

As you know, we've recently been adding many more metadata features into the base classes and also many more of the higher level tools use the discovered ROC metadata. However, it turns out that the metadata aggregation was a little too optimistic - Frédéric's onRequest() didn't implement any metadata handling (why would it? he didn't need to know) but nor did it delegate to the usual default handling of the super parent class.

Meanwhile the ext-system space metadata aggregation was too optimistic and wasn't expecting an endpoint to throw this curve ball. At which point all hell broke loose.

It was a relatively simple thing to add some tolerance and robustness to the metadata aggregation to ensure that it was tolerant of either missing or malformed metadata. In addition, since metadata is an ROC resource too, we added some transient caching hysteresis to give the system additional tolerance during phases of transient reconfiguration - you probably won't notice any difference but this is NK using the ROC abstraction to minimise the computational cost of computing dynamic metadata state.

Hence the update to the ext-system module.

Those of you who were at my keynote at the conference will appreciate that with the NK system we are constantly pushing at the limits of Turing/Godel completeness. So we don't (can't) promise to have made the system completely "Pete-proof". But you should now find its more tolerant in the face of misconfiguration.

Lang Python

It turns out that the Jython team recently released Jython 2.5.2 - which they claim is 3x faster on one benchmark and 20% on another. It also seems like the Jython coverage of the CPython class library is now pretty complete. But, possibly more significant, is that the change history appears to have fixed a number of memory leaks in the python class libraries.

The integration with the active:python endpoint was pretty seamless - just requiring a minor tweak to the resource loader (which allows Python to load classes from the ROC address space - yes, python classes are resources, everything's a resource).

I asked Tom Geudens to try out a 2.5.2 based preview build since he's a hard-core Pythonista and has much more knowledge and coverage of the Python class libraries. He reports that it is plug-replaceable with the existing library, I quote...

Functionality all works. Furthermore I added one more test to the bench :

from time import time
t = time()

s = 'lksdajflakjdsflku09uweoir'
for x in range(30):
    s = "".join((s, s[len(s)/2:]))

resp=context.createResponseFrom('duration: ' + str(time()-t))
resp.setMimeType("text/html")

Tom says he didn't see any immediately obvious difference on the performance - but a more solid class library is still reason enough to update.

Incidentally - one thing that might be of interest. As you know we provide many flavours of dynamic language all integrated using the ROC language runtime pattern. Of all the "classical" dynamic languages - Jython compiles to significantly the tightest bytecode and consequently is the "fastest choice" for dynamic coding. It's been a while since we compared languages but last time we looked Jython ran fractionally slower than native Java. Whereas some languages, like Groovy, in contrast can be 30x slower.

But before we start getting dragged into a "my-language is better than yours" debate. Remember that the nature of ROC's composite approach is that most of your code (80%) is there to do "composition" (ie sequencing requests, and maybe a little business logic). This means that for a given solution, the real time spent "inside a language" is invariably very small (<10%) compared with a thread's extrinsic time actually doing real stuff in other requested endpoints. (Switch on the visualizer and take a look - it captures local and extrinsic time for all endpoint executions)

So the moral is, it doesn't matter which language you use - its probably fast enough. Also NetKernel is doing its level best to ensure that we aim never to run your code anyway! (Tom Hicks are you out there? "Its all in the thermodynamics man".)

IBM J9 JVM

NetKernel is designed to be scalable - our belief is that NetKernel should run on the smallest embedded JVM right through to big iron mainframes. And, as we demonstrated at the conference, it runs really pretty well on a tiny embedded Guru Plug computer with a 48MB heap on an embedded JVM for the Arm 5 core.

We also take great care to ensure that NetKernel doesn't have any exotic requirements for the Java variant it runs on. It is written to use only vanilla Java SE APIs and is fully self-supporting beyond this. We develop on the Sun Java 5 SDK, but also regularly test and deploy on OpenJDK and many developers have Macs and use NK on the Apple JVM (which used to be) shipped with OSX. We also have, in the past, validated that NK boots and runs on the (former)BEA JRockit VM.

However, these being "interesting times" for Java, with the relationship to Oracle still in flux. Some of you may be interested to hold a reserve card in your pocket.

So we recently installed NK on the IBM J9 JVM. Given our experience on other Java platforms we were quite surprised to see this go by in the boot logs...

Hmmmm. That's odd?

A quick search dug out the following thread, with some other user reporting the same InvalidKeyException...

http://stackoverflow.com/questions/4869904/whats-wrong-with-ibms-jce-provider

It turns out that IBM has implemented their JCE security provider to be very fussy about the interface of the PKI key used to do encryption (and more importantly given NKs support for encrypted modules, decryption). It seems to think it knows best and that you must only encrypt with a public key and decrypt with a private key (which to be fair is the more common usage pattern).

Unfortunately, as the guy in the thread also needs, and some bits of NKEE relies upon, there is another pattern in which we require the ability to encrypt modules with a private key and, in a crypto-module factory to decrypt using a broadly distributed public key (actually its a lot more involved than this but that's the broad gist of it).

Unlike all the other JVMs we've experienced the IBM security provider doesn't permit the "private-crypt public-decrypt" pattern (is this a bug or a feature?).

JCE Workaround

Fortunately, this situation is easy to workaround.

First get hold of the BouncyCastle JCE security provider here...

http://www.bouncycastle.org/latest_releases.html

Then follow these instruction to make it the JVM's priority security provider...

http://www.bouncycastle.org/specifications.html#install

It basically comes down to editing $JAVA_HOME/jre/lib/security/java.security and adding a line:

security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider

You'll now find that you can boot NetKernel and all is well.

Encrypted Modules

Everything inside a secure module is encrypted using RSA public-key cryptography - including all file resources and including compiled bytecode *and* any java jars shipped within the NK module.

We regularly work with ISVs who use NK as their OEM platform. One of the nifty features (to the eyes of an ISV with a product to sell) is that NK's encrypted module capability is extensible and supports multiple tiers of keys. Its therefore straightforward to be able to build your own encrypted modules and deploy them to a regular NKEE platform with your own authorised activation license.

There are other (less blatantly commercial) potential uses for this capability. For example you could readily create and deploy an encrypted "trust-module" containing your critical JDBC connection configurations (see this article where I discuss the patterns for JDBC connection resources) - so you would have cryptographically encrypted connection settings to your DB's that no one but the creator of the module ever knows.

Or if you were so inclined, you could encrypt your entire set of application modules.

For obvious reasons we don't publicise the full details of how the encrypted module capability is implemented. But if this is something that is of interest, please get in touch and we can explain how you can take advantage of it.

Unattended Install

Earlier in the week I got a gentle reminder from Jeff Hulten that I'd promised to add the ability to do unattended installs to the NKEE bootable jar distribution. And so I did.

If you now download a copy of the NKEE distro you can initiate automatic install as follows...

java -Dunattended.install.directory=/home/pjr/NKEE-4.1.1 -jar 1060-NetKernel-EE-4.1.1.jar

Which will fire up NK from the jarred distro, automatically self-install to the target directory and then shutdown.

Unattended install is triggered by the presence of the "unattended.install.directory" system property. There are also a set of optional properties to set up proxy settings and to expand jarred modules...

• unattended.install.directory - name of directory in which to install the NK distro (mandatory)
• unattended.install.proxyHost - proxy server hostname (optional)
• unattended.install.proxyPort - proxy server port (optional)
• unattended.install.proxyUsername - proxy server username (optional)
• unattended.install.proxyPassword - proxy server password (optional)
• unattended.install.proxyNTDomain - NTLM proxy server domain name (optional)
• unattended.install.proxyNTWorkstation - NTLM workstation name (optional)
• unattended.install.expand - expand jarred modules (optional "true" - defaults to "false", not expanded)

If, for any reason there is a failure during the install, a message will be written to stderr and the java instance will exit with a negative exit code.

Performance

Jeff is a system configuration expert at BestBuy.com and needed this capability in order to set up on-demand NK instances using chef.

On my laptop a complete unattended install (boot, install to disk, jvm shutdown) takes 4 seconds - which is about as near realtime-on-demand as you can get.

Chris Cormack, did a quick test of the new capability and observed another interesting benefit - you can use this feature to install a new distro onto a server which is already running an existing NK without having to shut it down.

Tom's Book Status Report

Tom Geuden's sent this note...

Progress this week has been (too) slow and does not warrant a new iteration to the book. However, the daily build : - Is updated to reflect the demise of an infamous terrorist. - Does contain the beginnings of Appendix E. Not a lot on Version Control in the module itself yet, but you can already learn how to add stuff in the Backend HTTPFulcrum Control Panel. And admit it, you all want to be able to do that !

Here's the link to the daily build : http://www.netkernelbook.org/serving/pdf/hello_netkernel_nk4.pdf

As always, your feedback and input is highly appreciated. You can send it to tom(dot)geudens(at)hush(dot)ai"

Cheers Tom!

Gravity Works as Expected

The results of the astonishingly beautiful Gravity Probe B experiment are in. Local gravity, with small mass, is consistent with relativity theory...

http://www.bbc.co.uk/news/science-environment-13286241

As they say in d' Physics 'hood:

"Big up, Albert E - you is d' man".

Have a great weekend,