NetKernel News Volume 2 Issue 31

June 3rd 2011

What's new this week?

• Repository Updates
• Groovy 1.8.0
• Overlapping Processes Tip
• Dynamic Throttle Adjustment
• On Resource Mutability
• #inthefuture
• Virtual Parlour Game

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repository...

• email-core 1.3.1
  • Modified encoding so that text body parts are now sent with default UTF-8 encoding to be consistent with NK's default.
• http-client 2.6.1
  • Enhanced Oauth implementation to support non-OOB callbacks
• http-server 2.5.1
  • Fixed http so that 302 redirects do not occur if response has exception.
• json-core 1.3.1
  • added JSON<-->HDS transreptor pair
• lang-dpml 1.14.1
  • Fix to DPML so that response headers are not lost
• lang-groovy 1.8.1
  • Updated to use Groovy 1.8.0 library (see below)
• lang-ncode 1.3.1
  • built-in sequence accessor - fix to ignore scheme/active-type arguments
  • built-in transrept - fix argument naming bug
  • added getHeader/setHeader built-in accessors to get and set response headers
• layer0 1.61.1
  • transrept methods on NKF context now allow INKFRequestReadOnly
• layer1 1.28.1
  • Update to Throttle to support dynamic config (see below)
  • added from FromNodeList transreptor to make output from HDSXPath more flexible
• module-standard 1.44.1
  • Made inconsistent expiry messages more verbose

The following update is available in the NKEE repository

• nkee-dev-tools 0.20.1
  • stop mutable resource warning by adding constructor declare to inhibit

Groovy 1.8.0

We've shipped an update to lang-groovy to use the latest 1.8.0 release of Groovy. All our unit tests are good, we did extensive QA stress/leak testing and the groovy project's issue tracker has no serious open issues; so we feel confident this is a solid production ready update.

The release notes for 1.8.0 are here...

http://docs.codehaus.org/display/GROOVY/Groovy+1.8+release+notes

For our use of groovy as a dynamic language runtime the most notable enhancements are:

• Performance improvements
• Native support for JSON

I've not noticed any significant performance change - but then I wasn't really looking. However, I can attest that the JSON support is useful and may mean you no longer need to import/use our json-core library's Java-JSON API.

The Groovy-ista seem to be quite excited by the new support for closure memoization but, when you're used to ROC's extrinsic systemic memoization, its intrinsic nature and necessity for end-user management of referential integrity is a bit underwhelming. But then, as you know, I have an unusual perspective on languages and management of computational state ;-)

Our expectation and experience is that, from a language perspective, this release is backwards compatible with your existing groovy 1.7.x code - but please let us know if you discover anything unexpected.

FYI our production applications are running on this update with no problems

Overlapping Processes Tip

One small job I did this week was to contribute the WiNK wiki's fulltext index and search code to Chris' nk4um project (prompted by Jeff Rogers' enquiry). The basic design is to create an external search index sitting outside of the DB-layer using the tools in our Lucene-based text-search library. Every new post is independently indexed and search results are able to correlate the post, the topic and the author to provide a "linked-data" resource back to the forum topic resources.

One of the requirements with any extrinsic search index is that you have to seed it by indexing existing posts (you may also, for example in a disaster recovery scenario, want to do a full reindex too). I added a full reindex tool to the backend admin services and it was while testing this that I inadvertantly discovered a nice optimisation tip.

The full reindex does the following.

1. Sources the set of all post identities. (Every post has a unique primary key id)
2. For each id, dispatch a request to the individual post indexer

The individual indexer, sources the post resource (content, title, author etc), generates a declarative index entry and adds it (with analysis/extraction) to the index using the active:textIndex service. (You can examine the source here)

Our staging RDBMS server, with a development copy of our nk4um DB, is in the office. I was at home with a VPN connection. Which meant that my DB access is not entirely snappy. I started a full reindex and was getting about 3 post indexes per second (not so bad on the damp-piece of string connection my ISP has the balls to call "broadband"). But our test db has 3000 posts in it - so it was going to be a long night...

So while it was doing its thing I started reviewing my index implementation. I realised that SOURCing the post entry was high-latency and dictated by the network/db performance, but in addition there was a finite index/analysis cost on the active:textIndex call. So I thought hmmm, why don't we overlap the two? So I edited the code and changed the active:textIndex request to a fire and forget asynchronous sub-request by changing

context.issueRequest(req)

to

context.issueRequestAsync(req)

So now the main thread would go back to the reindex dispatcher, and intitiate another index call for another post (with the DB SOURCE starting again) while my async request was off indexing the previous one. I'd overlapped the processes to eliminate as much latency as I could.

My implementation uses the groovy runtime (I had to ask permission from Chris to keep my contribution in this original language instead of converting it to Java accessors as he's used throughout the rest of the app!). So I just saved the edited source, I didn't even consider that it was still chugging away on the reindex. Of course the new source expired the previous compiled code, it was instantly compiled and hot-replaced the old version. Without missing a beat the ongoing reindex process kicked up a gear and started doing 9 requests per second. 3x performance gain, hot swapped with less than one line change. ROCit Ship!

Incidentally, the full text search will appear in our live production nk4um real soon now. Chris needs to pull my contribution into the main development branch and just add the styling for the search presentation view.

Dynamic Throttle Adjustment

I spend too much time in a twilight zone between reality and conceptual principles. As such, sometimes I trip and say that something that is possible in principal is actually implemented in reality. One such example came up this week. I was discussing with Brian Sletten a demo he is preparing for Semtech 2011 conference next week. One requirement is to have dynamically aggregated metadata drive "non-functional" architectural structure. In particular he had an idea to dynamically control a throttle overlay wrapping a set of resources.

The throttle is conceptually very simple it is essentially a buffered asynchronous managed queue whereby a maximum number of concurrent requests are admitted simultaneously through to a wrapped space. If the concurrency is exceeded the requests are queued and when at maximum capacity, like a turnstyle, only when an inner request returns is a new request admitted.

The throttle is configured with a simple configuration resource like this...

My mistake of theoretical complacency was to tell Brian that since this is a resource (everything's a resource!) then the throttle could be dynamically reconfigured simply by mapping the identifier of the config resource to a dynamic configuration control endpoint.

When I actually went and looked at the implementation I was therefore disabused when I discovered that it was actually only SOURCEing its configuration parameter during the postCommission() phase and therefore would only reconfigure when the implementation space was recommissioned. Dynamic yes, but not "realtime dynamic" as I'd asserted to Brian!

Ah, mmm, yes. It was implemented quite a while ago and has covered all our steady state production needs very well. But then you think, hmmm actually a true dynamically reconfigurable throttle would be cool. So half-an-hour was all it took to refactor the configuration and move it from the postCommission() to an independent doConfigure() method that gets called at postCommission() (so its 100% backwards compatible for existing systems).

But it now uses the pattern of holding on to the INKFResponseReadOnly which is received when the configuration is SOURCEd. Now every time a request is received by the throttle, once the request has been dispatched or queued, we quickly sneek a peek at the expiration of the configuration resource. If it has expired we call doConfigure() again to use the state of the new config resource.

So now the throttle really does dynamically adjust based upon the state of its configuration resource.

Here's an example use...

Where, for example, active:dynamicThrottleConfig (my invented name - call it what you like) is mapped to an endpoint something like this...

import org.netkernel.layer0.representation.*
import org.netkernel.layer0.representation.impl.*

b=new HDSBuilder()
b.pushNode("config")
b.addNode("concurrency", "4")
b.addNode("queue", "250")

resp=context.createResponseFrom(b.getRoot())
resp.setExpiry(...TBD...)

So now the throttle will be atomically reconfigured whenever the response of the dynamic throttle controller expires. In practice, you could use dependency expiry on a configuration specified using a slider in a control panel tool. You could have the throttle be driven off external realtime network state from your switches or routers - in which case you'd probably use timed expiry to introduce some transient Nyquist hysteresis. You can also very naturally use the Golden Thread pattern to have the configuration be served but invalidated by an entirely independent system state change or even an external cluster management system.

On Resource Mutability

Following last week's discussion on consistency and the new feature that the fileset may be declared as mutable I received a follow-up email from Tom Geudens...

Which shows that I need to do a better job and explain this more clearly.

Firstly, some clarification. A central tenet of ROC is that representations are immutable. A representation is the snapshot of the state of a resource - it could (most likely will) be referenced by many other threads concurrently and so you should not modify a representation's state. Ideally representations are implemented with immutable interfaces, but often with POJO's that's not always possible, so its your duty not to mutate a representation.

However, it is not a tenet of ROC that resources are immutable. This just reflects a central tenet of reality that change is the only constant! You may, by all means, change the state of resources, and as we saw last week, it is the responsibility of the implementor of a resource "accessor" to decide how you wish to maintain consistency between any existing representation state and the new state of the resource.

But probably the thing I need to explain is why did we make the fileset mutable? Firstly it was not so that the file system would become the standard location to persist transient configuration state and so be a replacement for pds: or fpds:

The only reason we made the fileset mutable was so that the nCoDE tool could be easily configured in a development space and so write its persistable state to the same module in which its instantiated (and so be source control manageable too). Imagine how horrendous it would be if the nCoDE source code got persisted to a different medium than the rest of the developer's module code?

To see how we're doing this use the new module wizard to create a new nCoDE module and look at the fileset. Its mutable just to make life easy during development.

It certainly is not our intention that because fileset can be used mutably, that this is recommended for application state persistence.

Using the filesystem in this way is brittle and liable to production-time failure and deployment inconsistency. Imagine you deploy a new version of an application - if you store your configuration state in the module filesystem you have to reoconfigure every time you deploy a new version. pds: is provided to allow you to decouple the mutable state of your application from the immutable production state of a deployed module.^*

pds: has other benefits too. It is pluggable with its implementation - it is therefore very simple to provide an implementation that is not even tied to the current host platform. You can very easily have a fail-safe, cluster/cloud-safe location for your application configuration state that would allow you to instantly set up NK systems without the need to do any post-deployment configuration. In fact the whole no-SQL world of "bit-bucket" stores is ideal for backing a clustered pds: address space.

Hope this clarifies this for you Tom! Thanks for poking me!

Incidentally, another useful feedback thread was started by Jeff Rogers to discuss the implications of "delegated responsbility for consistency" in the forum here. This conversation is ongoing - please join in and share your thoughts.

* If you want a very nice example of how an application can use pds: to persist its configuration state, going from install, to production use, to progressive migration when new versions are deployed. Take a look at the implementation of the admin tools in Chris Cormack's nk4um application.

#inthefuture

What are we to do? Are we in danger of becoming the "generation that turned its back on the future"? We've already witnessed the last flight of Concorde and, just this week, the penultimate flight of the space shuttle.

I've never seen a space launch (the closest I've been is to be at Cape Kennedy at 7am before the tourists arrived and to have the unalloyed thrill of being the only person in the entire Saturn-V hall). But I did bear witness to the last flight of Concorde - our office is, relatively speaking, a stone's throw from the Filton airfield where Concorde was built and the last Concorde touched down - and so I witnessed the very last moment of the supersonic commercial jet-age.

So is that it? Are these the symptoms of the "the end of technological history"?

It's easy to be pessimistic. When the biggest technology of today exciting people (and markets) is the "breakthrough" of 'simultaneous access to shared state', in the form of twitter and facebook, its easy to be grumpy. Bloody hell, compared to building the shuttle this is utterly trivial.

I'm not jaundiced. In fact I get the piss taken out of me all too often when I semi-ironically say "Do you know we are living in the future"...

Not so long ago, it was considered an impossible dream that doors would slide-open and closed automatically - did you not watch Star Trek? Now you just go to the lobby of your office, the supermarket or corner shop and you're slapped by the future. (You maybe didn't notice cos the doors don't make any noise - why can't they do the swooshy/slidy noise like on the bridge of the Enterprise?).

We all carry around personal communicators that work, pretty much, anywhere on the planet (just like in Star Trek).

At any given moment, you are less than 1 minute away from a fair representative sample of all the accumulated knowledge of mankind (though some of it is behind paywalls - so 1 minute and a few dollars away!).

And we stand on the brink of a "new energy age". I don't care if you agree with the evidence for global warming or not - it is actually irrelevant. There is one incontrovertible truth: oil is a finite resource. It will not sustain our energy needs for another 100 years (which is no time at all). If our kids are to have the same standard of living as we've enjoyed, we have no choice but to find alternatives. Don't knock any stimulus that catalyses the transition. But any change yields unforeseen dividends, and so, as with the transition to steam, and then to oil, what comes next will definitely have indirect benefits. Imagine the transformative effect on the third world economies by cheap, effective micro-generation?

Looking at point technologies, we won't know for a while, but there's a reasonable chance that the discovery of Graphene will lead to an incredible blossoming of technologies.

We also have the transition to optical computing and holographic storage - Moore's law will dry up in electronics - but it'll keep on chugging in spin-tronics and photonics and quantumics.

So what about us? What about software? What about information engineering?

Oh oh. On the daily evidence I hear as anecdotes from the field - it would be easy to turn pessimistic again. I hear the worst horror stories (all strictly confidential you understand) of projects that fail before they start, fail because the solution doesn't match the reality of the realworld deployment. Or they do work but they can't be changed. Or they can be changed but the change didn't match the expectations and reality bites again. Over and over and over and over.

Well Brian Sletten pointed me to this article...

http://www.azarask.in/blog/post/the-wrong-problem/

Its a good read - by analogy with human-powered-flight it asks the question "Are we solving the wrong problem". The echo's I get from the real world almost certainly are in the affirmative.

But you know the worst part? Almost always, the types of information systems I see are just not very ambitious or interesting or really ground breaking. In short - we're really not being very ambitious - we're not even trying to tackle our own "human-powered-flight" challenges.

OK so Brian wouldn't have pointed out the article, and I wouldn't have referred to it and contrived this discussion if I didn't have a point to make about ROC right? So here it is.

When I first seriously looked at software, it made no sense. It felt just exactly like the monstrous monolithic engineering adopted in the conventional approaches to trying to solve human-powered aircraft.

For example, I once knew a very very smart software engineer. He would design the most amazingly detailed intricate solutions. Would stub-out every interface with complete code-coverage. The problem was, he'd do this all in perfect isolation from the real world. As soon as this perfect crystalline lattice of software was connected to the real world - there'd be a bit that didn't fit right. And then another bit. And then refactoring to fit the new reality meant something else didn't fit until what looked beautiful and pure in the beginning ended up looking like the hair-ball from hell.

Lest we forget, the road to hell is paved with perfect architectures.

The problem with classical software, is not that we can't solve problems that way; a Turing machine is a Turing machine, no matter what the language. It's that we, as engineers, just can't anticipate reality.

To solve really interesting, really new, really challenging problems, we have to allow for trial and error - we have to be able to experiment.

Just as with the ultimate winner of the human-flight challenge, we iteratively find good solutions when we are able to compose and recompose and adapt and modify.

This is the essence of ROC.

ROC is the playground of reconfigurable information engineering. It enables us to explore and create systems orders of magnitude more sophisticated and powerful and with excellent fidelity and adaptation to the realworld to which they are connected.

You don't believe me? Look at the World Wide Web. It is the most powerful and flexible information engineering solution ever created. It is a rudimentary and simple ROC system. It is reconfigurable. And, you know what? It never had a specification - we arrived here by experimentation.

ROC works. We can use it to turn back the tide of history - the future will be better than past.

Virtual Parlour Game

I started a running joke with my kids this week. We adopt a robotic voice and then say "In the future, x y z". Where x,y,z are absurd statements related to something we're doing or talking about. For example...

"In the future, Pete's newsletters will not end with a rant and a plug for ROC".

It occurred to me this morning that this would make an excellent virtual game (and I'm also curious to see how meme's propagate). So here's the rule.

1. Tweet using the hashtag: #itfgame (funny is good, funny and inspirational is best!)
2. Read with an imaginary robotic voice.

(Perhaps this game has already been invented? If so, you must remember I don't get out much. There seems to be a #inthefuture tag going about but the tweets I read were cringe-makingly "earnest" and not at all ironic!)

See you in the simultaneously accessed shared state space

#itfgame people will be unimpressed by shared databases