NetKernel News Volume 1 Issue 47

September 24th 2010

What's new this week?

• Repository Updates
• ROC Architecture Explorer - Preview
• Video of "Resource Oriented Cloud Architectures" talk
• Tom Guedens' NetKernel Book v0.2
• Modular Source Control

Catch up on last week's news here

Repository Updates

The following updates are available in both the NKEE and NKSE repositories...

• module-standard 1.28.1
  • added metadata to expose branch-merge configuration.
• system-core 0.17.1
  • various updates to improve structuring of meta-data to aid development of graphical explorer.
• wink-slinki 1.8.1
  • Improved method of implementing shrink-wrapped div for {centerh} no longer needs the +width@ active argument.

NKEE Apposite Client

The following update is available in the NKEE repository...

• nkee-apposite 1.19.1

After updating the HTTP client a couple of weeks ago, we introduced a new connection manager in the NKEE apposite client. It turns out that the default settings for the library used a read-socket-timeout of ten seconds. If you had a slow connection, or experienced any latency you therefore might have experienced timeouts giving poor results for sync and downloading from the repositories.

We've posted an updated client with a more sensible setting. If you're unlucky enough to be experiencing timeouts and therefore can't get to the update, then either rollback your system to before you took the http client updates, or make a clean install of NKEE (which still has the old technology) and then sync and re-update to get to the current state of the art.

ROC Architecture Explorer - Preview

After last week's sneak peak of the ROC architecture explorer we were swamped with "When can I get it?" questions.

This generated some internal discussion about the best way to proceed with the ongoing development path whilst still allowing the value of the initial work to be exploited.

We've taken a working snapshot of the prototype layout engine and wrapped it with a simple AJAX GUI to create an early-access preview tool.

Its simple and represents only a very early stage preview of where it is heading. But it works, and I've been using it to illustrate spacial architectural structure (see the video link below).

Due to its relative immaturity and limited cross-browser testing we've not made it a general apposite repository release. However, if you want to try it out, send me a note and we'll provide details of how you can get hold of a copy.

I've not had chance this week to deliver my promise of using it to explore ROC patterns but its not forgotten.

Video: Resource Oriented Cloud Architectures

I gave my talk on Resource Oriented Cloud Architectures at SkillsMatter in London on Tuesday. I used the architecture explorer to illustrate a number of the architectures I discussed. The SliNKi slides are online together with a video (sound quality is not perfect but if you turn it up its ok but you should definitely have the slides up at the same time). If you want to try the demos out for yourself they're installable from Apposite as demo1 and demo-nkp-mandelbrot.

• Presentation Slides
• Video

Feedback from those that came along was very positive and we're planning another SkillsMatter event later in the year. I'm thinking perhaps "ROC from First Principles". To cover the process of iterating between modelling, architecting and implementing endpoints to create an ROC solution.

Tom Guedens' NetKernel Book v0.2

Tom has sent the latest draft of his NK book. It now includes comprehensive coverage of how to setup and manage an NK installation as a daemon or windows service.

hello_netkernel_nk4_0.2.pdf

Please let Tom have any feedback on progress so far, and also ideas for content or even contributions of sections. He can be contacted at tom (dot) geudens "at" hush {dot} ai

Modular Source Control

I spent a little time exploring whether git offered anything we're not already getting from subversion.

Git's fully distributed, fully replicated revision history is very nice and gives good disaster resilience. Since it is localized by default, it is fast (but speed isn't a big concern as we've never had any problems with svn). It seems to be drop-dead simple to begin a local repository, while managing and administrating a public server looks straight forward too.

So is git the way forward? Well, apart from the learning curve and new processes it requires, there seems to be a particular fly in the ointment that doesn't sit well with the way we work. It seems like managing highly modular projects gets quite complex and requires quite disciplined and rigorous observation of external process and good practice.

My reading of matters is that it seems like git didn't start with modular development as part of its abstraction. In what way? Well, critically, branches and tags apply globally to the complete repository. Submodules were retrofitted in a later generation of the tool, but this essentially requires you to manage a tree of repositories within repositories and, has quite strict consistency rules.

I'm quite happy to be put right if I've not understood git sufficiently. But, let me explain how we structure our SVN source control and the modular use case we require...

Modules Everywhere

If you've looked at etc/modules.xml, read my discussion of the NK boot process, or glanced over the spacial architecture diagram I posted last week, you'll know that NK is a modular system. All told we have more than a thousand production modules ranging from the core distributions of NK, the extended library modules available in apposite, demo and unit test modules and our own production applications and services.

Modularity has always been one of the key design criteria for NetKernel. In the earliest days (using CVS about ten years ago) we made the mistake of having a repository structure in which modules were located in categorized directories. In those days NK had no online package download system, so the entire distribution shipped with all modules, and we would tag the entire repository whenever we created a release. Individual modules didn't really have a separate lifecycle. (Categorizational structure was a mistake because you always find something that defies classification and things get messy or lost).

Our build process back then was a self generating ant process - where each module had its own ant script. It was all relatively linked with the repository structure. This approach works very well for application scale projects (probably the things you work on), but it started to get out of hand as our number of modules grew and NetKernel evolved to become a full-scale operating platform.

SVN Design

We moved to svn about five years ago. We have four primary repositories; for the core NK kernel and system, library modules, production applications and demos, 3rd party projects etc. Each repository has a distinct independent instance and is mirrored for failover.

Within any given repository we use a common organisational structure. We completely dropped module classification in the structure, and have a top-level structure like this...

./module/
./scratch/
./test/

A first-level directory contains a flat, urn-style, directory for each module. For example, groovy lives in...

./module/urn.org.netkernel.lang.groovy/

(we use "." dots because windows can't cope with ":" colons in directory names).

Our general workflow is for development/experimental modules to live in scratch/ until they are production ready, at which point they are migrated to module/. The test/ directory contains unit test modules (typically one for each module).

Within any given module (here we'll use groovy again) we have the following branch structure...

./module/urn.org.netkernel.lang.groovy/trunk/src/
./module/urn.org.netkernel.lang.groovy/release/x.y.z/
./module/urn.org.netkernel.lang.groovy/branch/name-name-name/

Where a release is always built from trunk and development branches can go on in a named branch before being merged into trunk if/when it becomes releasable. (Branch naming choice is developer specific, we have no hard and fast rules for this)

Long ago, with the scale of system NK had become, we realized that for our needs, using the filesystem alone was not adequate to track the relationships between modules. (Although for application level projects it can be fine.)

For example, lang-groovy is built against the following modules (most of these are core system libraries but in our process we can still treat these as "logical modules")...

urn:com:ten60:core:layer0
urn:com:ten60:core:module:standard
urn:com:ten60:core:netkernel:api
urn:com:ten60:core:netkernel:impl
urn:org:netkernel:ext:layer1

Therefore we did the right thing and created a proper RDBMS management system to track dependencies, maintain versioning state and, critically, the module dependency versioning.

Our build application (called "Apposite Master"), can checkout and build a single module against its svn trunk and the trunk of all dependency modules, or automatically build a release version, or checkout and build any historical release version against the exact historical dependency module versions.

In each case the RDBMS state is used to dynamically generate a temporary ant build script for that part of the build-process. (We minimize build costs by maintaining verifiable build caches too, the build system checks the cache and svn, and then creates an ant script that only builds the unbuilt subset - its very ROC-like)

Just to complete the story, with NK's apposite repository architecture, we also have to manage packages of modules. Again with versioning and, in addition, with a logical package dependency model.

The package dependency model is logical in that it has nothing to do with the build process. Like the Debian apt model, it allows a package to state logical dependency relationships to other packages (even if they're not from the same repository), to describe mutual exclusions, to describe suggested optional packages (like dynamic import library add-ons) etc etc. Since it's logical and separate from the physical module repository, the package dependency state needs the Apposite Master RDBMS to manage the state and could not be modelled by branching at the physical repository level.

Finally we have distributions, which are bootable system builds; basically a functional subset of available repository packages. Each distribution has a corresponding online repository which offers additional installable packages for that distribution. The distribution repository also has to provide updates and security releases, all of which are tracked and managed by Apposite Master.

The Apposite Master system was actually the first in-house production application we developed as we were developing NK4. Somewhere along the way Apposite Master, running on a prototype NK4, took over completely and built each subsequent development release. So today, NetKernel builds NetKernel.

Module Branching Requirements

So you can see the situation, each release build of a module is automated, each built module's version number is reconciled from its module.xml version number, or if its not been updated by the developer, is auto incremented by the build tool and stamped into the module.xml and committed to trunk.

Critically for the source control discussion, each module release is then branched with an svn copy from trunk/src/ to release/x.x.x/ . Furthermore, since each module depends on other modules, the build process also ensures that if a dependency module is not in a stable branch, it too is auto released and a branch created for it in its own module/y.y.y.y/release/z.z.z./ - this ensures that if/when we want to rebuild the x.x.x module we have a consistent and reproducible state - which is essential if any retrospective bugfixes are needed on that production release.

Source control seems to be a particularly religious subject. Our multiple subversion repository model is an evolved design that suits our processes and has scaled very well. We have a critical requirement that a given module build is branched and that it also builds and branches its dependency modules if their svn trunks have unreleased changes.

I can see that git would require us to have a git repository for each module, then meta repositories for "super projects" (git's term) and any one repository would need to be branched separately and respect the rules of managing git "submodules". It all starts to feel pretty complex.

Finally we do a lot of work on new stuff that's not ready for prime-time and which we keep in scratch/ (For example that's currently where the new architecture explorer tool is being developed). I can see that git super projects could allow for this, but again it feels like the complexity could easily get out of hand.

As I said, I'm quite prepared to be enlightened. Just I've not seen anyone using git to manage something like our current fully modular structure. At the very least, maybe hearing how we do stuff might give you some ideas for your own approach.

Have a great weekend.