NetKernel News Volume 3 Issue 46

November 16th 2012

What's new this week?

• Repository Updates
• On Modularity
• Factors Affecting Partitioning
• Real World Example: NetKernel Portal Modularity Analysis

Catch up on last week's news here

Repository Updates

The following updates are available in the NKEE and NKSE repository...

• cache-ee 1.4.1
  • Updated to eliminate potential of integer overflow in threshold for very long lived cache items.

On Modularity

Having last week discussed how NetKernel implements its modularity, this week we're faced with a more important question: How do you decide how to partition your system into modules? What goes in which box?

In the world of classical Java servlets, the decision is pretty much made for you. Libraries go in jar's. The application is a set of classes perhaps in a directory structure but perhaps also in a jar. Everything goes into one big WAR container.

When you work down at the physical "language level", system partitioning is necessarily constrained by the tight low-level coupling of code - no matter which language you are working in.

Conversely, the logical and transient coupling of ROC means that we have a lot more freedom.

Pragmatically, we'd like to embrace ROC's modular freedom to make our solution's lifetime economic cost sit in a sweet spot.

We can instinctively sense that the two extremes are not in the sweet spot.

Putting everything in a single module results in a monolithic solution with little opportunity for progressive refinement, enhancements or reuse via recomposition. Equally, putting each endpoint in its own module offers the ultimate in flexibility but is entirely impractical both for deployment and for development.

As with any problem in engineering - the game is to find a configuration that strikes the right balance.

But, also with any engineering solution, there's no single answer. We need to use our judgement and experience - but that's not say there aren't clear factors to weigh in the balance...

Factors Affecting Partitioning

The following factors are valuable aspects to consider in the partitioning of a modular ROC system. Note, they are not exclusive, even when your problem has clearly defined characteristics in one category it may still be further modularised based upon other factors...

Domain Knowledge

Expertise in a domain requiring specialist knowledge is a natural indicator of a modular boundary condition. Group capabilities that require specialist knowledge into their own modules.

Examples

• a tax calculation service would be a very natural thing to put in a module and could be used as black-box unit in an overall solution.
• external connectivity to 3rd party services with a requirement for knowledge of their REST APIs.
• CRUD data operations on a system of record.

Rate of Change

Grouping together things that are likely to evolve at the same rate is a very important determinant of a system's long-term economic value. The rate of change will have slow moving units and fast moving units. Keeping them decoupled is a good thing.

For example, a solution will have some modules with very stable long-term architectural structure that might never require any attention. Equally a smoothly operating team will be looking to have a continuous flow of regular updates and enhancements with perhaps a monthly or even weekly release cycle.

Examples

• Transports are usually put in fulcrum modules and once created are generally very stable
• Architectural frameworks are generally stable over the long term - for example the NetKernel admin tools, the documentation system etc.
• 3rd party service connectivity is often medium term stable
• Application / Solution channels are more volatile and need to be able to be rapidly evolved.

Physical Resource Model

Sometimes we are required to utilise an existing investment in a code library or physical resource model. Invariably a legacy codebase will dictate language choice and also physical classpath requirements. This is a natural physical level pressure to group this into a module.

Interestingly, its perfectly reasonable to use an NK module purely as a Java class library. By having an empty rootspace and exporting its internal classpath an NK module can be an isolated and versioned library that can be imported by other modules. By selectively exporting only a defined subset of the classpath one can also provide much stronger classpath encapsulation than with classical Java application servers.

Creative Freedom

It depresses me that classical software development often marginalises creative freedom. Software is a richer and more expressive medium for creativity than any other. Creative software solutions are also the ones that are successful and which customers value and will regularly use.

Partitioning modules so that developers can take personal responsibility for that module as an isolated unit allows them to deliver the minimum necessary result to satisfy a given design brief, but, more often, it provides an opportunity to exceed the brief with innovative and creative contributions.

Modularity unleashes creativity.

Developer Specialism

Through modularity it is possible to create a heterogeneous mix of skillsets in the development team. It could be a low-level specialism in a library or technology. It could be in an aspect of the solution - for example it might be graphic design skills in content and presentation.

Modularity allows specialisms to be focused. Of course, if you're ruthless, it also means that specialisms can be replaced with alternates should this be a useful requirement to the overall solution.

Policy Knowledge and Compliance

It is generally not sufficient to build a solution. It is also important to show that it respects compliance requirements and satisfies necessary constraints.

By putting compliance enforcement into modules applied in the boundary conditions of an architecture the orthogonal pressures to satisfy the demands for corporate governance do not impact the progressive development cycle of the first order solution.

Real World Example: NetKernel Portal Modularity Analysis

From the discussion above and your own instincts you'll already have some sense of how to partition modularity. But unfortunately in engineering there's no substitute for experience and I can't hand you down a magic recipe.

Instead it might help to take a look at the high level modular structure of a large system.

Below is a thirty thousand foot view of the NetKernel portal (beneath it is a link to an SVG image so you can zoom in on the details).

SVG

Each of the rectangular boxes is a module - twenty in total. You can see that most modules contain more than one address space but lets not concern ourselves with the logical ROC architecture. Lets think about the physical modularity and the reasons things are partitioned into this set of boxes.

Starting on the left, there is clearly a primary focus for the system - this is the "application fulcrum" and it is providing transport level architectural structure such as throttle and HTTPSession overlay. Essentially this is invariant and once created it rarely if ever will need to be changed. Naturally it goes in a module that to all intents and purposes is an architectural "time capsule" - we should never need to open it to change the insides.

Next in, we see the compliance and audit layers - these are boundary conditions that ensure that security constraints are enforced, and that authenticated users are given membership of appropriate privilege groups and that generic application-level channel usage logging is captured.

Following that, we have the portal framework - this is the overlay which provides the tabular portal structure that the end-user sees as "the portal" - in fact this is a simple wrapper layer that simply decorates the access to each individual portal channel provided from the layers below.

All of these top tiers are stable and, from the solution evolution's perspective have very slow (or zero) rates of change. Each of these units was developed independently of the other and so specialist domain knowledge is kept isolated and if necessary they can be replaced without impacting anything in the system. None of these modules have any "physical code" - they are all just architectural compositions assembled by configuring overlays, mappers and dynamic imports etc.

Moving to the center of the diagram we see the "middle tier" which consists of the application "portlets". These are the independent application channels whose capabilities are merged by the layers above to present the user-level "portal".

These items quite naturally are bite-sized units that can be given to individual developers or if necessary developer teams for autonomous responsibility.

For some of the simpler portal channels, providing a small but defined capability (like the NKEE download portal), we've used a single module for the entire portal.

For the more complex portals (like the customer support services) we sub-partitioned the problem. Often there will be a module for the front-end rendering layer - the web view and application which will also have a corresponding set of data services which provide access to the DB. Keeping these two pieces separate allows for independent development, isolation of domain knowledge, isolation of testing and independent rates of evolution.

Finally, looking at the right hand side, we see a set of "system data services". These are the services which are common to more than one portal, or, even, more than one layer of the architecture. Frequently a service that was discovered to be common would have been migrated from a dedicated portal module down to the data services layer.

The modular configuration of this portal solution enables the overall system to be progressively enhanced and is structured so that new capabilities can be introduced with zero impact on the existing solution. Our experience is that this modular arrangement has a very low economic cost of change.

Hopefully the criterion discussed above and this practical tour of a real world solution gives you a starting point for planning your own modular partitions. Of course it wouldn't be the newsletter if I didn't blatently end by plugging that if you need assistance we're available for consultancy advise and practical help!