NetKernel News Volume 8 Issue 1

May 26th 2017

• Repository Updates
• Status Report
• HowTo - module.xml Code Completion/Validation
• HowTo - Developing dynamically deployed NetKernel modules using both Gradle and Eclipse simultaneously
• HowTo - Integrate NetKernel XUnit testing with Eclipse JUnit Testing
• High Performance Computing and ROC - Articles

Repository Updates

Sorry for the very low profile of late. As you will have inferred bandwidth has been consumed by the insight that NetKernel and ROC have powerful answers to hard problems in the world of FinTech. We'll have a lot more to say about this soon.

In the mean time there has been lots going on. The list of updates in Apposite indicates that many continual refinements have continued to be released.

I'd hoped to have time to write a more in depth comment piece - there's plenty to share - but for today, here's a set of productivity tips which hopefully will be valuable...

PS If you're experienced with Python and its extensive data science, FinTech libraries and would be interested in trying out integration of native python/libs with ROC and NetKernel then please drop me a line - we have a very powerful new runtime that seamlessly bridges ROC and Native Python. Think native performance quant calculations with ROC resource management and dependency caching!

(We are waiting for a bugfix we submitted to the JEP project to be pushed out in official release. We need this fixed so as to support dynamic python transreption (compiling) on both Python 2.7 and 3. As soon as JEP 3.7 is published we'll ship our new runtinme in apposite.)

A RelaxNG schema for the standard module module.xml is provided out of the box...

http://localhost:1060/tools/schema/standard-module-2.0-schema-1.0.rng

The reason it's RelaxNG is that the open structure required by the module definition, such as, user defined <meta/> data and the extensible configuration of endpoints parameters means that its just not possible to represent this flexibility with rigid XSD schemas. This is a problem since most IDEs only have basic XML editors with at best XSD validation. Better XML editors can use RelaxNG to provide code completion and validation.

I recently discovered that its trivial to configure the Atom editor as a code completing NetKernel module.xml editor.

Here's the recipe...

1. Get a copy of Atom editor https://atom.io/
2. Install the package "linter-autocomplete-jing"
3. Accept any additional package dependencies
4. Add the following line to the top of your module.xml (NK safely ignores this so it makes no difference to deployment)

<?xml-model
      href="http://resources.1060research.com/schema/standard-module-2.0-schema-1.0.rng"
      type="application/xml"
      schematypens="http://relaxng.org/ns/structure/1.0"?>

Your module.xml looks something like this (note it must be declared at the top before the root <module/> element)

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model
       href="http://resources.1060research.com/schema/standard-module-2.0-schema-1.0.rng"
       type="application/xml"
       schematypens="http://relaxng.org/ns/structure/1.0"?>
<module version="2.0">
     <!--Now type away with auto-complete-->
</module>

You are now all set with full code completion and validity checking when editing your module.xml

What sort of things does this help with? Well all the built-in standard endpoints are covered by the schema such as

• system - classpath export, dynamic etc
• fileset - options for attributes and <regex/>, <glob/> etc
• mapper - config for endpoints with <grammar/> completion and declarative <request/> assistance
• pluggable-overlay - options for request etc.
• overlay - <space/> and space endpoints recursively.

Not counting all the other powerful extensions you can install in Atom.

Local/Offline

If you want to work offline you can change the reference to use the local copy served from your running NetKernel...

<?xml-model
      href="http://localhost:1060/tools/schema/standard-module-2.0-schema-1.0.rng"
      type="application/xml"
      schematypens="http://relaxng.org/ns/structure/1.0"?>

Which is still relatively portable but requires anyone editing this to have a copy of NK running. Alternatively you can download this as a file and change this to a file reference - but then you lose any portability.

This recipe lets you set up an Eclipse project to develop a NetKernel module that is dynamically deployed to NetKernel and can be built either using Gradle or by automatic edit driven compilation in Eclipse.

You will need to have set up Gradle v3.3.0+ so that it is on your console PATH. You will need Eclipse v4.6.0 with the Buildship v2.0.0+ plugin.

Here's the recipe...

At the console gradle steps...

1. Set up your build.gradle using the NetKernel gradle plugin (see NetKernel gradle plugin docs for details).
2. Make sure your build.gradle also applies the eclipse plugin with the line: apply plugin: 'eclipse'
3. In your project directory type "gradle build" (tests that basic gradle build is working)
4. Type "gradle eclipse" (sets up a fresh gradle project metadata - you can clean up any existing project with "gradle cleanEclipse")

In Eclipse...

1. Right click the Working Set where you want to have the module project
2. Choose Import...
3. Choose General->Existing Projects into Workspace
4. Browse to find Directory, Select and click Finish
5. Right click on the newly created project and choose "Properties"
6. Go to Java BuildPath -> Source at the bottom change target build directory to point to the gradle build dir: build/urn.your.module.urn-your.version.number/
7. Finally tell Eclipse that you can also use Gradle here - Right click on project and choose "Configure->Add Gradle Nature"
8. Right click on project and select "Gradle->Refresh Project"
9. look at the "Gradle Tasks" window - select the Refresh Icon at the top - Gradle will now find all gradle projects and list their tasks. Your project will now be listed. Expand it and choose build task!!

Of course as usual you need to tell NetKernel to use the target build path as dynamic module so either set up a module.d/ entry that references the build target build/urn.your.module.urn-your.version.number/ or alternatively use the NK gradle plugin task "gradle deployModuleXXXXX" for your specified NetKernel instance to do this automatically.

Congratulations - you are now simultaneously building using either auto compile/build directly from Eclipse or manual dev tasks using Gradle. NetKernel will dynamically hot-deploy the changed module whether it is being modified by Eclipse or Gradle.

This one comes courtesy of Richard Smith...

I've just been experimenting with XUnit testing, and thought I'd try to find a way of triggering NetKernel to run the XUnit tests for a module from Eclipse. I know there is a gradle NetKernel task that runs (all?) XUnit tests in a NK instance, but I thought I'd try integrating with Eclipse's jUnit support.

I came up with a proof of concept jUnit test that uses a httpclient library to trigger a local running NK instance to execute a module's XUnit tests, and then parses the returned xml test results to assert the success (or otherwise) of each test:.

https://gist.github.com/rjsmith/7cdbcedee63a7a4bb7dcda9df48af4dd

I'm used to a jUnit - based TDD development workflow in Eclipse, so this would let me run classic jUnit tests on physical endpoint code, as well as running NK - based tests of the module architecture.

Thanks Richard

Tony has written a couple of discussions of high performance computing and ROC/NetKernel...

Travelling Salesman Problem

This paper offers a detailed analysis of the application of NetKernel and Resource oriented computing to the well known Travelling Salesman Problem. It shows how the NetKernel cache shares the compute time optimisations that dynamic programming has for this problem. Further it shows that the NetKernel cache has a number of additional benefits.

http://blog.durablescope.com/post/TravellingSalesmanProblem/

Generalizing Dynamic Programming

Dynamic programming is an approach to efficiently implementing certain classes of algorithms. This paper explores the background and technical details of this approach and compares it's mechanisms to NetKernel caching.

http://blog.durablescope.com/post/DynamicProgramming/

Have a great weekend