Summary of open discussion at the Nov 2015 Machinekit Meetup

On friday afternoon and sunday morning we held open discussions about a range of themes, and formed an opinion on how to proceed with the project. I started out with a recap of the original goals, as it helps focusing on the road ahead:

Goal of the Machinekit Project

The core goal was and still is to make the HAL/realtime stack usable as a component of a larger project, and enable remote operation on a wide range of platforms (the "realtime appliance" idea), including new remote UI techniques.

The existing CNC stack will continue to work, on top of the separated HAL stack, and should generally run existing configurations as is (or with trivial modifications).

Direction of the Machinekit project

To summarize the key changes of direction - we agreed the best course of action to be:

• split the project horizontally into packages (and eventually separate git repositories): a CNC stack, the HAL/realtime stack, and the machinetalk support package for remote clients. The CNC package will depend on the HAL/RT and machinetalk packages.
  This is the key change: making the HAL stack reusable by other projects.
• end support for the - essentially unused - RTAI and Xenomai-kernel flavors - this is required to simplify the build process and dependencies, and ease the previous step.

To understand what led up to this - here are the items discussed as I recollect them (with some personal observations added in):

Topics discussed

Code size: the machinekit repository is a huge monolithic "blob". It contains everything from very low-level stuff to UI's (many of them deprecated), plus the whole CNC and realtime stacks. The amount of code is staggering - comparable in code size to mysql3 or roughly three times the Emacs editor source code, with a developer base diameter which is a fraction of comparable projects.

Build complexity: Having everything - used or not - in one blob adds several dimensions of complexity: different RT kernels, various user interfaces, and the ongoing transition of replacing the middleware stack. This has resulted in a staggering number of prerequisite packages, which are very hard to get right, as the mailing list and tracker shows. The RTAI kernel threads build has added to that by making it very hard to use more standard tools like cmake, causing us to limp along with the legacy build scripts.

Reuse in other projects: The HAL/realtime stack is one-of-a-kind and would be a great building block for other projects, like ROS, where it could be used as a 'realtime playout' and hardware interaction vehicle. And that part is actually rather small compared to the whole repository. But fact is: the unmanageable code size, build dependencies and build system make it extremely hard to adopt the realtime stack in a separate project - the complexity involved essentially precludes this, and this is why it did not happen. As things stand, adopting the realtime stack into another project is very unlikely to happen. So our priority must be to jettison complexity and simplify things wherever we can.

Different rates of change: Most of the development around Machinekit has been in the HAL/realtime and machinetalk areas, and the UI development based thereon. But I think the major changes are in place or readying, and things are stabilizing overall. The actual CNC stack ontop so far has seen only minor changes. While the CNC stack will eventually be migrated to machinetalk, we are only part of the way into this work.

Tentative machinekit "stable branch": There have been suggestions to fork off a stable branch, focusing more on a versioned release rather than rapid development. This certainly makes sense for folks desiring to run off a less volatile code base, and the C4 process does encourage that. But it is important to note that this actually adds to the problems listed above while solving none of them - the "blob", the build complexity, external dependency, reusability: that all stays where it is, and would be managed in separate branches, spreading resources even thinner.

Talking to Machinekit remotely: Both Alex and Bob have presented remote API's into Machinekit (Python and node.js), and remote clients will need to make use of the machinetalk stack to some extent - but certainly not the whole repository: requiring to build Machinekit on a remote client just for this part defeats the very purpose of going remote, which is to simplify things on the remote client rather than incur a build on a possibly separate platform. This suggests spinning out the required parts of machinetalk into a package which can be build separately (and more easily on target), and end managing protobuf as git subtree as we do now. We created the basic capability to use Machinekit remotely - we now need to wrap it such that this can be used by mere mortals.

Random bits and loose ends

• We will be maintaining a build server, CI builds (likely separate from package builds), and package repos for all currently supported architectures. 
• I was surprised in the very strong interest in remote both remote operations, and actual use of the new remote UI's. Some of this comes from the new capabilities,  and some from the relative freedom of licensing when creating a remote Machinekit UI.
• The installation counts are significant and much higher than I thought - around 1400 distinct SD image downloads, and several hundred package users.
• the resulting HAL/realtime package will have external package dependencies which are a fraction of the current list - making it much easier to reuse in other projects. 
• The current HAL revision under way suggests using a vehicle for inline API documentation (possibly doxygen or asciidoc-based as the zeroMQ project uses it)

Plans for a Machinekit Foundation

An open source project on github only goes so far. At times it makes sense to have a bit more formal structure, and a legal entity at hand - for improved professional standing, but also as an endpoint for corporate or consortia memberships like EtherCAT or i-ROS. There is no intent to create any superstructure over the Machinekit project per se - rather a utility to be used when appropriate.

One of the obvious purposes of such an entity could be to hold the domain name ownerships of the project (I currently hold these at my private expense), which will remove another dependency on a single person. I'd be happy to transfer ownership eventually. Same thing could go say for hosting contracts - non-profits also get usually better rates than individuals.

The role model we considered is the Open Source Robotics Foundation,  a not-for-profit organization under American law. It turned out that setting up such an entity can be done with moderate financial and recurring paperwork exposure. The majority felt this would be a useful step to take, and we will strive to have this entity setup before our next meetup.

And the winner is: RT-PREEMPT

Many might remember that one of the key contributions of the unified build branch of LinuxCNC - which eventually turned into Machinekit - was providing support to multiple realtime kernels. At the time, LinuxCNC only could make use of RTAI, distributing a well-aged version thereof.

RTAI still yields the best latency figures. But that comes at a huge cost: having application code run in-kernel is not only unsafe at any speed, but fraught with an enormous build complexity, kernel version dependency and recurring maintenance chores. And that has not changed - as well as the restriction that RTAI runs on Intel architectures only. Moreover, the future of the RTAI project has become clouded as it has always been a bit of a one-man show lacking a healthy community around it to take over just in case.

The alternatives supported by Machinekit are Xenomai and RT-PREEMPT.

Xenomai shares some history with RTAI - both are hypervisor kernels: the idea is to have a minimal, RT-capable scheduler and interrupt handler underneath the actual Linux kernel. RT threads use this hypervisor to achieve better timing than possible with a vanilla Linux kernel. Other than RTAI - where RT applications need to run in-kernel as modules similar to device drivers, Xenomai supports a threading model which almost looks like normal Posix threads - except that only rather restricted use of the Linux API can be made from such a thread. Xenomai does support a wide range of architectures, which is why it is the mainstay of running Machinekit on ARM platforms.  Again this comes at a cost: Xenomai still requires a rather intrusive kernel patch and is available for a limited range of underlying Linux kernel versions. And given the fact that many embedded manufacturers choose to use a rather specific, sometimes outdated kernel version and sometimes do not upstream their patches into the Torvalds mainline Linux kernel, the chances for getting a working Xenomai kernel for such platforms is pretty low.  That is the main reason for both RTAI and Xenomai kernels being "well aged" on most platforms.

The third alternative is RT-PREEMPT - a set of patches to the standard Linux kernel to improve it's timing behavior, but without introducing a separate API, or a hypervisor. This project  has been over a decade in the making, and at times there were doubts if it would make it into Linux mainline kernel. Initially being substantially higher latency than the hypervisor breed,  over the last year or so huge progress has been made in terms of performance delivered on Intel platforms in particular, but also on ARM platforms: I recently tried an RT-PREEMPT kernel on a Raspberry-2 and it delivers slightly better latency than Xenomai on the Beaglebone, so it's getting pretty close.

In the past, the argument for minimum latency has been its usefulness for software-based step generation and quadrature encoders - the actual servo cycle computations do not need that low latency. But fact is - the PC's parallel port is an extinct piece of hardware (and even then RT-PREEMPT can deliver reasonable software step rates). Plus, inexpensive FPGA hardware can deliver higher performance if needed.

Picking one among the above choices needs to made by all users of realtime applications, not just us. Some funding for RT work has come from the financial industry for high-frequency trading in the past, and the automotive industry has a healthy interest as well, demonstrated for instance by the participation in the Linux Realtime Workshop conference series. This revolves mostly around the autonomous driving theme.

There was some real good news recently: the Linux Foundation announced that it will adopt the RT-PREEMPT project with the goal of bringing it into the mainline kernel (note list of sponsors!). This assures the funding of the remaining work, and IMO it is reasonable to expect that in a few years - probably more than one, but certainly less than five - obtaining a RT-PREEMPT kernel will be just a build option of the mainline kernel. Already now building kernels is much, much simpler than any of the other options - and once that effort goes mainline and manufacturers actually support that kernel, it will be much simpler to obtain RT kernels for any platform, not just a few select ones.

Is interesting to note that the Xenomai3 effort provides a common API over both the hypervisor-style and RT-PREEMPT kernels. Since Xenomai seems to enjoy a healthy industrial user base, it is important to offer a migration path to its users towards what - probably not only I - consider the winner of the RT kernels competition.

These developments have some far-reaching implications for the Machinekit project, some of which were discussed at the recent meetup.  Among those are:

• the performance edge provided by RTAI has become so small that it is by far outweighed by the enormous build complexity it entails for Machinekit. We have therefore decided to end support for RTAI.
• With RT-PREEMPT very likely to go mainline, it is likely efforts are made by hardware vendors to support this rather than other options, meaning both range of supported hardware will widen, as well as functionality and performance improvements to show up here first. It will take a while until this "sinks in" with all involved, but the direction is clear.
• For the time being we'll retain the Xenomai2 builds; while Xenomai2 is in maintenance mode already, we do have stable kernels around which perform great. And the build process is easy and robust. But there is not much point in a Xenomai3 port to just run RT-PREEMPT underneath - the rt-preempt flavor already does that. Any Xenomai3 hypervisor flavor needs to be weighed against the performance edge it has over RT-PREEMPT, and it looks like this edge is shrinking.
• Ending support for kernel-threads enables removing the absurdly complex legacy build system by more mainstream tools like cmake.
• using userland threads exclusively opens new options: so far HAL realtime code was restricted to C, as C++ is not supported for kernel modules. This has both potential for simplifying HAL itself, as well as making it easier to integrate with C++-based systems like ROS and Orocos, or bringing in improved solutions to old problems.

Machinekit meetup - summary of talks

Here's a list of all talks held at the meetup, including links to recorded video and slides:

• Bob van der Linden, node.js-machinetalk, an upcoming node.js - based remote API into the machinekit CNC and HAL stacks (slides)

• Alexander Rössler, python-machinetalk, on a Python-based remote API into the machinekit CNC and HAL stacks (video slides)

•  Charles Steinkuehler, Platform Options for Machinekit: Current and Future, review and outlook on "what is cooking in the hardware space", (video)

• Charles Steinkuehler, The need and use of multicore HAL (video slides)

• Michael Haberler & Mick Grant, HAL “multiKore”: making HAL safe & fast for multicore platforms - about enabling safe use of multiple core processors for realtime tasks (video slides)

• Bas de Brujin, Update on CANopen driver for HAL (video slides)

• Jon Elson, PicoSystems, Comments on the success of making CRAMPS boards and retrofit of 1996 photoplotter with a Beagle Bone (video)

• Alexander Rössler, Building Qt5 UI's with the MachinekitSDK, covering creating user interfaces in the MachineSDK Vagrant VM and future possibilities of the MachinekitSDK (video slides)