Summary of open discussion at the Nov 2015 Machinekit Meetup

On friday afternoon and sunday morning we held open discussions about a range of themes, and formed an opinion on how to proceed with the project. I started out with a recap of the original goals, as it helps focusing on the road ahead:

Goal of the Machinekit Project

The core goal was and still is to make the HAL/realtime stack usable as a component of a larger project, and enable remote operation on a wide range of platforms (the "realtime appliance" idea), including new remote UI techniques.

The existing CNC stack will continue to work, on top of the separated HAL stack, and should generally run existing configurations as is (or with trivial modifications).

Direction of the Machinekit project

To summarize the key changes of direction - we agreed the best course of action to be:

• split the project horizontally into packages (and eventually separate git repositories): a CNC stack, the HAL/realtime stack, and the machinetalk support package for remote clients. The CNC package will depend on the HAL/RT and machinetalk packages.
  This is the key change: making the HAL stack reusable by other projects.
• end support for the - essentially unused - RTAI and Xenomai-kernel flavors - this is required to simplify the build process and dependencies, and ease the previous step.

To understand what led up to this - here are the items discussed as I recollect them (with some personal observations added in):

Topics discussed

Code size: the machinekit repository is a huge monolithic "blob". It contains everything from very low-level stuff to UI's (many of them deprecated), plus the whole CNC and realtime stacks. The amount of code is staggering - comparable in code size to mysql3 or roughly three times the Emacs editor source code, with a developer base diameter which is a fraction of comparable projects.

Build complexity: Having everything - used or not - in one blob adds several dimensions of complexity: different RT kernels, various user interfaces, and the ongoing transition of replacing the middleware stack. This has resulted in a staggering number of prerequisite packages, which are very hard to get right, as the mailing list and tracker shows. The RTAI kernel threads build has added to that by making it very hard to use more standard tools like cmake, causing us to limp along with the legacy build scripts.

Reuse in other projects: The HAL/realtime stack is one-of-a-kind and would be a great building block for other projects, like ROS, where it could be used as a 'realtime playout' and hardware interaction vehicle. And that part is actually rather small compared to the whole repository. But fact is: the unmanageable code size, build dependencies and build system make it extremely hard to adopt the realtime stack in a separate project - the complexity involved essentially precludes this, and this is why it did not happen. As things stand, adopting the realtime stack into another project is very unlikely to happen. So our priority must be to jettison complexity and simplify things wherever we can.

Different rates of change: Most of the development around Machinekit has been in the HAL/realtime and machinetalk areas, and the UI development based thereon. But I think the major changes are in place or readying, and things are stabilizing overall. The actual CNC stack ontop so far has seen only minor changes. While the CNC stack will eventually be migrated to machinetalk, we are only part of the way into this work.

Tentative machinekit "stable branch": There have been suggestions to fork off a stable branch, focusing more on a versioned release rather than rapid development. This certainly makes sense for folks desiring to run off a less volatile code base, and the C4 process does encourage that. But it is important to note that this actually adds to the problems listed above while solving none of them - the "blob", the build complexity, external dependency, reusability: that all stays where it is, and would be managed in separate branches, spreading resources even thinner.

Talking to Machinekit remotely: Both Alex and Bob have presented remote API's into Machinekit (Python and node.js), and remote clients will need to make use of the machinetalk stack to some extent - but certainly not the whole repository: requiring to build Machinekit on a remote client just for this part defeats the very purpose of going remote, which is to simplify things on the remote client rather than incur a build on a possibly separate platform. This suggests spinning out the required parts of machinetalk into a package which can be build separately (and more easily on target), and end managing protobuf as git subtree as we do now. We created the basic capability to use Machinekit remotely - we now need to wrap it such that this can be used by mere mortals.

Random bits and loose ends

• We will be maintaining a build server, CI builds (likely separate from package builds), and package repos for all currently supported architectures. 
• I was surprised in the very strong interest in remote both remote operations, and actual use of the new remote UI's. Some of this comes from the new capabilities,  and some from the relative freedom of licensing when creating a remote Machinekit UI.
• The installation counts are significant and much higher than I thought - around 1400 distinct SD image downloads, and several hundred package users.
• the resulting HAL/realtime package will have external package dependencies which are a fraction of the current list - making it much easier to reuse in other projects. 
• The current HAL revision under way suggests using a vehicle for inline API documentation (possibly doxygen or asciidoc-based as the zeroMQ project uses it)

Plans for a Machinekit Foundation

An open source project on github only goes so far. At times it makes sense to have a bit more formal structure, and a legal entity at hand - for improved professional standing, but also as an endpoint for corporate or consortia memberships like EtherCAT or i-ROS. There is no intent to create any superstructure over the Machinekit project per se - rather a utility to be used when appropriate.

One of the obvious purposes of such an entity could be to hold the domain name ownerships of the project (I currently hold these at my private expense), which will remove another dependency on a single person. I'd be happy to transfer ownership eventually. Same thing could go say for hosting contracts - non-profits also get usually better rates than individuals.

The role model we considered is the Open Source Robotics Foundation,  a not-for-profit organization under American law. It turned out that setting up such an entity can be done with moderate financial and recurring paperwork exposure. The majority felt this would be a useful step to take, and we will strive to have this entity setup before our next meetup.