Monday, November 16, 2015

Summary of open discussion at the Nov 2015 Machinekit Meetup

On friday afternoon and sunday morning we held open discussions about a range of themes, and formed an opinion on how to proceed with the project. I started out with a recap of the original goals, as it helps focusing on the road ahead:

Goal of the Machinekit Project

The core goal was and still is to make the HAL/realtime stack usable as a component of a larger project, and enable remote operation on a wide range of platforms (the "realtime appliance" idea), including new remote UI techniques.

The existing CNC stack will continue to work, on top of the separated HAL stack, and should generally run existing configurations as is (or with trivial modifications).

Direction of the Machinekit project

To summarize the key changes of direction - we agreed the best course of action to be:

  • split the project horizontally into packages (and eventually separate git repositories): a CNC stack, the HAL/realtime stack, and the machinetalk support package for remote clients. The CNC package will depend on the HAL/RT and machinetalk packages.
    This is the key change: making the HAL stack reusable by other projects.
  • end support for the - essentially unused - RTAI and Xenomai-kernel flavors - this is required to simplify the build process and dependencies, and ease the previous step.
To understand what led up to this - here are the items discussed as I recollect them (with some personal observations added in):

Topics discussed

Code size: the machinekit repository is a huge monolithic "blob". It contains everything from very low-level stuff to UI's (many of them deprecated), plus the whole CNC and realtime stacks. The amount of code is staggering - comparable in code size to mysql3 or roughly three times the Emacs editor source code, with a developer base diameter which is a fraction of comparable projects.

Build complexity: Having everything - used or not - in one blob adds several dimensions of complexity: different RT kernels, various user interfaces, and the ongoing transition of replacing the middleware stack. This has resulted in a staggering number of prerequisite packages, which are very hard to get right, as the mailing list and tracker shows. The RTAI kernel threads build has added to that by making it very hard to use more standard tools like cmake, causing us to limp along with the legacy build scripts.

Reuse in other projects: The HAL/realtime stack is one-of-a-kind and would be a great building block for other projects, like ROS, where it could be used as a 'realtime playout' and hardware interaction vehicle. And that part is actually rather small compared to the whole repository. But fact is: the unmanageable code size, build dependencies and build system make it extremely hard to adopt the realtime stack in a separate project - the complexity involved essentially precludes this, and this is why it did not happen. As things stand, adopting the realtime stack into another project is very unlikely to happen. So our priority must be to jettison complexity and simplify things wherever we can.

Different rates of change: Most of the development around Machinekit has been in the HAL/realtime and machinetalk areas, and the UI development based thereon. But I think the major changes are in place or readying, and things are stabilizing overall. The actual CNC stack ontop so far has seen only minor changes. While the CNC stack will eventually be migrated to machinetalk, we are only part of the way into this work.

Tentative machinekit "stable branch": There have been suggestions to fork off a stable branch, focusing more on a versioned release rather than rapid development. This certainly makes sense for folks desiring to run off a less volatile code base, and the C4 process does encourage that. But it is important to note that this actually adds to the problems listed above while solving none of them - the "blob", the build complexity, external dependency, reusability: that all stays where it is, and would be managed in separate branches, spreading resources even thinner.

Talking to Machinekit remotely: Both Alex and Bob have presented remote API's into Machinekit (Python and node.js), and remote clients will need to make use of the machinetalk stack to some extent - but certainly not the whole repository: requiring to build Machinekit on a remote client just for this part defeats the very purpose of going remote, which is to simplify things on the remote client rather than incur a build on a possibly separate platform. This suggests spinning out the required parts of machinetalk into a package which can be build separately (and more easily on target), and end managing protobuf as git subtree as we do now. We created the basic capability to use Machinekit remotely - we now need to wrap it such that this can be used by mere mortals.

Random bits and loose ends

  • We will be maintaining a build server, CI builds (likely separate from package builds), and package repos for all currently supported architectures. 
  • I was surprised in the very strong interest in remote both remote operations, and actual use of the new remote UI's. Some of this comes from the new capabilities,  and some from the relative freedom of licensing when creating a remote Machinekit UI.
  • The installation counts are significant and much higher than I thought - around 1400 distinct SD image downloads, and several hundred package users.
  • the resulting HAL/realtime package will have external package dependencies which are a fraction of the current list - making it much easier to reuse in other projects. 
  • The current HAL revision under way suggests using a vehicle for inline API documentation (possibly doxygen or asciidoc-based as the zeroMQ project uses it)

Plans for a Machinekit Foundation

An open source project on github only goes so far. At times it makes sense to have a bit more formal structure, and a legal entity at hand - for improved professional standing, but also as an endpoint for corporate or consortia memberships like EtherCAT or i-ROS. There is no intent to create any superstructure over the Machinekit project per se - rather a utility to be used when appropriate.

One of the obvious purposes of such an entity could be to hold the domain name ownerships of the project (I currently hold these at my private expense), which will remove another dependency on a single person. I'd be happy to transfer ownership eventually. Same thing could go say for hosting contracts - non-profits also get usually better rates than individuals.

The role model we considered is the Open Source Robotics Foundation,  a not-for-profit organization under American law. It turned out that setting up such an entity can be done with moderate financial and recurring paperwork exposure. The majority felt this would be a useful step to take, and we will strive to have this entity setup before our next meetup.

Sunday, November 15, 2015

And the winner is: RT-PREEMPT

Many might remember that one of the key contributions of the unified build branch of LinuxCNC - which eventually turned into Machinekit - was providing support to multiple realtime kernels. At the time, LinuxCNC only could make use of RTAI, distributing a well-aged version thereof.

RTAI still yields the best latency figures. But that comes at a huge cost: having application code run in-kernel is not only unsafe at any speed, but fraught with an enormous build complexity, kernel version dependency and recurring maintenance chores. And that has not changed - as well as the restriction that RTAI runs on Intel architectures only. Moreover, the future of the RTAI project has become clouded as it has always been a bit of a one-man show lacking a healthy community around it to take over just in case.

The alternatives supported by Machinekit are Xenomai and RT-PREEMPT.

Xenomai shares some history with RTAI - both are hypervisor kernels: the idea is to have a minimal, RT-capable scheduler and interrupt handler underneath the actual Linux kernel. RT threads use this hypervisor to achieve better timing than possible with a vanilla Linux kernel. Other than RTAI - where RT applications need to run in-kernel as modules similar to device drivers, Xenomai supports a threading model which almost looks like normal Posix threads - except that only rather restricted use of the Linux API can be made from such a thread. Xenomai does support a wide range of architectures, which is why it is the mainstay of running Machinekit on ARM platforms.  Again this comes at a cost: Xenomai still requires a rather intrusive kernel patch and is available for a limited range of underlying Linux kernel versions. And given the fact that many embedded manufacturers choose to use a rather specific, sometimes outdated kernel version and sometimes do not upstream their patches into the Torvalds mainline Linux kernel, the chances for getting a working Xenomai kernel for such platforms is pretty low.  That is the main reason for both RTAI and Xenomai kernels being "well aged" on most platforms.

The third alternative is RT-PREEMPT - a set of patches to the standard Linux kernel to improve it's timing behavior, but without introducing a separate API, or a hypervisor. This project  has been over a decade in the making, and at times there were doubts if it would make it into Linux mainline kernel. Initially being substantially higher latency than the hypervisor breed,  over the last year or so huge progress has been made in terms of performance delivered on Intel platforms in particular, but also on ARM platforms: I recently tried an RT-PREEMPT kernel on a Raspberry-2 and it delivers slightly better latency than Xenomai on the Beaglebone, so it's getting pretty close.

In the past, the argument for minimum latency has been its usefulness for software-based step generation and quadrature encoders - the actual servo cycle computations do not need that low latency. But fact is - the PC's parallel port is an extinct piece of hardware (and even then RT-PREEMPT can deliver reasonable software step rates). Plus, inexpensive FPGA hardware can deliver higher performance if needed.

Picking one among the above choices needs to made by all users of realtime applications, not just us. Some funding for RT work has come from the financial industry for high-frequency trading in the past, and the automotive industry has a healthy interest as well, demonstrated for instance by the participation in the Linux Realtime Workshop conference series. This revolves mostly around the autonomous driving theme.

There was some real good news recently: the Linux Foundation announced that it will adopt the RT-PREEMPT project with the goal of bringing it into the mainline kernel (note list of sponsors!). This assures the funding of the remaining work, and IMO it is reasonable to expect that in a few years - probably more than one, but certainly less than five - obtaining a RT-PREEMPT kernel will be just a build option of the mainline kernel. Already now building kernels is much, much simpler than any of the other options - and once that effort goes mainline and manufacturers actually support that kernel, it will be much simpler to obtain RT kernels for any platform, not just a few select ones.

Is interesting to note that the Xenomai3 effort provides a common API over both the hypervisor-style and RT-PREEMPT kernels. Since Xenomai seems to enjoy a healthy industrial user base, it is important to offer a migration path to its users towards what - probably not only I - consider the winner of the RT kernels competition.

These developments have some far-reaching implications for the Machinekit project, some of which were discussed at the recent meetup.  Among those are:

  • the performance edge provided by RTAI has become so small that it is by far outweighed by the enormous build complexity it entails for Machinekit. We have therefore decided to end support for RTAI.
  • With RT-PREEMPT very likely to go mainline, it is likely efforts are made by hardware vendors to support this rather than other options, meaning both range of supported hardware will widen, as well as functionality and performance improvements to show up here first. It will take a while until this "sinks in" with all involved, but the direction is clear.
  • For the time being we'll retain the Xenomai2 builds; while Xenomai2 is in maintenance mode already, we do have stable kernels around which perform great. And the build process is easy and robust. But there is not much point in a Xenomai3 port to just run RT-PREEMPT underneath - the rt-preempt flavor already does that. Any Xenomai3 hypervisor flavor needs to be weighed against the performance edge it has over RT-PREEMPT, and it looks like this edge is shrinking.
  • Ending support for kernel-threads enables removing the absurdly complex legacy build system by more mainstream tools like cmake.
  • using userland threads exclusively opens new options: so far HAL realtime code was restricted to C, as C++ is not supported for kernel modules. This has both potential for simplifying HAL itself, as well as making it easier to integrate with C++-based systems like ROS and Orocos, or bringing in improved solutions to old problems.

Tuesday, November 10, 2015

Machinekit meetup - summary of talks

Here's a list of all talks held at the meetup, including links to recorded video and slides:
  • Bob van der Linden, node.js-machinetalk, an upcoming node.js - based remote API into the machinekit CNC and HAL stacks (slides)
  • Alexander Rössler, python-machinetalk, on a Python-based remote API into the machinekit CNC and HAL stacks (video slides)
  •  Charles Steinkuehler, Platform Options for Machinekit: Current and Future, review and outlook on "what is cooking in the hardware space", (video)
  • Charles Steinkuehler, The need and use of multicore HAL (video slides)
  • Michael Haberler & Mick Grant, HAL “multiKore”: making HAL safe & fast for multicore platforms - about enabling safe use of multiple core processors for realtime tasks (video slides)
  • Bas de Brujin, Update on CANopen driver for HAL (video slides)
  • Jon Elson, PicoSystems, Comments on the success of making CRAMPS boards and retrofit of 1996 photoplotter with a Beagle Bone (video)
  • Alexander Rössler, Building Qt5 UI's with the MachinekitSDK, covering creating user interfaces in the MachineSDK Vagrant VM and future possibilities of the MachinekitSDK (video slides)

Sunday, October 4, 2015

ANNOUNCE: Machinekit Meetup

Nov 6-8/2015, Madison WI

We are please to invite anyone interested to the second "real world" Machinekit meeting, taking place on November 6/8 at the Tormach facility in Madison, Wisconsin (USA).  Please feel free to join us - share your plans, exchange experience, and help shape the future of Machinekit!

Rough agenda so far:
  • Charles will talk on "Platform Options for Machinekit: Current and Future", covering the upcoming newer ARM & ARM+FPGA options besides the existing armhf/x86/amd64 options
  • Alex will talk on "Building Qt5 UI's with the MachinekitSDK"
  • I will present on HALmk ("HAL-MultiKore") - about enabling safe use of multiple core processors for realtime tasks, and the future of kernels for machinekit
Other topics we could think of
  • firming up plans and procedures for a machinekit stable branch
  • status reports from the nodejs-machinetalk and python-machinetalk projects (remote API's)
.. and anything you would want to bring up/contribute to!

More venue and agenda details will be posted shortly, so watch for details, and speak up if you're planning to attend. So far most folks are arriving sometime Friday and leaving sometime Sunday.

Charles, John, Alex, Michael, Bas

Wednesday, June 24, 2015

Denford Novamill Conversion
Designspark user stuartChilds has a multi-part blog entry documenting the conversion of a Denford Novamill with a non-working controller to using a BeagleBone Black running Machinekit with a Probotix cape and Gecko stepper drivers.  Part 1 covers the mechanical and wiring changes, while part 2 details software integration and configuration.  Upcoming posts will cover the spindle drive, CAD/CAM, and "making chips".

Tuesday, March 31, 2015

Machinekit To Aid With Heart Research

Recently I was contacted by a Professor Andreas Lueger of Medical University of Graz, Austria, Dept. of Internal Medicine, probing me with some rather intricate questions on Machinekit. Andreas is head of the emergency room there, helipad on the roof and all. So if you drive recklessly around here, you stand a chance for a helicopter ride and being stitched up by Andreas and his team, but that is certainly not “motion control” as we know it. That made me curious, and I visited Andreas to probe what he is up to:

Andreas in business attire, as found in his professional habitat
Beyond stitching up patients, Andreas is deeply involved in heart research - investigating atrial fibrillation, a precursor of heart attacks. And that is some very geeky project  - for his research, he develops the electronics based on a TI MSP430, as well as the software for his own Wifi-enabled pacemakers - a whole new meaning of “embedded” for me. Andreas brought down current consumption to a couple of hundred nanoampéres, and does solder the parts under his own stereo microscope:

What can I say? at last an MD who talks our language! And when I visited him, he had a bug report for Machinekit, including a screenshot from his oscilloscope. I was impressed.

Now that pacemaker certainly is not running Machinekit, so where do we come in? Well, in the next iteration of the project, Andreas plans to integrate acceleration sensors in the setup, to measure actual mechanical performance of heart muscles. And for manufacturing these sensor attachments Andreas is designing a custom CNC machine with FreeCAD:

Now we’re talking.. that machine will run Machinekit! Here comes the twist: Andreas plans to use Trinamic servo stepper motors. Those are very smart motors with an embedded controller and power stages attached; among other features, these motors can detect and report a stall, and autonomously compensate for lost steps:

Trinamic motors can be controlled through various methods: Step/Direction pins, RS232, RS485, USB, and CANbus. Now out of these options, CANbus is the most interesting one because it operates on a much higher level than the other options; the motor can report back (heartbeat, stall, position, velocity etc).  It’s really a much more intelligent motor subsystem than we usually use in Machinekit, like servos or stepgens.

I found all this terribly geeky, and beyond that also valuable for the project overall: if we investigate and collectively learn how to integrate CANbus, we open up Machinekit to literally thousands of peripherals with a professional bus system - as opposed to homegrown methods like hooking up an Arduino over RS232 or USB. The Beaglebone's TI3359 chip comes with two CAN devices on board - add MCP2562 bus drivers and you are done; or use an appropriate cape. For other platforms, PCI cards and USB adapters do the trick.

CANbus peripherals come in many shapes and forms - like this controller used by Claudio Lorini with support for the zedboard which he recently merged into machinekit; but also more mundane devices like your car's windshield wipers and power windows: yes - your car is a one extensive CANbus installation (provided your car was built in the current millennium). And if you research that topic, you will find there’s a whole “car CANbus” hacking scene out there, using Arduinos and whatnot to tap into their car’s CANbus, like this example

Where does this leave us? I have decided to support Andreas with this project, and while it is not absolutely certain the Trinamics will work as motion units as planned, it will be a very interesting expedition. And along the journey, Machinekit will learn to support CANbus peripherals.

And that exciting story is the background why you saw references to CANbus in my recent postings to the Machinekit list!

Saturday, March 14, 2015

Machinekit Running on Odroid C1

Machinekit already runs on x86 machines, the BeagleBone, and the Raspberry Pi.  Now GP Orcullo reports he has Machinekit running on the Odroid C1 using rt-preempt.  There are no hardware drivers yet, but it shouldn't take long to port GPIO access and get some motors moving.  If you're interested, there's a link to a bootable Debian Wheezy image in the Machinekit list post.

I'm anxiously waiting for boards based around all the new Big/Little parts coming out, which combine Cortex-A cores for good OS performance and Cortex-M parts for real-time tasks.  Then there's the BeagleBoard X15 coming out soon with Big/Little + DSPs + PRUs!  Exciting times in the ARM world!