Summary of open discussion at the Nov 2015 Machinekit Meetup

On friday afternoon and sunday morning we held open discussions about a range of themes, and formed an opinion on how to proceed with the project. I started out with a recap of the original goals, as it helps focusing on the road ahead:

Goal of the Machinekit Project

The core goal was and still is to make the HAL/realtime stack usable as a component of a larger project, and enable remote operation on a wide range of platforms (the "realtime appliance" idea), including new remote UI techniques.

The existing CNC stack will continue to work, on top of the separated HAL stack, and should generally run existing configurations as is (or with trivial modifications).

Direction of the Machinekit project

To summarize the key changes of direction - we agreed the best course of action to be:

• split the project horizontally into packages (and eventually separate git repositories): a CNC stack, the HAL/realtime stack, and the machinetalk support package for remote clients. The CNC package will depend on the HAL/RT and machinetalk packages.
  This is the key change: making the HAL stack reusable by other projects.
• end support for the - essentially unused - RTAI and Xenomai-kernel flavors - this is required to simplify the build process and dependencies, and ease the previous step.

To understand what led up to this - here are the items discussed as I recollect them (with some personal observations added in):

Topics discussed

Code size: the machinekit repository is a huge monolithic "blob". It contains everything from very low-level stuff to UI's (many of them deprecated), plus the whole CNC and realtime stacks. The amount of code is staggering - comparable in code size to mysql3 or roughly three times the Emacs editor source code, with a developer base diameter which is a fraction of comparable projects.

Build complexity: Having everything - used or not - in one blob adds several dimensions of complexity: different RT kernels, various user interfaces, and the ongoing transition of replacing the middleware stack. This has resulted in a staggering number of prerequisite packages, which are very hard to get right, as the mailing list and tracker shows. The RTAI kernel threads build has added to that by making it very hard to use more standard tools like cmake, causing us to limp along with the legacy build scripts.

Reuse in other projects: The HAL/realtime stack is one-of-a-kind and would be a great building block for other projects, like ROS, where it could be used as a 'realtime playout' and hardware interaction vehicle. And that part is actually rather small compared to the whole repository. But fact is: the unmanageable code size, build dependencies and build system make it extremely hard to adopt the realtime stack in a separate project - the complexity involved essentially precludes this, and this is why it did not happen. As things stand, adopting the realtime stack into another project is very unlikely to happen. So our priority must be to jettison complexity and simplify things wherever we can.

Different rates of change: Most of the development around Machinekit has been in the HAL/realtime and machinetalk areas, and the UI development based thereon. But I think the major changes are in place or readying, and things are stabilizing overall. The actual CNC stack ontop so far has seen only minor changes. While the CNC stack will eventually be migrated to machinetalk, we are only part of the way into this work.

Tentative machinekit "stable branch": There have been suggestions to fork off a stable branch, focusing more on a versioned release rather than rapid development. This certainly makes sense for folks desiring to run off a less volatile code base, and the C4 process does encourage that. But it is important to note that this actually adds to the problems listed above while solving none of them - the "blob", the build complexity, external dependency, reusability: that all stays where it is, and would be managed in separate branches, spreading resources even thinner.

Talking to Machinekit remotely: Both Alex and Bob have presented remote API's into Machinekit (Python and node.js), and remote clients will need to make use of the machinetalk stack to some extent - but certainly not the whole repository: requiring to build Machinekit on a remote client just for this part defeats the very purpose of going remote, which is to simplify things on the remote client rather than incur a build on a possibly separate platform. This suggests spinning out the required parts of machinetalk into a package which can be build separately (and more easily on target), and end managing protobuf as git subtree as we do now. We created the basic capability to use Machinekit remotely - we now need to wrap it such that this can be used by mere mortals.

Random bits and loose ends

• We will be maintaining a build server, CI builds (likely separate from package builds), and package repos for all currently supported architectures. 
• I was surprised in the very strong interest in remote both remote operations, and actual use of the new remote UI's. Some of this comes from the new capabilities,  and some from the relative freedom of licensing when creating a remote Machinekit UI.
• The installation counts are significant and much higher than I thought - around 1400 distinct SD image downloads, and several hundred package users.
• the resulting HAL/realtime package will have external package dependencies which are a fraction of the current list - making it much easier to reuse in other projects. 
• The current HAL revision under way suggests using a vehicle for inline API documentation (possibly doxygen or asciidoc-based as the zeroMQ project uses it)

Plans for a Machinekit Foundation

An open source project on github only goes so far. At times it makes sense to have a bit more formal structure, and a legal entity at hand - for improved professional standing, but also as an endpoint for corporate or consortia memberships like EtherCAT or i-ROS. There is no intent to create any superstructure over the Machinekit project per se - rather a utility to be used when appropriate.

One of the obvious purposes of such an entity could be to hold the domain name ownerships of the project (I currently hold these at my private expense), which will remove another dependency on a single person. I'd be happy to transfer ownership eventually. Same thing could go say for hosting contracts - non-profits also get usually better rates than individuals.

The role model we considered is the Open Source Robotics Foundation,  a not-for-profit organization under American law. It turned out that setting up such an entity can be done with moderate financial and recurring paperwork exposure. The majority felt this would be a useful step to take, and we will strive to have this entity setup before our next meetup.

And the winner is: RT-PREEMPT

Many might remember that one of the key contributions of the unified build branch of LinuxCNC - which eventually turned into Machinekit - was providing support to multiple realtime kernels. At the time, LinuxCNC only could make use of RTAI, distributing a well-aged version thereof.

RTAI still yields the best latency figures. But that comes at a huge cost: having application code run in-kernel is not only unsafe at any speed, but fraught with an enormous build complexity, kernel version dependency and recurring maintenance chores. And that has not changed - as well as the restriction that RTAI runs on Intel architectures only. Moreover, the future of the RTAI project has become clouded as it has always been a bit of a one-man show lacking a healthy community around it to take over just in case.

The alternatives supported by Machinekit are Xenomai and RT-PREEMPT.

Xenomai shares some history with RTAI - both are hypervisor kernels: the idea is to have a minimal, RT-capable scheduler and interrupt handler underneath the actual Linux kernel. RT threads use this hypervisor to achieve better timing than possible with a vanilla Linux kernel. Other than RTAI - where RT applications need to run in-kernel as modules similar to device drivers, Xenomai supports a threading model which almost looks like normal Posix threads - except that only rather restricted use of the Linux API can be made from such a thread. Xenomai does support a wide range of architectures, which is why it is the mainstay of running Machinekit on ARM platforms.  Again this comes at a cost: Xenomai still requires a rather intrusive kernel patch and is available for a limited range of underlying Linux kernel versions. And given the fact that many embedded manufacturers choose to use a rather specific, sometimes outdated kernel version and sometimes do not upstream their patches into the Torvalds mainline Linux kernel, the chances for getting a working Xenomai kernel for such platforms is pretty low.  That is the main reason for both RTAI and Xenomai kernels being "well aged" on most platforms.

The third alternative is RT-PREEMPT - a set of patches to the standard Linux kernel to improve it's timing behavior, but without introducing a separate API, or a hypervisor. This project  has been over a decade in the making, and at times there were doubts if it would make it into Linux mainline kernel. Initially being substantially higher latency than the hypervisor breed,  over the last year or so huge progress has been made in terms of performance delivered on Intel platforms in particular, but also on ARM platforms: I recently tried an RT-PREEMPT kernel on a Raspberry-2 and it delivers slightly better latency than Xenomai on the Beaglebone, so it's getting pretty close.

In the past, the argument for minimum latency has been its usefulness for software-based step generation and quadrature encoders - the actual servo cycle computations do not need that low latency. But fact is - the PC's parallel port is an extinct piece of hardware (and even then RT-PREEMPT can deliver reasonable software step rates). Plus, inexpensive FPGA hardware can deliver higher performance if needed.

Picking one among the above choices needs to made by all users of realtime applications, not just us. Some funding for RT work has come from the financial industry for high-frequency trading in the past, and the automotive industry has a healthy interest as well, demonstrated for instance by the participation in the Linux Realtime Workshop conference series. This revolves mostly around the autonomous driving theme.

There was some real good news recently: the Linux Foundation announced that it will adopt the RT-PREEMPT project with the goal of bringing it into the mainline kernel (note list of sponsors!). This assures the funding of the remaining work, and IMO it is reasonable to expect that in a few years - probably more than one, but certainly less than five - obtaining a RT-PREEMPT kernel will be just a build option of the mainline kernel. Already now building kernels is much, much simpler than any of the other options - and once that effort goes mainline and manufacturers actually support that kernel, it will be much simpler to obtain RT kernels for any platform, not just a few select ones.

Is interesting to note that the Xenomai3 effort provides a common API over both the hypervisor-style and RT-PREEMPT kernels. Since Xenomai seems to enjoy a healthy industrial user base, it is important to offer a migration path to its users towards what - probably not only I - consider the winner of the RT kernels competition.

These developments have some far-reaching implications for the Machinekit project, some of which were discussed at the recent meetup.  Among those are:

• the performance edge provided by RTAI has become so small that it is by far outweighed by the enormous build complexity it entails for Machinekit. We have therefore decided to end support for RTAI.
• With RT-PREEMPT very likely to go mainline, it is likely efforts are made by hardware vendors to support this rather than other options, meaning both range of supported hardware will widen, as well as functionality and performance improvements to show up here first. It will take a while until this "sinks in" with all involved, but the direction is clear.
• For the time being we'll retain the Xenomai2 builds; while Xenomai2 is in maintenance mode already, we do have stable kernels around which perform great. And the build process is easy and robust. But there is not much point in a Xenomai3 port to just run RT-PREEMPT underneath - the rt-preempt flavor already does that. Any Xenomai3 hypervisor flavor needs to be weighed against the performance edge it has over RT-PREEMPT, and it looks like this edge is shrinking.
• Ending support for kernel-threads enables removing the absurdly complex legacy build system by more mainstream tools like cmake.
• using userland threads exclusively opens new options: so far HAL realtime code was restricted to C, as C++ is not supported for kernel modules. This has both potential for simplifying HAL itself, as well as making it easier to integrate with C++-based systems like ROS and Orocos, or bringing in improved solutions to old problems.

Machinekit meetup - summary of talks

Here's a list of all talks held at the meetup, including links to recorded video and slides:

• Bob van der Linden, node.js-machinetalk, an upcoming node.js - based remote API into the machinekit CNC and HAL stacks (slides)

• Alexander Rössler, python-machinetalk, on a Python-based remote API into the machinekit CNC and HAL stacks (video slides)

•  Charles Steinkuehler, Platform Options for Machinekit: Current and Future, review and outlook on "what is cooking in the hardware space", (video)

• Charles Steinkuehler, The need and use of multicore HAL (video slides)

• Michael Haberler & Mick Grant, HAL “multiKore”: making HAL safe & fast for multicore platforms - about enabling safe use of multiple core processors for realtime tasks (video slides)

• Bas de Brujin, Update on CANopen driver for HAL (video slides)

• Jon Elson, PicoSystems, Comments on the success of making CRAMPS boards and retrofit of 1996 photoplotter with a Beagle Bone (video)

• Alexander Rössler, Building Qt5 UI's with the MachinekitSDK, covering creating user interfaces in the MachineSDK Vagrant VM and future possibilities of the MachinekitSDK (video slides)

ANNOUNCE: Machinekit Meetup

Nov 6-8/2015, Madison WI

We are please to invite anyone interested to the second "real world" Machinekit meeting, taking place on November 6/8 at the Tormach facility in Madison, Wisconsin (USA).  Please feel free to join us - share your plans, exchange experience, and help shape the future of Machinekit!

Rough agenda so far:

• Charles will talk on "Platform Options for Machinekit: Current and Future", covering the upcoming newer ARM & ARM+FPGA options besides the existing armhf/x86/amd64 options
• Alex will talk on "Building Qt5 UI's with the MachinekitSDK"
• I will present on HALmk ("HAL-MultiKore") - about enabling safe use of multiple core processors for realtime tasks, and the future of kernels for machinekit

Other topics we could think of

• firming up plans and procedures for a machinekit stable branch
• status reports from the nodejs-machinetalk and python-machinetalk projects (remote API's)

.. and anything you would want to bring up/contribute to!

More venue and agenda details will be posted shortly, so watch for details, and speak up if you're planning to attend. So far most folks are arriving sometime Friday and leaving sometime Sunday.

Charles, John, Alex, Michael, Bas

Denford Novamill Conversion

Designspark user stuartChilds has a multi-part blog entry documenting the conversion of a Denford Novamill with a non-working controller to using a BeagleBone Black running Machinekit with a Probotix cape and Gecko stepper drivers.  Part 1 covers the mechanical and wiring changes, while part 2 details software integration and configuration.  Upcoming posts will cover the spindle drive, CAD/CAM, and "making chips".

Machinekit To Aid With Heart Research

Recently I was contacted by a Professor Andreas Lueger of Medical University of Graz, Austria, Dept. of Internal Medicine, probing me with some rather intricate questions on Machinekit. Andreas is head of the emergency room there, helipad on the roof and all. So if you drive recklessly around here, you stand a chance for a helicopter ride and being stitched up by Andreas and his team, but that is certainly not “motion control” as we know it. That made me curious, and I visited Andreas to probe what he is up to:

Andreas in business attire, as found in his professional habitat

Beyond stitching up patients, Andreas is deeply involved in heart research - investigating atrial fibrillation, a precursor of heart attacks. And that is some very geeky project  - for his research, he develops the electronics based on a TI MSP430, as well as the software for his own Wifi-enabled pacemakers - a whole new meaning of “embedded” for me. Andreas brought down current consumption to a couple of hundred nanoampéres, and does solder the parts under his own stereo microscope:

What can I say? at last an MD who talks our language! And when I visited him, he had a bug report for Machinekit, including a screenshot from his oscilloscope. I was impressed.

Now that pacemaker certainly is not running Machinekit, so where do we come in? Well, in the next iteration of the project, Andreas plans to integrate acceleration sensors in the setup, to measure actual mechanical performance of heart muscles. And for manufacturing these sensor attachments Andreas is designing a custom CNC machine with FreeCAD:

Now we’re talking.. that machine will run Machinekit! Here comes the twist: Andreas plans to use Trinamic servo stepper motors. Those are very smart motors with an embedded controller and power stages attached; among other features, these motors can detect and report a stall, and autonomously compensate for lost steps:

Trinamic motors can be controlled through various methods: Step/Direction pins, RS232, RS485, USB, and CANbus. Now out of these options, CANbus is the most interesting one because it operates on a much higher level than the other options; the motor can report back (heartbeat, stall, position, velocity etc).  It’s really a much more intelligent motor subsystem than we usually use in Machinekit, like servos or stepgens.

I found all this terribly geeky, and beyond that also valuable for the project overall: if we investigate and collectively learn how to integrate CANbus, we open up Machinekit to literally thousands of peripherals with a professional bus system - as opposed to homegrown methods like hooking up an Arduino over RS232 or USB. The Beaglebone's TI3359 chip comes with two CAN devices on board - add MCP2562 bus drivers and you are done; or use an appropriate cape. For other platforms, PCI cards and USB adapters do the trick.

CANbus peripherals come in many shapes and forms - like this controller used by Claudio Lorini with support for the zedboard which he recently merged into machinekit; but also more mundane devices like your car's windshield wipers and power windows: yes - your car is a one extensive CANbus installation (provided your car was built in the current millennium). And if you research that topic, you will find there’s a whole “car CANbus” hacking scene out there, using Arduinos and whatnot to tap into their car’s CANbus, like this example. 

Where does this leave us? I have decided to support Andreas with this project, and while it is not absolutely certain the Trinamics will work as motion units as planned, it will be a very interesting expedition. And along the journey, Machinekit will learn to support CANbus peripherals.

And that exciting story is the background why you saw references to CANbus in my recent postings to the Machinekit list!

Machinekit Running on Odroid C1

Machinekit already runs on x86 machines, the BeagleBone, and the Raspberry Pi.  Now GP Orcullo reports he has Machinekit running on the Odroid C1 using rt-preempt.  There are no hardware drivers yet, but it shouldn't take long to port GPIO access and get some motors moving.  If you're interested, there's a link to a bootable Debian Wheezy image in the Machinekit list post.

I'm anxiously waiting for boards based around all the new Big/Little parts coming out, which combine Cortex-A cores for good OS performance and Cortex-M parts for real-time tasks.  Then there's the BeagleBoard X15 coming out soon with Big/Little + DSPs + PRUs!  Exciting times in the ARM world!

IoT Vienna Presentation

Alexander Rössler recently gave a presentation on Machinekit and Machinetalk at the IoT Vienna Talks March 2015.  If you're interested in the new remote user interfaces and distributed HAL capabilities, take a look!

2015 Midwest RepRap Festival Coming Up

The Machinekit vinyl stickers for the 2015 Midwest RepRep Festival hosted by SeeMeCNC arrived recently.  I'll be handing out stickers and showing off various 3D printers running Machinekit soon in Goshen, IN.

Jason Kridner (of BeagleBoard.org) will also be there showing off his BeagleStock1, a SeeMeCNC Rostock Max v2 driven by ‪Machinekit running on a BeagleBone Black.

There will even be some complete BeagleBone and Machinekit powered electronics kits (BBB + CRAMPS + DRV8832) for your 3D printer or mini-mill being given away as door prizes!


If you're anywhere near the Midwest (or have some spare frequent flyer miles) and are interested in 3D printing, the MRRF is something you won't want to miss! 
 

Loadable Trajectory Modules

Michael Haberler has recently succeeded in separating the trajectory planner from the motion controller and made it a new HAL vtable module.  While the details quickly get into technical programming issues, this seemingly esoteric feature actually means big changes are possible with higher level configurations.

The new vtable modules allows HAL components, kinematics modules, and now the trajectory planner to be loaded and instantiated dynamically.  That means you no longer need to know ahead of time how many of a particular component your HAL file needs, and you can do things like load more than one kinematics module or trajectory planner.

What good is that, you ask?  Well, the ability to dynamically load modules means you can craft modular HAL files without jumping through hoops to add up all the modules before you load anything.  One HAL file might instantiate three PID loops for XYZ motion, and another included HAL file might add two more PIDs for temperature control.  Even better, since you can use Python to work with HAL, you could have a Python program setup a complete HAL configuration based on a few parameters with a library of subroutines to craft basic modules like a servo driven axis, a stepper driven axis, a temperature control loop, etc.

As for being able to dynamically load things like the trajectory planner and kinematics modules, that means you can now essentially have more than one "machine" running in the same HAL space.  This provides an advanced way to handle things like tool changers and is helpful anywhere you need more than one control system.  Perhaps you have a pick-and-place machine that includes a separate mechanism to place finished PCBs on a conveyer and load a new blank PCB.  Maybe your robot arm also has a gimbal mounted camera you want to control independently.  There are lots of times having more than one trajectory planner and kinematics module available is very helpful once you go beyond the world of traditional CNC machine tool applications.

New BeagleBone Images Avaialble

Thanks to the tireless efforts of Robert C. Nelson, Machinekit uSD images for the BeagleBone are now part of the regular Debian image build cycle.  The first image is available now, with updates expected about every two weeks.

I will update my Machinekit page as soon as I get a chance, but for now just follow along with any of the BeagleBone getting started guides to put the xz compressed image onto a uSD card.  Once you're running the image, you can easily perform updates using the standard Debian package management tools.

Many thanks to Robert for building the images (and the required Xenomai kernel), everyone working on the Machinekit project for the packages, and to all of you who use these tools to drive your machines.

3D Printing With Machinekit

While it is possible to 3D print with a "traditional" CNC controller like Mach3, LinuxCNC, or Machinekit, there are a lot of small problems that get in the way.  There is a lot of ongoing work in the Machinekit project to make this process much easier.

Alexander Rössler has been working on gcode remapping and a standard set of signal names that will reduce the amount of gcode post-processing required and improve print quality (migrating "queue-busting" custom M1xx codes for setting things like heater temperatures and fan speed to inline analog output values that keep the hot-end moving).

Bas de Bruijn has been reworking the documentation files and build scripts, allowing documentation to exist in a separate (and easier for non-programmers to modify) repository.  Very soon you should see online documentation built automatically just like the buildbot packages for the executable files.

Jean-Paul Moniz has been working on getting some of the fancier bed-probing and auto-calibration features common in Arduino firmware working on Machinekit.  I (Charles Steinkuehler) helped this effort slightly by exporting some homing settings to HAL that previously required a shutdown and restart of the linuxcnc application to change (traditional machine tools don't need to change their homing details very often).

In addition, work is continuing with remote interfaces, velocity based extrusion, simplifying the example HAL files, and other changes to help make Machinekit more accessible for new users looking for a solution with more "upside" than an AVR or Cortex M4.  Machinekit scales from systems like the BeagleBone and Raspberry-Pi to multi-core x86 systems with custom FPGA hardware interfaces.

I'm sure I missed several folks who've reported progress on the mailing list, but it's not intentional.  There's just so much happening with Machinekit already this year it's hard to keep track of it all!

Remember, the best way to keep up to date is to monitor the Machinekit Google group.  Drop in and say hi!