Real-Time Programming with Xenomai 3 - Part 1: Installation and Basic Setup

In my lab, we recently started moving away from Simulink’s Real-Time packages and towards Real-Time Linux for implementing the low level control of our robots. I thought I would document what I went through to get Xenomai (A Real-Time framework for linux) working stably as a resource for others trying to get started on the same thing.

What is Real-Time?

The word “real-time” is used in a lot of different fields to mean different things. Some people also mistake real-time systems for high performance systems. I use the word real-time to refer to systems that guarantee consistent responses to events within time constraints (also called deadlines) with low variability regardless of system load. This is something that’s not easy to do. A normal linux operating system is not real-time.

Say you want to implement a program in C on a regular linux computer that toggles a pin on the parallel port every millisecond. A naive implementation would be as a simple loop that toggles the pin and waits for 1 millisecond. This may work OK most of the time but the moment the system is loaded by something else, the program may start running its loops slower because the process may be preempted by the kernel or be swapped out of memory in favor of another memory intensive program.

A real-time operating system will be designed so that regardless of the system load, the program that toggles the pin can do the operation every millisecond give or take a few microseconds. This is the reason that a high performance computer with the latest Core i7 processor running a non-realtime OS can be less “real-time” than a low end microcontroller running a single optimized control task. Real-time systems are used in mission critical control systems such as those on fly-by-wire aircraft, satellites, exoplanetary rovers, cardiac pacemakers or car engine control units.

A real-time operating system (RTOS) usually has an API for creating and running real-time tasks and uses a scheduling algorithm that’s different from what’s used by general purpose operating systems like Windows and Linux. There are many RTOSs in the wild. Wikipedia has a great list of them. Some RTOS’s like FreeRTOS are meant to be used in embedded microcontrollers. Some - like RTLinux are used when the real-time application needs to be run on a full blown operating system. Using real-time with full operating systems also allow you to take advantage of a lot of existing software and functionality that the operating system will have like networking and math libraries like GSL.

Real-Time Operating Systems for Control

When implementing a control system such as a PID controller digitally, real-time response times become very important. One of the assumptions made when developing digital control systems is that of constant sampling time. When you implement a control loop in code, if the constant sampling rate assumption is not met or if the controller responds too late to changes in the system state, it could lead to the system becoming unstable.

The choice of using Xenomai to implement our real time controllers was mostly due to the fact that it has very good documentation apart from being free and open source.

Installing Xenomai

To get Xenomai running on a linux system, you need to compile a modified kernel. I used Lubuntu 12.04 (which ships with kernel 3.2). The version of the kernel that I compiled is 3.18.20. Choose a version of the kernel that is close in version number to the one that the distribution ships with to minimize issues.

Before starting, create a fresh folder to act as your workspace. Also make sure you have plenty of disk space available. The 3.18.20 kernel requires just over 11 GB of free disk space to compile successfully. Newer versions of the kernel need more. 20 GB should be safe.

These are the steps that I’ve been following to get a freshly installed Lubuntu system working with xenomai. Keep in mind that if this is the first time you’re compiling a kernel, things are bound to go wrong. Be willing to debug your compilation patiently. I had to go through the compilation process dozens of times, making tweaks at each step to get my first successful kernel image.

1. Go to the downloads section of the xenomai website and look for an ipipe patch (it’s a file with a .patch extension) that is for a kernel version that’s close to the version your distribution ships with. This is the kernel version that you’ll be compiling.
2. Go to kernel.org and download the version of the linux kernel that exactly matches the kernel version on the ipipe patch file name.
3. Download the xenomai source from the xenomai website.
4. Unzip the files into separate folders.
5. Apply the xenomai patch to the kernel: xenomai-3.0.4/scripts/prepare-kernel.sh --linux=linux-<version> --ipipe=patch-<patch-version>.patch --arch=x86_64
6. cd into the kernel source directory and run make menuconfig and make the following changes:
   • Power Management and ACPI Options -> CPU Frequency Scaling - DISABLE
   • Power Management and ACPI Options -> ACPI Support -> Processor - DISABLE
   • Power Management and ACPI Options -> CPU Idle -> CPU Idle PM Support - DISABLE
   • Device Drivers -> Input Device Support -> Generic Input Layer -> Miscellaneous Devices -> PC Speaker Support - DISABLE
   • Processor type and features -> Processor family: Select the exact processor on the motherboard. This is important for things to work properly.
   • Xenomai/cobalt -> Core Features -> Shared Interrupts - ENABLE (If you want shared interrupts to work)
   • In the Xenomai/Cobalt drivers section enable all the drivers for the devices that you’ll be using. Consider compiling the drivers as modules (using the m key) so that it is easier to load and unload and debug using modprobe/insmod/rmmod
7. Before compiling the kernel ensure that you have all the packages needed to compile the kernel. Use apt-get or any package manager.
   • gcc
   • make
   • Autoconf
   • libtool
   • kernel-package
   • build-essential
   • fakeroot
   • dh-autoconf
8. cd into the kernel source folder again and compile the kernel using the command: sudo CONCURRENCY_LEVEL=8 CLEAN_SOURCE=no fakeroot make-kpkg --initrd --append-to-version -xenomai-realtime --revision 1.0 kernel_image kernel_headers
9. Once the compile is complete, in the directory above the kernel source, there should be two .deb files: One starting with linux-headers and one starting with linux-image. These are the kernel image and header debian packages that you can use to install the kernel.
10. Use sudo dpkg -i <package-name> to install each of the packages
11. Update the initramfs using the command sudo update-initramfs -c -k <kernel-version>-xenomai-realtime && sudo update-grub
12. cd into the unzipped xenomai source folder and run sudo ./configure followed by sudo make && make install
13. Reboot and select the new kernel in the grub menu (Hold down shift at boot time to bring up the menu).

If everything went well, xenomai should be installed on your system now. You might be tempted to start running some tests immediately but I think it’s worth taking some more time to set up your build environment properly to make development easier.

Some important xenomai related executables (including xeno-config) are in the /usr/xenomai/bin folder. You need to append this to the PATH environment variable. The library files that the code is linked against are installed in the /usr/xenomai/lib folder. You need to append this to the LD_LIBRARY_PATH environment variable. My preferred way of doing this is to edit the ~/.bashrc file with export statements. Open up your .bashrc file (it’s in your home folder, hidden) by running leafpad ~/.bashrc and add the following two lines to it at the end.

export LD_LIBRARY_PATH=”/usr/xenomai/lib:$LD_LIBRARY_PATH”```

When you try to run a compiled xenomai application afterwards, you will need to run it as superuser (using the sudo command). It might complain that it cannot find the library files. This is because environment variables are cleared when running a program as root. To fix this permanently:

 1. Open up a terminal and go to the `/etc/ld.so.conf.d` folder.
 2. Create a new file called xenomai.conf (as superuser)
 3. Add the line `/usr/xenomai/lib` to the file.
 4. Run `sudo ldconfig`
 
_Now_ you can finally try to run some tests. If everything went well, running `sudo /usr/xenomai/bin/latency` should run the latency test program that outputs some numbers on to the terminal that shows you the latency figures of your system. It should be in the tens of microseconds range (If not something's wrong).

The process of getting real-time linux with Xenomai or any other framework running for the first time can be a little painful but once your installation is stable, it'll serve you well for a very long time. Once you have a working compiled kernel package, installation on future hardware will also go much faster. 

In Part 2 of this series of blog posts, I will go through the process of writing a simple periodic real-time task using Xenomai's real-time API.