Debian for Embedded Systems - Seriously?

On a regular basis I come across “white papers” and presentations that compare the usage of Debian to solution XY for embedded use cases. There I read claims such as “Debian is only suitable for a quick prototype”, “Debian systems are hard to reproduce”, “you can’t conveniently cross compile for Debian systems”, “long term support is a nightmare” and “you can’t cross bootstrap images for a foreign platform”.

Let’s debunk those myths for good by giving plenty of evidence that the things have dramatically changed over the last few years.

For a proper technical analysis we tear apart the components of a typical embedded system:

• Hardware: Decent hardware with good mainline and vendor support is key for a successful embedded system.
• Bootloader: The first software that runs on the embedded system is the bootloader.
• Kernel: The bootloader will load the kernel and hand over the control to it during the startup.
• Root file system: The kernel will mount the root file system that contains the applications you want to run on the embedded system.
• Tools: Typically embedded systems are tailored for their exact use case and therefore you need powerful tools for the tailoring and the application development.
• Support: Nowadays embedded systems are complex, and you will not be able to take over the maintenance burden for the entire system. Therefore, you need an ecosystem that helps you to support your system.

Let’s go into more details:

Hardware

The embedded world got literally flooded by boards that are capable of running Debian and even come with proper Debian support. The most prominent representative is the Raspberry Pi (see also pi-gen tool and rpi23-gen-image tool) seconded by devices supported by Armbian (see also Armbian github repository). Of course, you won’t be able to flood the market with a clunky setup and therefore the above solutions are easy to use and perfectly suitable not only for students that want to learn more about embedded systems. In many regards established board vendors can learn a lot from the above communities: ease of use, longevity, documentation, upstreaming, …

Ok, now you could say that the above boards are for tinkerer. Depending on your use case this might be true and therefore I will provide a list of industrial grade devices and boards/SOMs running Debian:

• Moxa IIoT gateways: The UC-8200 Series are running Debian based Moxa Industrial Linux.
• Compulab IIoT gateways: Compulab sells IIoT gateways with Debian preloaded.
• Variscite SOMs: Variscite sells SOMs with comprehensive Debian support.
• Toradex SOMs: The Toradex board support packages are built using Yocto, but they decided to support Debian containers through their Torizon platform.
• Siemens SIMATIC IOT2050: Also the SIMATIC IOT2050 comes with Debian preloaded.

The above vendors are all renowned, and they share the strategy to deliver proper Debian support with their hardware. I am sure there are plenty of additional good examples.

Verdict: A lot of great, industrial embedded hardware comes with out of the box Debian support.

Bootloader and Kernel

No matter which flavor of Linux you bring to your embedded board the situation for the bootloader, and the kernel is the same: You would like to use the upstream source code (U-Boot, Linux Kernel) but at some point you detect that the latest features of your CPU are not yet upstream and therefore you need the changes done by the silicon vendor (example NXP: U-Boot, Linux Kernel) that get further refined and integrated by the SOM/device vendor (example Variscite: U-Boot, Linux Kernel). Obviously you need to cross compile those components for your target system and for Debian I recommend packaging the Kernel as a Debian package. Other than this the kernel does usually not differ between a Debian based system, and a system that got built using any other solution.

Verdict: The bootloader and the kernel does not make the real difference between Debian and any other solution.

Root File System

A Debian root file system is built from Debian packages. The majority of the packages that go into an embedded Debian system can directly be pulled as binary packages from an official Debian repository. Did you know? The official Debian repositories do also host the corresponding source code. If needed, an official Debian package can also be tweaked and then stored on a private Debian repository. Furthermore, you will write your own applications, and I strongly recommend that you also make them available as Debian packages using your private Debian repository.

After all you will end up with a setup that might look like this:

Both, the developer and the build server will be able to build target system images without having the need to access a target device. The process to build the root file system for the target device is finally highly reproducible, does fit perfectly into a modern CI/CD pipeline and does easily scale up to big teams.

Verdict: The root file system of an embedded Debian system can be tweaked where needed, and the usage of prebuilt binary packages does speed up the workflow. As you do not only have the Debian packages available for the architecture of your target system (e.g. arm64) but also for the architecture of your development machine (e.g. amd64) you can easily build a digital twin of your target system and run it natively as a LXD container on your host system. This has proven to be extremely convenient for developers.

Tools

One hidden gem for embedded device development is the cross compiler. Some twenty years ago it felt like black magic to get such a cross compiler compiled. Nowadays installing a cross compiler on Debian is as easy as switching on your computer: sudo apt install crossbuild-essential-arm64. Done!

The story does not end here: To compile an application for your target system you also need the corresponding development libraries and this is where Debian has another shiny feature: multiarch. Multiarch lets you install “foreign” libraries (e.g. arm64) alongside the libraries of your host system (e.g. amd64). Now you are ready to cross compile complex applications.

I strongly recommend the usage of containers so that your development environment does correspond to your target system:

So we have tackled the cross compilation - now let’s move on to the image build: The only thing that the Debian embedded community could be blamed for is that there are (too) many options:

• Elbe: Debian based E.mbedded L.inux B.uild E.nvironment.
• Apertis: Collaborative OS platform for products.
• ISAR: Integration System for Automated Root filesystem generation. Isar is also used by Siemens Omni OS.
• edi: Embedded Development Infrastructure. My favourite because of Ansible and the digital twin approach - but honestly, I am biased.

As described above, your Debian setup will require a Debian repository. Here are my recommendations:

• packagecloud: A reliable hosted solution for Debian package management.
• Aptly: The swiss army knife for self hosted Debian package management.
• Artifactory: Artifactory can be hosted on premise, or you can opt for a cloud instance subscription.

Finally, here are some other tools and services that I would like to mention:

• snapshot.debian.org: “The snapshot archive is a wayback machine that allows access to old packages based on dates and version numbers.”
• Mender: Secure and reliable over-the-air updates for embedded systems. Mender fits perfectly into the Debian ecosystem.
• Reproducible Builds: As an embedded developer you aim for a system that is highly reproducible.

Verdict: The tooling for embedded Debian is great and continues to become even better. The only thing you might struggle with is the wide choice - but as a Linux developer you are used to it - aren’t you?

Support

If you once built embedded systems for highly regulated medical devices then you know the painful paperwork process when doing a major version upgrade. You absolutely want to avoid this. At the same time you also need to fulfill the cybersecurity requirements and therefore you need targeted security updates.

This turns out to be a unique selling point of Debian - you get long term support:

If this long term support is beneficial for your project too, then you should join the Debian Long Term Support initiative and also the Civil Infrastructure Platform project might be of interest.

If you need personalized help, I am sure you will get good support from companies like Linutronix, Ilbers or Siemens (formerly Mentor Graphics).

Verdict: Debian has a support ecosystem that has attracted many big players over the past few years. The longevity has proven to be a key selling point.

Conclusion

All the above examples clearly show that you can go far beyond the “fast prototype” using Debian. Alone edi is helping to build images for an amazing number of embedded devices that are used all around the world. Given that you have a clean setup your Debian image builds are reproducible. The cross compilation couldn’t be much smoother, and the long term support has become a key selling point. The resulting images might be a bit bigger than when built with other approaches. In any case it is also possible to generate them for a foreign platform - this is what I do on a daily basis.

I am very happy how Debian has evolved over the past few years: Debian for embedded systems - seriously!