Installing Devuan GNU+Linux or Debian GNU/Linux on ARM64-based SbCs that are not officially supported


If you want to self-host stuff at home using an ARM64-based board running Linux, both Debian GNU/Linux and Devuan GNU+Linux are excellent distribution choices. Unfortunately, they officially support only very few ARM64-based boards. Yet, you can run them unmodified on a lot more boards - if you manage to install them. After discussing some basics and showing how to check the support status for a particular board, this article describes my recommended way for performing the actual installation by a concrete example.

Why use official distributions?

Before we dwelve into details, let's discuss some basic things first. I often hear the argument that people want to solve basic stuff like installing the OS as quickly as possible and focus on the more interesting stuff in the layers above. Usually, these people end up downloading and flashing one of the countless unofficial OS images from the Internet. What's wrong with that?

Choosing a good base OS for your setup is the foundation for running a trustworthy and secure system. Sure, this might be irrelevant for a quick and dirty test setup. However, if you want to run your system for years and put personal data on it, this aspect is critical.

Personally, I discourage using unofficial images for a number of reasons:

  • Many unofficial images bundle additional software that is not packaged in the official repositores. Therefore, you never get updates for them. This is bad, especially for critical components such as the kernel.
  • In addition to trusting the upstream project (such as Debian), you also have to trust the projects and individuals that built the images.
  • Apart from trusting these third parties, you also have to trust their build infrastructure.
  • Setting up the system yourself, you can optimize the setup to your needs from the very beginning.

On the other side, running an official distribution, you can be sure to get the best “vanilla” experience. In addition, all the base software for your system comes from official sources and was build on their official infrastructure.

Unofficially supported boards

The Linux kernel supports a variety of ARM64-based boards. Unlike x86 machines, ARM64-based boards require a board-specific device tree binary (dtb) that is built from source. As Debian aims to be an universal OS, it ships a generic ARM64 kernel that enables support for as many of these boards as possible by shipping dozens of dtbs. This is also true for Devuan since it uses Debian's unmodified kernel.

The current OS versions (Debian 10 and Devuan 3) ship version 4.19 of the Linux kernel. To look them up the boards they theoretically support, do the following:

  • visit to packages.debian.org.
  • use “search package directories” with “only show exact matches” and search for the package linux-image-arm64 in the stable distribution.
  • hit the search button and click on the result.
  • since linux-image-arm64 is a metapackage that depends on the current kernel, check which version it is (as of now, it is linux-image-4.19.0-9-arm64). Click on this package to show more details.
  • click on “list of files” to see all files contained in this actual kernel package and check the dtb files listed there.

If your board is listed here, chances are good you will be able to run the official distribution of Debian or Devuan on it.

If your board is not listed, don't give up yet - it might still be supported by a newer Linux kernel. Later in this tutorial, I will show how you can install a newer kernel in the current stable release and get regular updates for it. However, this only works if your board is supported at least in the “unstable” release.

Don't worry too much about the name “unstable” - the kernels in Debian unstable are based on well-tested kernels from upstream and usually work well. They are updated frequently and, thus, receive security updates. While I consider them okay as a daily driver, they are not as stable as the kernels from the stable distribution.

To check if your board has support in the unstable kernel, repeat the previous steps but choose “unstable” instead of “stable” for the distribution. If your board is still not listed, then you are out of luck for now as your board is currently not supported (but it might get official support in the future).

Installation

You have a board that is at least unofficially supported by the unstable kernel (see previous section)? Great - let's get our hands dirty!

In the following, I will assume you want to install Devuan 3 on a RockPro64 by Pine64 [1]. You can also use this guide with small variations if you want to install Debian instead of Devuan or use a different hardware.

Concept

My installation concept covers the following parts:

  • Installing a device-independent base system using the official installer
  • Customizing the base system
  • creating tar archives for boot and root
  • building the bootloader (device-specific)
  • assembling a device-specific image
  • flashing the assembled image

This might seem a lot of work, but note that the first three parts are device-independent. Thus, won't have to redo them when installing other devices in the future.

Note: This concept borrows and enhances some ideas from my previous debootstrap-based approach.

Requirements and assumptions

Apart from having an unofficially supported board, I assume you have basic knowledge about the following:

  • creating and operating virtual machines with KVM/QEMU using virt-manager
  • operating your ARM64 board using the serial console

In addition, you will need:

  • a dedicated physical x86 machine with a fresh installation of Debian GNU/Linux 10 or Devuan GNU+Linux 3 with at least 2 GiB of RAM and 20 GiB of storage
  • a user account with a user named youruser
  • an empty microSD card with a capacity at least 2 GiB
  • a working internet connection

Of course, the tutorial should also work with other systems, distributions and environments but some steps may differ. Also, note that using a VM is only possible if you enable nesting as we will exercise the ability to run ARM64-VMs ourselves. Using very old hardware (e. g. a single-core 32bit machine) is theoretically possible but practically infeasible

Disclaimer

All data and information provided in this tutorial is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this tutorial and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event, the author will be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this tutorial.

Step 1: Installing the base system

First of all, let's create a directory where we will store all the assets produced in this tutorial:

mkdir -p /home/youruser/assets

Then, download the mini ISO for arm64 from the following location:

https://pkgmaster.devuan.org/devuan/dists/beowulf/main/installer-arm64/current/images/netboot/mini.iso

If you want to have Debian instead of Devuan, use the following:

http://deb.debian.org/debian/dists/stable/main/installer-arm64/current/images/netboot/mini.iso

Make sure you also have qemu and virt-manager installed:

apt install qemu-efi-aarch64 qemu-system-arm virt-manager

Next, create a new virtual machine using the aarch64 architecture as shown here:

screen010

In the next step, choose the mini.iso you downloaded previously as ISO and select “Debian 10 (stretch)” as OS. Give the VM the default of 1 GiB of RAM and as many cores as you have. Next, create a disk image with at least 5 GiB (or stick with the default) and leave the default network settings (using NAT). Finally, fire up your new VM. You should get a screen like mine:

screen020

After selecting the first option, the “oldschool” text installer should boot and give you the following screen:

screen025

Now run through the installer but follow these guidelines:

  • choose a generic hostname and domain name (we want to build a generic, reuseable image)
  • choose simple passwords for root and the user you create (again, we want to build a generic image)
  • use deb.devuan.org as pkg mirror
  • when partitioning the disks select “Guided - use entire disk” and “All files in one partition” and accept the computed layout. Don't worry about the actual size of the filesystems - we will pull all files from the installation and repartition it later anyways.
  • when installing packages, unselect all packages for installation in tasksel (our aim is to get a minimal image, you can install packages later if you need them).

After installing, the system should boot up as shown here:

header

Step 2: Customizing the base system

f2fs

While the installer does not support installing to f2fs filesystems, f2fs is the filesystem I recommend for flash media. Install this package to get support for it:

apt install f2fs-tools

Add the module to /etc/initramfs-tools/modules:

echo "f2fs" >> /etc/initramfs-tools/modules
echo "crc32" >> /etc/initramfs-tools/modules

And regenerate the initial ramdisk:

update-initramfs -u

Disk encryption

Our setup assumes that you want to run boot and root unencrypted but have the ability to encrypt other filesystems (e. g. /home). If you want, you could also encrypt the rootfs and decrypt it via dropbear later (this is not covered in this tutorial). To enable crypto support, install cryptsetup utilities:

apt install cryptsetup

DTB file handling

Since we want to use a separate /boot partition later, we need to make sure the dtb files for the current kernel are there. Therefore, we create a dtb directory there first:

mkdir -p /boot/dtbs

And add the following script that copies the dtb files automatically each time a new kernel is installed or upgraded:

cat <<'EOF' >> /etc/kernel/postinst.d/copy-dtbs
#!/bin/sh

set -e
version="$1"

echo Copying current dtb files to /boot/dtbs....
cp -a /usr/lib/linux-image-${version}/. /boot/dtbs/
EOF

Let's make this script executable:

chmod +x /etc/kernel/postinst.d/copy-dtbs

And run it once manually to copy the dtb files for the currently installed kernel:

/etc/kernel/postinst.d/copy-dtbs `uname -r`

Kernel from unstable

As discussed previously, it might be necessary to run the kernel from the unstable distribution in order for your board to work. In addition, some board features might work better with kernels from unstable.

Note: Some people recommend installing kernels from backports instead of from unstable. However, I discourage this approach as the kernels in backports often receive security updates quite late.

So, which kernel should you choose if your board is supported in stable as well? It depends. The only advice I can give is: If all features of your board that are relevant to you work with the stable kernel, stick with it. If not, try the unstable kernel.

To install the unstable kernel, we need to first add the unstable distribution to apt:

cat <<'EOF'>>/etc/apt/sources.list.d/unstable.list
deb http://deb.devuan.org/merged unstable main
EOF

And specify that we only want to install the kernel from unstable (and no other packages):

cat <<'EOF'>>/etc/apt/preferences.d/99unstable
# Never prefer packages from unstable
Package: *
Pin: release a=unstable
Pin-Priority: 1

# Allow upgrading kernel from unstable
Package: linux-image-arm64
Pin: release a=unstable
Pin-Priority: 500
EOF

Finally, we can safely upgrade the kernel:

apt update
apt dist-upgrade

To make sure the new kernel boots, let's do a reboot:

reboot

Time synchronization

Most SbCs don't have a backup battery for their realtime clock (RTC). This results in the date and time being wrong each time you disconnect power. In general, even if your board has a backup battery, it is a good idea to setup time synchronization.

Synchronizing date and time can be done using the NTP protocol. Debian ships several NTP clients you can choose from. Personally, I recommend OpenBSD's openntpd as it is very lightweight and has been developed with security in mind. To install it:

apt install openntpd

It should start and also run on the next boot. In case you want to test it or sync manually, you can invoke it as follows:

ntpd -s -d

Bootloader configuration

Create a directory to hold the bootloader configuration:

mkdir -p /boot/extlinux

And create an extlinux boot configuration file (the bootloader u-boot looks them up):

cat <<'EOF' >> /boot/extlinux/extlinux.conf
TIMEOUT 2
PROMPT 1
DEFAULT devuan

LABEL devuan
MENU LABEL Linux devuan
KERNEL /vmlinuz
INITRD /initrd.img
DEVICETREEDIR /dtbs
APPEND root=LABEL=root
EOF

Disable suspend-to-RAM

WIP: The described approach does not work yet, follow the discussion in the Devuan forums [2].

As we won't use suspend to RAM, do the following:

echo "RESUME=none" > /etc/initramfs-tools/conf.d/resume

To make this change effective, we have to update the initramfs:

update-initramfs -u -k all

SSH server

Installing the SSH server is a bit tricky because it will generate unique key pairs for this installation - something you don't want for reuseable images. Therefore, I recommend doing a snapshot of the VM or backing up its disk image before installing the SSH server. To install the SSH server:

apt install --no-install-recommends openssh-server

You can manually regenerate the keys as follows:

service ssh stop
rm -v /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server
service ssh start

Final “devastating” changes

The following changes will make the VM unbootable. Therefore, I recommend doing a snapshot of the VM or backing up its disk image before commiting them.

Remove the grub bootloader (we will not use it):

apt autoremove grub-efi-arm64

Replace the fstab file:

cat <<'EOF' > /etc/fstab
# <file system> <mount point>   <type>  <options>       		<dump>  <pass>
LABEL=root	/               f2fs    defaults errors=remount-ro	0       1
LABEL=boot	/boot           ext4    errors=remount-ro        	0       2
EOF

Step 3: Creating tar archives

After tweaking the system to our needs, we want to create tar archives that will contain the contents that will be used for the boot and root partitions later. To do this, we will mount the disk images of the VM directly on the host and extract the files from there.

First, power down your VM:

poweroff

To extract the files from the VM, load the nbd module on your host.

modprobe nbd max_part=8

Mount the VM's disk image (replace the path according to where you stored your virtual disk):

qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/devuan-aarch64.qcow

Mount the boot partition (the guided installation installs to the second partition by default):

mount /dev/nbd0p2 /mnt

Create tar archives with our rootfs and bootfs and store them in our assets:

cd /mnt
tar cfvzp /home/youruser/assets/devuan-aarch64-rootfs.tar.gz --exclude="/mnt/boot" .
cd boot
tar cfvzp /home/youruser/assets/devuan-aarch64-bootfs.tar.gz .

Finally, clean up:

cd
umount /mnt
qemu-nbd -d /dev/nbd0

Step 4: Building the bootloader

We build u-boot on the host machine.

Install dependencies:

apt install device-tree-compiler build-essential gcc make git libssl-dev python3-dev bison flex bc libssl-dev make gcc swig gcc-aarch64-linux-gnu gcc-arm-none-eabi

Clone ARM's Trusted-Firmware source repository:

git clone https://github.com/ARM-software/arm-trusted-firmware
cd arm-trusted-firmware

Checkout the latest stable tag (check with git tag, it was v2.3 at the time of writing):

git checkout v2.3

Compile the firmware (for other platforms, replace rk3399 - e. g. by rk3288 if you build for a Rock64 board):

make CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31

Leave the build directory:

cd ..

Next, clone the u-boot repository:

git clone git://git.denx.de/u-boot.git 

Checkout the latest stable tag (again, check with git tag):

git checkout v2020.04

Create a symlink to the previously produced arm trusted firmware build (if you built for a different board, adapt the link):

ln -s ../arm-trusted-firmware/build/rk3399/release/bl31/bl31.elf bl31.elf

Create the configuration from the default config (again, adapt it if you have a different board):

make CROSS_COMPILE=aarch64-linux-gnu- BL31=bl31.elf rockpro64-rk3399_defconfig

Start the build (change -j16 to match the number of your actual cpu threads):

make -j16 CROSS_COMPILE=aarch64-linux-gnu- BL31=bl31.elf all u-boot.itb

Save the assets:

cp u-boot/idbloader.img /home/youruser/assets/
cp u-boot/u-boot.itb /home/youruser/assets/

Step 5: Assembling a device-specific image

With all goodies in place, we can now create a flashable image for your board. Unless you need more than four partitions I recommend sticking with a dead-simple MBR partition scheme instead of using GPT.

Before we start, let's install kpartx:

apt install kpartx

First, change to our assets directory:

cd /home/youruser/assets

Create an image file for your device:

dd if=/dev/zero of=devuan3-rockpro64.img bs=1M count=2048

Then, create a template to hold our partition table (adjust the sizes of the boot and root partition if you have a bigger microSD card, but don't make the root partition too big):

cat <<'EOF'>>sfdisk.template
label: mbr
unit: sectors
first-lba: 64

start=        2048, size=       16384
start=       18432, size=      614400, bootable
start=      632832, size=     7372800
EOF

Now, apply the template to the disk image:

/sbin/sfdisk devuan3-rockpro64.img < sfdisk.template

Refresh the partition information:

kpartx -v -a devuan3-rockpro64.img

This should result in three devices (loop0p1 to loop0p3) showing up.

WARNING: Double-check this! If you have other active loop devices already, the following command might destroy your data!

Now, create an ext2 filesystem (preferred over ext4 due to less wear) to hold our boot partition and an f2fs filesystem to hold our root partition:

mkfs.ext2 -m0 -L boot /dev/mapper/loop0p2
mkfs.f2fs -l root /dev/mapper/loop0p3

Mount the boot partition and extract the tar archive:

mount /dev/mapper/loop0p2 /mnt
cd /mnt
tar xzvpf /home/youruser/assets/devuan-aarch64-bootfs.tar.gz .
sync
cd
umount /mnt

And do the same for the rootfs:

mount /dev/mapper/loop0p3 /mnt
cd /mnt
tar xzvpf /home/youruser/assets/devuan-aarch64-rootfs.tar.gz .

Before we unmount it, take the chance to adjust the settings for the serial console (only needed for Devuan, in Debian systemd automatically figures out the serial device). To do that, edit the file /mnt/etc/inittab and adjust the following line:

T0:23:respawn:/sbin/getty -L ttyAMA0 115200 screen

For the RockPro64, the serial device is ttyS2 (not ttyAMA0) and it uses a baud rate of 15000000 (not 115200). Again, other boards may need different settings.

In our case, change the line as follows:

T0:23:respawn:/sbin/getty -L ttyS2 1500000 screen

With that last change done, unmount the new rootfs

sync
cd /home/youruser/assets
umount /mnt

As the last step, we need to write the bootloader to the device.

Warning: This procedure is SoC-specific! Other SoCs require different commands!

For our RockPro64. do the following:

dd if=idbloader.img of=devuan3-rockpro64.img seek=64 conv=notrunc
dd if=u-boot.itb of=devuan3-rockpro64.img seek=16384 conv=notrunc

Your image is now ready to be flashed.

Step 6: Flashing the assembled image

To flash it to an microSD card (replace /dev/sdX by the actual device):

dd if=devuan3-rockpro64.img of=/dev/sdX bs=1M
sync

That's it! Hook up the serial console to your board, put in the microSD card and enjoy.

External References

(The providers of these resources are solely responsible for them - see legal notice).

History / Changelog

  • [2020-06-21] Initial writeup

Comments

(Comment features are provided by external parties and are not monitored by me.)

Join the discussion on Mastadon (external resource)