Sometimes it can be useful to be able to manually install an Ubuntu system – either from an existing running system, or a live installation media environment. Fortunately this process is rather straightforward.

Motivation

Approximately once per month, I find myself in need of reinstalling an Ubuntu server. These systems can be spread across the world, with no means of physical access and sometimes limited IPMI KVM capabilities.

It might be a time consuming process to obtain KVM access and when I finally get access to the interface, the virtual media facility doesn’t work correctly.

What’s the solution? Well, assuming the system is not completely borked, we still have SSH access! Servers typically have at least two drives, which also plays an important role in crafting alternative solutions.

(sidenote: This process is also helpful when subiquity crashes and we want to carry on without a long reboot by just dropping into the shell)

Prerequisites

We will need to have at least one empty disk where we will put the new installation. If the server has multiple drives but they are all in a redundant RAID configuration, one option is simply to kick one drive out (mark failed, remove) and zero the superblock to make sure the array metadata is cleared out.

Manual installation

In the steps outlined below, I’m working with a system with two blank drives, sdb and sdc. If only one drive is available, one option is to skip the RAID creation altogether and work directly with the disk.

Alternatively, you can create a degraded array with one drive missing. Once the new system is fully installed and booted into, the new array can simply be expanded with the remaining drives from the old array.

There are two ways to install Ubuntu, either with EFI or Legacy/BIOS booting. Steps that differ for each one are marked with “EFI ONLY” or “BIOS BOOT ONLY” accordingly.

Preparing the system and storage

# Install dependencies required to bootstrap the system
apt install debootstrap arch-install-scripts mdadm xfsprogs
### EFI ONLY ###
# Create the EFI partitions
fdisk /dev/sdb # (repeat for /dev/sdc)
# ➝ [n]ew partition, 512MB
# ➝ [t] ➝ "ef"
# 
mkfs.fat -F 32 /dev/sdb1
mkfs.fat -F 32 /dev/sdc1
### END EFI ONLY ###
# Prepare the root filesystem block devices
fdisk /dev/sdb # (repeat for /dev/sdc)
# ➝ [n]ew partition
# ➝ [w]
# Create a root filesystem RAID1 device
# ! Only one option needs to be selected here !
# We do NOT want to create a RAID over the EFI partitions
### BIOS BOOT ONLY ###
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
### END BIOS BOOT ONLY ###

### EFI ONLY ###
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb2 /dev/sdc2
### END EFI ONLY ###
# Create a filesystem and mount it to /mnt
mkfs.xfs -K /dev/md0
mount /dev/md0 /mnt

Bootstrapping Ubuntu

In this case we are installing Focal, or in other words, Ubuntu 20.04 LTS.

# Deboostrap Ubuntu Focal
debootstrap focal /mnt http://us.archive.ubuntu.com/ubuntu/
# Generate an fstab for the target system,
# getting rid of the auto-created swap in the process
genfstab -U /mnt | grep -iv swap >> /mnt/etc/fstab
# Before we can chroot into the target system,
# we need to mount special filesystems to paths
# they should be present in
mkdir -p /mnt/{proc,sys,dev,dev/pts}
mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /dev/pts /mnt/dev/pts
mount --bind /sys /mnt/sys
# Chroot into the newly installed target
chroot /mnt

Basic configuration

What we have now is a completely barebones system that will not boot by itself and won’t be overly useful either. Let’s add some basic configuration.

# Enable typical APT sources - restricted, universe, multiverse, backports
cat <<EOF > /etc/apt/sources.list
deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
deb http://us.archive.ubuntu.com/ubuntu/ focal universe
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu focal-security main restricted
deb http://security.ubuntu.com/ubuntu focal-security universe
deb http://security.ubuntu.com/ubuntu focal-security multiverse
EOF
# Install minimal utilities
apt update
apt -y upgrade
apt -y install curl nano mdadm initramfs-tools \
    ubuntu-minimal openssh-server xfsprogs xfsdump parted gdisk
# Install a kernel - we can pick which one we want (pick one)
#NORMAL: # apt -y install linux-image-generic
#HWE:    # apt -y install --install-recommends linux-generic-hwe-20.04
# Generate a locale configuration
locale-gen en_US.UTF-8
locale-gen en_GB.UTF-8
update-locale LANG=en_US.UTF-8
# Enable root to SSH into the machine
sed -i 's/^#PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
# Set the root password
passwd root
# Set the hostname to something distinctive, so it is immediately
# apparent which system the server booted into
echo "newsystem" > /etc/hostname
# Create a persistent mdadm configuration
/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf
# Update the initramfs
update-initramfs -u
# Generate keys for SSH
ssh-keygen -A

Installing the bootloader

The system is almost configured, it’s time to install GRUB so we can actually boot into it.

### EFI ONLY ###
# Mount both EFI partitions
mkdir /boot/efi
mkdir /boot/efi2

mount /dev/sdb1 /boot/efi
mount /dev/sdc1 /boot/efi2

# Mount EFI vars
mount -t efivarfs none /sys/firmware/efi/efivars
# ... and install grub
apt install grub-efi-amd64
dpkg-reconfigure grub-efi-amd64
### END EFI ONLY ###
# Install grub on the target drives
grub-install /dev/sdb
grub-install /dev/sdc

# Just in case
update-grub

Network configuration

Now it’s time to create a network configuration so our new server can reach the internet. If we’re running this on a server that already has a working network connectivity, great, we don’t need to do almost anything.

If not, we need to recreate it from scratch.

# Exit the chroot
exit

# Copy the existing network configuration if present
cp /etc/netplan/* /mnt/etc/netplan/

Now, there is one thing that I will highly suggest. While Linux in theory has persistent device naming and the device names should remain the same, in practice this has proven to not be extremely reliable on some platforms.

The simplest fix to make sure we don’t get locked out of the reinstalled server by having a non-working network is to simply add matching based on interface MAC addresses into the Netplan configuration.

So for example, the resulting Netplan file might look like this:

network:
 version: 2
 renderer: networkd
 ethernets:
   eno1:
     match:
       macaddress: ac:1f:6b:1a:1b:1c
     addresses: [ 192.168.1.100/24 ]
     gateway4: 192.168.1.1
     nameservers:
         addresses:
             - "8.8.8.8"

Closing thoughts

We can now reboot the server and if all goes well (fingers crossed!), it will boot into the new install. If it does not, it might be that it’s booting from the wrong drive for example. That can be remediated by overwriting GRUB on that drive (grub install /dev/sda) or changing the boot order (efibootmgr).