Rescue or backup entire host

We will take a look at multiple unfortunate scenarios - all in one - none of which appear to be well documented, let alone intuitive when it comes to either:

• troubleshooting a Proxmox VE host that completely fails to boot; or
• a need to create a full host backup - one that is safe, space-efficient and the re-deployment scenario target agnostic.

Entire PVE host install (without guests) typically consumes less than 2G of space and it makes no sense to e.g. go about cloning entire disk (partitions), which a target system might not even be able to fit, let alone boot from.

Rescue not to the rescue

Natural first steps while attempting to rescue a system would be to aim for the bespoke PVE ISO installer ¹ and follow exactly the menu path:

• Advanced Options > Rescue Boot

This may indeed end up booting up partially crippled system, but it is completely futile in a lot of scenarios, e.g. on otherwise healthy ZFS install, it can simply result in an instant error:

• error: no such device: rpool
• ERROR: unable to find boot disk automatically

Besides that, we do NOT want to boot the actual (potentially broken) PVE host, we want to examine it from a separate system that has all the tooling, make necessary changes and reboot back instead. Similarly, if we are trying to make a solid backup, we do NOT want to be performing this on a running system - it is always safer for the entire system being backed up to be NOT in use, safer than backing up a snapshot would be.

ZFS on root

We will pick the “worst case” scenario of having a ZFS install. This is because standard Debian does NOT support it out-of-the box and while it would be appealing to simply make use of corresponding Live System ² to boot from (e.g. Bookworm for the case of PVE v8), this won’t be of much help with ZFS as provided by Proxmox.

status: The pool uses the following feature(s) not supported on this system:
  com.klarasystems:vdev_zaps_v2
action: The pool cannot be imported. Access the pool on a system that supports
  the required feature(s), or recreate the pool from backup.

We will therefore make use of the ISO installer, however go for the not-so-intuitive choice:

• Advanced Options > Install Proxmox VE (Terminal UI, Debug Mode)

This will throw us into terminal which would appear stuck, but in fact it would be ready for input reading:

Debugging mode (type 'exit' or press CTRL-D to continue startup)

Which is exactly what we will do at this point, press C^D to get ourselves a root shell:

root@proxmox:/# _

This is how we get a (limited) running system that is not our PVE install that we are (potentially) troubleshooting.

Get network and SSH access

This step is actually NOT necessary, but we will opt for it here as we will be more flexible in what we can do, how we can do it (e.g. copy & paste commands or even entire scripts) and where we can send our backup (other than a local disk).

Assuming the network provides DHCP, we will simply get an IP address with dhclient: ⁴

dhclient -v

The output will show us the actual IP assigned, but we can also check with hostname -I, ⁵ which will give us exactly the one we need without looking at all the interfaces.

We will now install SSH server: ⁷

apt update
apt install -y openssh-server

Further, we need to allow root to actually connect over SSH, which - by default - would only be possible with a key, either manually editing the configuration file and looking for PermitRootLogin ⁸ line that we uncomment and edit accordingly, or simply appending the line with:

cat >> /etc/ssh/sshd_config <<< "PermitRootLogin yes"

Time to start the SSH server:

mkdir /run/sshd
/sbin/sshd

One last thing, let’s set ourselves a password for the root:

passwd

And now remote connect from another machine - and use it to make everything further down easier on us:

ssh root@10.10.10.101

Import the pool

We will proceed with the ZFS on root scenario, as it is the most tricky. If you have any other setup, e.g. LVM or BTRFS, it is much easier to just follow readily available generic advice on mounting those filesystems.

All we are after is getting access to what would ordinarily reside under the root (/) path, mounting it under a working directory such as /mnt. This is something that a regular mount ¹⁰ command will NOT help us with in a ZFS scenario.

If we just run the obligatory zpool import ¹¹ now, we would be greeted with:

   pool: rpool
     id: 14129157511218846793
  state: UNAVAIL
status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

	rpool       UNAVAIL  unsupported feature(s)
	  sda3      ONLINE

And that is correct. But a pool that has not been exported does not signify anything special beyond that the pool has been marked by another “system” and is therefore presumed to be unsafe for manipulation by others. It’s a mechanism to prevent the same pool being accessed by multiple hosts at same time inadvertently - something, we do not need to worry about here.

We could use the (in)famous -f option, this would be even suggested to us if we were more explicit about the pool at hand:

zpool import -R /mnt rpool

cannot import 'rpool': pool was previously in use from another system.
Last accessed by (none) (hostid=9a658c87) at Mon Jan  6 16:39:41 2025
The pool can be imported, use 'zpool import -f' to import the pool.

But we do NOT want this pool to then appear as foreign elsewhere. Instead, we want current system to think it is the same as the one originally accessing the pool. Take a look at the hostid ¹² that is expected: 9a658c87 - we just need to write it into the binary /etc/hostid file - there’s a tool for that: ¹³

zgenhostid -f 9a658c87

Now importing a pool will go without a glitch… Well, unless it’s been corrupted, but that would be for another guide.

zpool import -R /mnt rpool

There will NOT be any output on the success of the above, but you can confirm all is well with: ¹⁴

zpool status

Chroot and fixing

What we have now is the PVE host’s original filesystem mounted under /mnt/ with full access to it. We can perform any fixes, but some tooling (e.g. fixing a bootloader - something out of scope here) might require paths to be as-if real from the viewpoint of a system we are fixing, i.e. such tool could be looking for config files in /etc/ and we do not want to worry about having to explicitly point it at /mnt/etc while preserving the imaginary root under /mnt - in such cases, we simply want to manipulate the “cold” system as if it was currently booted one. That’s where chroot ¹⁵ has us covered:

chroot /mnt

And until we then finalise it with exit, our environment does not know anything above /mnt and most importantly it considers /mnt to be the actual root (/) as would have been the case on a running system.

Now we can do whatever we came here for, but in our current case, we will just back everything up, at least as far as the host is concerned.

Full host backup

The simplest backup of any Linux host is simply a full copy of the content of its root / filesystem. That really is the only thing one needs a copy of. And that’s what we will do here with tar: ¹⁶

tar -cvpzf /backup.tar.gz --exclude=/backup.tar.gz --one-file-system / 

This will back up everything from the (host’s) root (/ - remember we are chroot’ed), preserving permissions, and put it into the file backup.tar.gz on the very (imaginary) root, without eating its own tail, i.e. ignoring the very file we are creating here. It will also ignore mounted filesystems, but we do not have any in this case.

Once done, we exit the chroot, literally:

exit

What you do with this archive - now residing in /mnt/backup.tar.gz is completely up to you, the simplest possible would be to e.g. securely copy it out over SSH, even if only just a fellow PVE host: ¹⁷

scp /mnt/backup.tar.gz root@10.10.10.11:~/

The above would place it into the remote system’s root’s home directory (/root there).

apt install -y sshfs
mkdir /backup
sshfs root@10.10.10.11:/root /backup

That’s it

Once done, time for a quick exit: ^{19 20}

zfs unmount rpool
reboot -f

And there you have it, safely booting into an otherwise hard to troubleshoot setup with bespoke Proxmox kernel guaranteed to support the ZFS pool at hand and complete backup of the entire host system.

If you wonder how this is sufficient, how to make use of such “full” backup (of less than 1G) and ponder the benefit of block cloning entire disks with de-duplication (or lack thereof on encrypted volumes) only to later find out the target system needs differently sized partitions with different capacity disks, or even different filesystems and is a system booting differently - there’s none and we will demonstrate so in a follow-up guide.