Initial setup

Run on client:

sudo git clone https://git.jim.sh/jim/borg-setup.git /opt/borg
sudo /opt/borg/initial-setup.sh

Customize /opt/borg/config.yaml as desired.

Cheat sheet

After setup, the copy of this file on the client will have the variables in this section filled in automatically

Configuration

Hostname: ${HOSTNAME}
Base directory: ${BORG_DIR}
Destination: ${BACKUP_USER}@${BACKUP_HOST}
Repository: ${BACKUP_REPO}

Commands

See when next backup is scheduled:

systemctl list-timers borg-backup.timer

See status of most recent backup:

systemctl status --full --lines 999999 --no-pager --all borg-backup

Watch log:

journalctl --all --follow --unit borg-backup

Start backup now:

sudo systemctl start borg-backup

Interrupt backup in progress:

sudo systemctl stop borg-backup

Show backups and related info:

sudo ${BORG_DIR}/borg.sh info
sudo ${BORG_DIR}/borg.sh list

Run Borg using the read-write SSH key:

sudo ${BORG_DIR}/borg.sh --rw list

Mount and look at files:

mkdir mnt
sudo ${BORG_DIR}/borg.sh mount :: mnt
sudo -s # to explore as root
sudo umount mnt

Compaction and remote access

Old backups are "pruned" automatically, but because the SSH key is append-only, no space is actually recovered on the server, it's just marked for deletion. If you are sure that the client system was not compromised, then you can run compaction manually directly on the backup host by logging in via SSH (bitwarden ssh ${BACKUP_HOST} / ${BACKUP_USER}) and compacting there:

ssh ${BACKUP_USER}@${BACKUP_HOST} borg/borg compact --verbose --progress ${BACKUP_REPO}

This doesn't require the repo key. That key shouldn't be entered on the untrusted backup host, so for operations that need it, use a trusted host and run borg remotely instead, e.g.:

${BORG_BIN} --remote-path borg/borg info ${BACKUP_USER}@${BACKUP_HOST}:borg/${HOSTNAME}

The repo passphrase is in bitwarden borg ${HOSTNAME} / repo key.

Design

• On server, we have a separate user account "jim-backups". Password for this account is in bitwarden in the "Backups" folder, under ssh backup.jim.sh.

• Repository keys are repokeys, which get stored on the server, inside the repo. Passphrases are stored:

  • on clients (in /opt/borg/passphrase, for making backups)
  • in bitwarden (under borg <hostname>, user repo key)

• Each client has two passwordless SSH keys for connecting to the server:

  • /opt/borg/ssh/id_ecdsa_appendonly
    • configured on server for append-only operation
    • used for making backups
  • /opt/borg/ssh/id_ecdsa_notify
    • configured on server for running borg/notify.sh only
    • used for sending email notifications on errors

• Systemd timers start daily backups:

  /etc/systemd/system/borg-backup.service -> /opt/borg/borg-backup.service
  /etc/systemd/system/borg-backup.timer -> /opt/borg/borg-backup.timer
  

• Backup script /opt/borg/backup.py uses configuration in /opt/borg/config.yaml to generate our own list of files, excluding anything that's too large by default. This requires borg 1.2 or newer.

Notes

Building Borg binary from git

sudo apt install python3.9 scons libacl1-dev libfuse-dev libpython3.9-dev patchelf
git clone https://github.com/borgbackup/borg.git
cd borg
virtualenv --python=python3.9 borg-env
source borg-env/bin/activate
pip install -r requirements.d/development.txt
pip install pyinstaller
pip install llfuse
pip install -e .[llfuse]
pyinstaller --clean --noconfirm scripts/borg.exe.spec
pip install staticx

# for x86
staticx -l /lib/x86_64-linux-gnu/libm.so.6 dist/borg.exe borg.x86_64

# for ARM; see https://github.com/JonathonReinhart/staticx/issues/209
staticx -l /lib/arm-linux-gnueabihf/libm.so.6 dist/borg.exe borg.armv7l

Then run borg.x86_64. Confirm the version with borg.armv7l --version.

Note: This uses the deprecated llfuse instead of the newer pyfuse3. pyfuse3 doesn't work because, at minimum, it pulls in trio which requires ssl which is explicitly excluded by scripts/borg.exe.spec.