Initial setup

Run on client:

sudo git clone https://git.jim.sh/jim/borg-setup.git /opt/borg
sudo /opt/borg/initial-setup.sh

Customize /opt/borg/config.yaml as desired.

Cheat sheet

After setup, the copy of this file on the client will have the variables in this section filled in automatically

Configuration

Hostname: ${HOSTNAME}
Base directory: ${BORG_DIR}
Destination: ${BACKUP_USER}@${BACKUP_HOST}
Repository: ${BACKUP_REPO}

Commands

See when next backup is scheduled:

systemctl list-timers borg-backup.timer

See status of most recent backup:

systemctl status --full --lines 999999 --no-pager --all borg-backup

Watch log:

journalctl --all --follow --unit borg-backup

Start backup now:

sudo systemctl start borg-backup

Interrupt backup in progress:

sudo systemctl stop borg-backup

Show backups and related info:

sudo ${BORG_DIR}/borg.sh info
sudo ${BORG_DIR}/borg.sh list

Run Borg using the read-write SSH key:

sudo ${BORG_DIR}/borg.sh --rw list

Mount and look at files:

mkdir mnt
sudo ${BORG_DIR}/borg.sh mount :: mnt
sudo -s # to explore as root
sudo umount mnt

Prune old backups. Only run if sure local system was never compromised, as object deletion could have been queued during append-only operations. Requires SSH key password from bitwarden.

sudo ${BORG_DIR}/prune.sh

Design

• On server, we have a separate user account “jim-backups”. Password for this account is in bitwarden in the “Backups” folder, under ssh backup.jim.sh.

• Repository keys are repokeys, which get stored on the server, inside the repo. Passphrases are stored:

  • on clients (in /opt/borg/passphrase, for making backups)
  • in bitwarden (under borg <hostname>, user repo key)

• Each client has two SSH keys for connecting to the server:

  • /opt/borg/ssh/id_ecdsa_appendonly
    • configured on server for append-only operation
    • used for making backups
    • no password
  • /opt/borg/ssh/id_ecdsa
    • configured on server for read-write operation
    • used for manual recovery, management, pruning
    • password in bitwarden (under borg [hostname], user read-write ssh key)

• Pruning requires the password and is a manual operation, and should only be run when the client has not been compromised.

  sudo /opt/borg/prune.sh
  

• Systemd timers start daily backups:

  /etc/systemd/system/borg-backup.service -> /opt/borg/borg-backup.service
  /etc/systemd/system/borg-backup.timer -> /opt/borg/borg-backup.timer
  

• Backup script /opt/borg/backup.py uses configuration in /opt/borg/config.yaml to generate our own list of files, excluding anything that’s too large by default. This requires borg 1.2.0b1 or newer, which is why the setup scripts download a specific version.

Notes

Building Borg.bin binary from git

git clone https://github.com/borgbackup/borg.git
cd borg
virtualenv --python=python3 borg-env
source borg-env/bin/activate
pip install -r requirements.d/development.txt
pip install pyinstaller
pip install llfuse
pip install -e .[llfuse]
pyinstaller --clean --noconfirm scripts/borg.exe.spec

Note: This uses the deprecated llfuse instead of the newer pyfuse3. pyfuse3 doesn’t work because, at minimum, it pulls in trio which requires ssl which is explicitly excluded by scripts/borg.exe.spec.