My backup scripts and tools
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Jim Paris 1beda9d613 setup: pick host-dependent start time 1 hour ago
.gitignore misc: ignore .venv dir 1 day ago
Borg.bin borg: update binary to fix upstream bug 6009 1 day ago
Makefile makefile: reload systemd unit files after rebase 11 hours ago
Pipfile Implement filesystem scanning with configurable filters 5 days ago
Pipfile.lock Implement filesystem scanning with configurable filters 5 days ago
README.md setup: allow hostname to be overridden 16 hours ago
backup.py backup: fix borg exit code handling for ret=0 10 hours ago
borg.sh borg.sh: fix ssh option for read-write mode 2 hours ago
config.yaml config: remove /efi, it probably doesn't exist 1 day ago
initial-setup.sh setup: pick host-dependent start time 1 hour ago
notify.sh notify: fix notify.sh to work with server side; adjust text 1 day ago
prune.sh prune: use new vars.sh 1 day ago

README.md

Initial setup

Run on client:

sudo git clone https://git.jim.sh/jim/borg-setup.git /opt/borg
sudo /opt/borg/initial-setup.sh

Customize /opt/borg/config.yaml as desired.

Cheat sheet

After setup, the copy of this file on the client will have the variables in this section filled in automatically

Configuration

Hostname: ${HOSTNAME}
Base directory: ${BORG_DIR}
Destination: ${BACKUP_USER}@${BACKUP_HOST}
Repository: ${BACKUP_REPO}

Commands

See when next backup is scheduled:

systemctl list-timers borg-backup.timer

See status of most recent backup:

systemctl status --full --lines 999999 --no-pager --all borg-backup

Watch log:

journalctl --all --follow --unit borg-backup

Start backup now:

sudo systemctl start borg-backup

Interrupt backup in progress:

sudo systemctl stop borg-backup

Show backups and related info:

sudo ${BORG_DIR}/borg.sh info
sudo ${BORG_DIR}/borg.sh list

Run Borg using the read-write SSH key:

sudo ${BORG_DIR}/borg.sh --rw list

Mount and look at files:

mkdir mnt
sudo ${BORG_DIR}/borg.sh mount :: mnt
sudo -s # to explore as root
sudo umount mnt

Prune old backups. Only run if sure local system was never compromised, as object deletion could have been queued during append-only operations. Requires SSH key password from bitwarden.

sudo ${BORG_DIR}/prune.sh

Design

  • On server, we have a separate user account “jim-backups”. Password for this account is in bitwarden in the “Backups” folder, under ssh backup.jim.sh.

  • Repository keys are repokeys, which get stored on the server, inside the repo. Passphrases are stored:

    • on clients (in /opt/borg/passphrase, for making backups)
    • in bitwarden (under borg <hostname>, user repo key)
  • Each client has two SSH keys for connecting to the server:

    • /opt/borg/ssh/id_ecdsa_appendonly
      • configured on server for append-only operation
      • used for making backups
      • no password
    • /opt/borg/ssh/id_ecdsa
      • configured on server for read-write operation
      • used for manual recovery, management, pruning
      • password in bitwarden (under borg <hostname>, user read-write ssh key)
  • Pruning requires the password and is a manual operation, and should only be run when the client has not been compromised.

    sudo /opt/borg/prune.sh
    
  • Systemd timers start daily backups:

    /etc/systemd/system/borg-backup.service -> /opt/borg/borg-backup.service
    /etc/systemd/system/borg-backup.timer -> /opt/borg/borg-backup.timer
    
  • Backup script /opt/borg/backup.py uses configuration in /opt/borg/config.yaml to generate our own list of files, excluding anything that’s too large by default. This requires borg 1.2.0b1 or newer.

Notes

Building Borg.bin binary from git

git clone https://github.com/borgbackup/borg.git
cd borg
virtualenv --python=python3 borg-env
source borg-env/bin/activate
pip install -r requirements.d/development.txt
pip install pyinstaller
pip install llfuse
pip install -e .[llfuse]
pyinstaller --clean --noconfirm scripts/borg.exe.spec

Then see dist/borg.exe. Confirm the version with dist/borg.exe --version.

Note: This uses the deprecated llfuse instead of the newer pyfuse3. pyfuse3 doesn’t work because, at minimum, it pulls in trio which requires ssl which is explicitly excluded by scripts/borg.exe.spec.