Using Restic for backups with Orbit

This guide will take you through installing and configuring restic to manage backups on our object storage system, Orbit.

Restic is a modern, secure and efficient backup system. It compresses and deduplicates data to save space and securely encrypts all data in transit and at rest. It supports storing backups on various storage systems, including Orbit.

For this guide you’ll need a Brightbox account and an Orbit container and API Client. The API Client should have read/write access to the container. And you’ll need some data to backup!

Install restic

For our guide we’re using Ubuntu Linux so we can use apt to install restic but you can install it using other methods and even on MacOS and Windows

$ sudo apt update
$ sudo apt install restic -y

Configure restic

We configure restic with environment variables, but in this example we’ll put them into a local file for convenience. So decide on a name for the config file, such as ~/.restic and put the environment variables in it:

OS_AUTH_URL=https://orbit.brightbox.com/v3
OS_PROJECT_NAME=acc-xxxxx
OS_USERNAME=cli-yyyyy
OS_PASSWORD=xxxxxxxxxxxx
RESTIC_REPOSITORY=swift:restic:/backups
RESTIC_PASSWORD=xxxxxxxxxxxxxxxxxxxx

Then export those variables into your current environment like this:

$ export $(< .restic)

Now we can run the various restic commands and it will read the config from the environment.

OS_PROJECT_NAME should be your Brightbox account identifier, OS_USERNAME is your API client identifier and OS_PASSWORD is your API client secret, and RESTIC_REPOSITORY specifies the Orbit container and the directory within it to store the backups. That’s the storage configuration.

RESTIC_PASSWORD is the password used to encrypt the restic repository and you’ll need this to restore data from restic in the future. Generate a strong one and keep it safe!

If you don’t use a config file like this, take care to avoid your passwords getting saved into your shell history file!

Initialize the repository

So now let’s initialize the repository on Orbit:

$ restic init
created restic repository 4088e17631 at swift:restic:/backups

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

That’s written some metadata to Orbit to define the repository.

Backups

Our first backup

Let’s backup some data. In this example I’ve downloaded a copy of the linux kernel source code which will be the data we want to backup. It’s in a directory called kernel-source/:

$ restic backup kernel-source/

repository 4088e176 opened (version 2, compression level auto)
no parent snapshot found, will read all files
[0:00] 100.00%  1 / 1 index files loaded

Files:       81717 new,     0 changed,     0 unmodified
Dirs:         5317 new,     0 changed,     0 unmodified
Added to the repository: 1.287 GiB (268.266 MiB stored)

processed 81717 files, 1.261 GiB in 0:34
snapshot 154ba7a9 saved

As you can see from the output, restic found 1.2 GiB of new data but it only took 268 MiB in Orbit after compression.

Each backup creates in the repository what restic calls a snapshot. We can list snapshots with the snapshots command:

$ restic snapshots

repository 4088e176 opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths
---------------------------------------------------------------------------------
154ba7a9  2024-05-15 16:32:52  srv-ggijw               /home/ubuntu/kernel-source
---------------------------------------------------------------------------------
1 snapshots

Our second backup

I’ve now downloaded an updated copy of the linux kernel source code and we’ll make a new backup and see what’s changed:

$ restic backup kernel-source/

repository 4088e176 opened (version 2, compression level auto)
using parent snapshot 154ba7a9
[0:00] 100.00%  2 / 2 index files loaded

Files:        2650 new, 80760 changed,     0 unmodified
Dirs:          201 new,  5251 changed,     0 unmodified
Added to the repository: 441.940 MiB (108.870 MiB stored)

processed 83410 files, 1.301 GiB in 0:24
snapshot 828c51f1 saved

As you can see from the output, restic found that a number of files and directories had changed, totalling 441 MiB and wrote a new snapshot which took 108 MiB after compression. So now we have two snapshots in the repository on Orbit:

$ restic snapshots

repository 4088e176 opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths
---------------------------------------------------------------------------------
154ba7a9  2024-05-15 16:32:52  srv-ggijw               /home/ubuntu/kernel-source
828c51f1  2024-05-15 16:37:52  srv-ggijw               /home/ubuntu/kernel-source
---------------------------------------------------------------------------------
2 snapshots

Excluding files from a backup

You can exclude files from backups using the --exclude argument:

$ restic backup --exclude *.tmp --exclude *.log kernel-source/

Or you can list all the files and patterns you want to exclude in a separate config file (e.g: ~/.restic-excludes) and tell restic to read them all from that:

$ restic backup --exclude-file ~/.restic-excludes

If you accidentially backed up some files that you’d prefer to have excluded, see the restic rewrite command to retroactively exclude files from existing repositories.

More details of how to include and exclude files is available in restic’s own documentation.

Restoring data

Let’s now say that we’ve lost some data and need to restore from backups. We can restore the most recent backup into a directory named restore like this:

$ restic restore latest --target restore

repository 4088e176 opened (version 2, compression level auto)
[0:00] 100.00%  3 / 3 index files loaded
restoring <Snapshot 828c51f1 of [/home/ubuntu/kernel-source] at 2024-05-15 16:37:52.193105891 +0000 UTC by ubuntu@srv-ggijw> to restore
Summary: Restored 88926 files/dirs (1.301 GiB) in 0:17

$ ls restore/kernel-source/

COPYING  Documentation  Kconfig   MAINTAINERS  README  block  crypto   fs       init      ipc     lib  net   samples  security  tools  virt
CREDITS  Kbuild         LICENSES  Makefile     arch    certs  drivers  include  io_uring  kernel  mm   rust  scripts  sound     usr

We can use the various include and exclude options to restore just specific files, or exclude kinds of files:

$ restic restore latest --target restore --include /kernel-source/README

repository 4088e176 opened (version 2, compression level auto)
[0:00] 100.00%  3 / 3 index files loaded
restoring <Snapshot 154ba7a9 of [/home/ubuntu/kernel-source] at 2024-05-15 16:32:52.931689604 +0000 UTC by ubuntu@srv-ggijw> to restore
Summary: Restored 2 / 1 files/dirs (727 B / 727 B) in 0:00

Instead of just the latest backup, you can restore data from any snapshot in the repository by specifying it’s snapshot id:

So if we want data from the first backup we did, which is snapshot ID 154ba7a9:

$ restic restore 154ba7a9 --target restore

repository 4088e176 opened (version 2, compression level auto)
[0:00] 100.00%  3 / 3 index files loaded
restoring <Snapshot 154ba7a9 of [/home/ubuntu/kernel-source] at 2024-05-15 16:32:52.931689604 +0000 UTC by ubuntu@srv-ggijw> to restore
Summary: Restored 87097 files/dirs (1.261 GiB) in 0:16

Deleting old backups

To delete specific snapshots from the respository, you use the forget command:

$ restic forget 154ba7a9

repository 4088e176 opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 files deleted

But rather than manually delete old backups, restic supports rotating backups according to a schedule. So for example, say you run a backup every day and want to keep the last 7 days of backups, but after that only 6 weekly backups and after that just 12 monthly backups you can tell restic to forget the appropriate snapshots like this:

$ restic forget --keep-daily 7 --keep-weekly 6 --keep-monthly 12

restic will figure out which backups to forget to keep to that schedule. Just run that forget command once in a while (ideally after every backup) to keep your backups under control.

Freeing up data from deleted backups

Forgetting snapshots doesn’t immediately free up data from the backend Orbit storage. To do that, run the prune command:

$ restic prune

repository 4088e176 opened (version 2, compression level auto)
loading indexes...
[0:00] 100.00%  3 / 3 index files loaded
loading all snapshots...
finding data that is still in use for 2 snapshots
[0:01] 100.00%  2 / 2 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  26 / 26 packs processed

to repack:             0 blobs / 0 B
this removes:          0 blobs / 0 B
to delete:             0 blobs / 0 B
total prune:           0 blobs / 0 B
remaining:        110545 blobs / 372.813 MiB
unused size after prune: 0 B (0.00% of remaining size)

done

You can also add the --prune argument when running the forget command to auto-prune.

Verifying the backup repository

It’s important to check that the repository database itself has no problems or corruption. To do this run the check command once in a while:

$ restic check

using temporary cache in /tmp/restic-check-cache-2054111389
repository 4088e176 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-2054111389
create exclusive lock for repository
load indexes
[0:00] 100.00%  3 / 3 index files loaded
check all packs
check snapshots, trees and blobs
[0:02] 100.00%  2 / 2 snapshots
no errors were found

Deeper verification

Orbit regularly checks data stored within it and repairs any corruption found so can be relied on not to silently corrupt data, but you can run a deeper restic check which will download all the backup data and fully verify it - essentially simulating a restore of all possible data:

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-367473836
repository 4088e176 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-367473836
create exclusive lock for repository
load indexes
[0:00] 100.00%  3 / 3 index files loaded
check all packs
check snapshots, trees and blobs
[0:01] 100.00%  2 / 2 snapshots
read all data
[0:11] 100.00%  26 / 26 packs
no errors were found

If you ever find any problems with the repository, the restic repair can be used to rebuild/repair things. More details are available in the troubleshooting documentation

Backup streamed data from other commands

Rather than a local file on the filesystem, it’s common to have a command that produces data that you’d like to then backup. A good example is a database dump tool such as mysqldump.

Restic can backup data piped to it using stdin and store it as a filename you provide, e.g:

$ mysqldump --all-databases | restic backup --stdin --stdin-filename production.sql

restic will blindly backup whatever output comes from your command, even if it fails to run so you need to handle detecting if backups from stdin are working properly. An upcoming version of restic (0.17) adds the --stdin-from-command flag which makes restic run the command for you and therefore can detect if it fails, so we recommend using that feature once it’s released

Store the restic passwords securely

In this guide we’ve stored the restic passwords in a config file for simplicity. At minimum, you should make sure that config file has the proper permissions and is protected from being accessed by other users on your server.

For a system-wide installation, it’s likely you’ll want to run restic as root so that it can access all files on a server, in which case you should put the config file somewhere like /etc/restic/restic.env and allow only root to access the /etc/restic directory. You can then use cron to run the daily restic backups etc.