Backup and disaster recovery
Introduction
While you might never use them, your backup plan (and the corresponding disaster recovery steps) are possibly your most important procedure.
You should:
Write it up fully
Test it from beginning to end, simulating an actual disaster
Run backups on a regular basis, ideally automatically
This page explains in detail how to properly backup an on-premise Wire installation, and how to recover from your backup if you ever need to.
Note that you should not trust this page (or your execution of it) to allow for a proper backup and restore, therefore, you should immediately (before it becomes critical to do so) back up your Wire installation and attempt (as a test) to restore it. This will ensure that you can in fact backup and restore Wire, and will catch any problem in the process before any problem becomes damaging to you.
Backing up
By nature, a significant part of a Wire installation is ephemeral, and not meant to be stored long-term or recovered: in case of trouble, those parts can just be started fresh with minimal impact on user experience.
The exceptions to this rule, are what you want to back up. In particular:
Your “wire-server” installation folder (used during the installation procedure)
Your Cassandra database
Your Minio data
If you save these, you can then whenever needed, create a fresh installation of Wire, re-import/re-install them on top of it, and get back to a working state.
Here is how to back up each:
Backing up Wire-server
To backup the wire-server folder, we simply use ssh to read it from the server in which it is installed:
ssh user@my-wire-server.your-domain.com 'cd /path/to/my/folder/for/wire-server/ && tar -cf - wire-server | gzip -9' > wire-server-backup.tar.gz
Where :
useris the user you used to install Wire on this server, typicallywireorrootmy-wire-server.your-domain.comis the domain name or IP address for the server with your Wire install/path/to/my/folder/for/wire-server/is the (absolute) path, on the server, where your wire-server folder is locatedwire-serveris the name of the folder for your wire-server install, typically (as per instructions) simplywire-serverwire-server-backup.tar.gzis the name of the file in which your wire-server folder will be stored
Once the command is done executing, make sure the file exists and is not empty with:
file wire-server-backup.tar.gz # Should say the file type is a TAR archive
ls -lh wire-server-backup.tar.gz # Should show a file size other than 0
tar tvf wire-server-backup.tar.gz # Should list the files inside your wire-server folder correctly
Now simply save this file in multiple locations as per your normal company backup procedures, and repeat this procedure on a regular basis as is appropriate.
Backing up Cassandra
Cassandra stores things such as user profiles/accounts, conversations, etc. It is the most critical data to backup/store/recover.
To backup your Cassandra database, do as follows:
You can read general information about connecting to your Cassandra node on this page
In particular, SSH into the Cassandra Virtual Machine with:
ssh user@cassandra-vm.your-domain.com
Where:
useris the user you used to install Wire on this server, typicallywireorrootcassandra-vm.your-domain.comis the domain name or IP address for the server with your Cassandra node
Make sure (while connected via ssh) your Cassandra installation is doing well with:
nodetool status
or (in newer versions)
nodetool -h ::FFFF:127.0.0.1 status
You should see a list of nodes like this:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.220.13 9.51MiB 256 100.0% 3dba71c8-eea7-4e35-8f35-4386e7944894 rack1
UN 192.168.220.23 9.53MiB 256 100.0% 3af56f1f-7685-4b5b-b73f-efdaa371e96e rack1
UN 192.168.220.33 9.55MiB 256 100.0% RANDOMLY-MADE-UUID-GOES-INTHISPLACE! rack1
As per the Cassandra documentation to backup your database, you will use the nodetool snapshot command.
First we need to edit the cassandra.yaml file (which you can if needed find with find -name cassandra.yaml) has auto_snapshot set to false:
auto_snapshot: false
Same with: snapshot_before_compaction
snapshot_before_compaction: false
After editing the file, make sure you restart cassandra with:
sudo service cassandra restart
You can find a list of all keyspaces by doing:
ls /mnt/cassandra/data/
Now (while connected via ssh, as per above), use nodetool to actually generate a snapshot of all tables:
nodetool snapshot --tag catalog-ks catalogkeyspace
This should succesfully save the snapshots to the disk.
You should now be able to find your snapshots in the snapshots list with:
nodetool listsnapshots
To actually save the files, you’ll need to locate them with:
find -name snapshots
Which should give you a list of paths to snapshots, such as:
/mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/snapshots
/mnt/cassandra/data/data/catalogkeyspace/magazine-446eae30c22a11e9b1350d927649052c/snapshots
Now to create a (local) backup of these snapshots, we use ssh the same way we did above for wire-server:
ssh user@cassandra-vm.your-domain.com 'cd /mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/ && tar -cf - snapshots | gzip -9' > cassandra-catalogkeyspace-journal-backup.tar.gz
Where :
useris the user you used to install Wire on this server, typicallywireorrootcassandra-vm.your-domain.comis the domain name or IP address for the server with your Wire install/mnt/cassandra/is the (absolute) path, on the server, where your cassandra folder is located, to which you add the location of the specific snapshot, found withfindabove
Repeat this for each of the snapshots.
Now simply save these files in multiple locations as per your normal company backup procedures, and repeat this procedure on a regular basis as is appropriate.
Batch Cassandra backup
The backup procedure above presumes manually backing up each Cassandra snapshot serially one by one.
If you want to back all files at once, you can use the find command to find all snapshots and pack them into a single archive:
ssh user@cassandra-vm.your-domain.com 'find /mnt/cassandra/ -name "snapshots" -print0 | tar -cvf - --null -T - | gzip -9 ' > cassandra-batch.tar.gz
useris the user you used to install Wire on this server, typicallywireorrootcassandra-vm.your-domain.comis the domain name or IP address for the server with your Wire install/mnt/cassandra/is the (absolute) path, on the server, where your cassandra folder is located, to which you add the location of the specific snapshot, found withfindabovecassandra-batch.tar.gzis the local file everything will be written to
This will (over ssh) find all snapshot folders in the cassandra folder, pack them into a tar file, gzip it to save space, and output it to a local file.
Backing up MinIO
MinIO emulates an Amazon-S3-compatible file-storage setup, and is used by Wire to store things such as file attachments, images etc.
If your specific installation is using the actual Amazon file storage (and not a local emulated alternative), you should skip this section (but still actually backup whatever you are using).
Similarly to Cassandra, to create a backup you need to SSH into the Virtual Machine running MinIO in your installation:
You can read general information about your MinIO node on this page.
SSH into the MinIO Virtual Machine with:
ssh user@minio-vm.your-domain.com
Where:
useris the user you used to install Wire on this server, typicallywireorrootminio-vm.your-domain.comis the domain name or IP address for the server with your MinIO node
To backup the MinIO data, we need to backup two servers over SSH, the same way we did for Cassandra and wire-server:
ssh user@my-minio-server.your-domain.com 'cd /var/lib/ && tar -cf - minio-server1 | gzip -9' > minio-server1-backup.tar.gz
ssh user@my-minio-server.your-domain.com 'cd /var/lib/ && tar -cf - minio-server2 | gzip -9' > minio-server2-backup.tar.gz
Where:
useris the user you used to install Wire on this server, typicallywireorrootmy-minio-server.your-domain.comis the domain name or IP address for the MinIO Virtual Machineminio-server[number]-backup.tar.gzis the name of the file in which each minio data folder will be stored
Automated regular backups
It is important to back up as often as possible. You can use cron to automatically run your backup commands on a regular basis.
For example, you can create the following shell script, and write it to /home/myuser/backup/wire-backup.sh:
#!/bin/sh
# Make the folder if it does not exist yet
mkdir -p /home/myuser/backup/data/
# Back up wire-server folder
ssh user@my-wire-server.your-domain.com 'cd /path/to/my/folder/for/wire-server/ && tar -cf - wire-server | gzip -9' > /home/myuser/backup/data/wire-server-backup.tar.gz
# Cause Cassandra to generate new snapshots
ssh user@cassandra-vm.your-domain.com 'nodetool snapshot --tag catalog-ks catalogkeyspace`
# Backup Cassandra snapshots to a file
ssh user@cassandra-vm.your-domain.com 'find /mnt/cassandra/ -name "snapshots" -print0 | tar -cvf - --null -T - | gzip -9 ' > /home/myuser/backup/data/cassandra-batch.tar.gz
# Backup MinIO files
ssh user@my-minio-server.your-domain.com 'cd /var/lib/ && tar -cf - minio-server1 | gzip -9' > /home/myuser/backup/data/minio-server1-backup.tar.gz
ssh user@my-minio-server.your-domain.com 'cd /var/lib/ && tar -cf - minio-server2 | gzip -9' > /home/myuser/backup/data/minio-server2-backup.tar.gz
# Tar all backup files into a single unified archive
tar -cf /home/myuser/backup/full-backup.tar /home/myuser/backup/data/*
# Make remote copies of this single file to remote hosts for redundancy
scp /home/myuser/backup/full-backup.tar user@remote-redundancy-host-one.your-domain.com:/path/to/backup/folder/
scp /home/myuser/backup/full-backup.tar user@remote-redundancy-host-two.your-domain.com:/path/to/backup/folder/
Make the file executable with:
chmod +x /home/myuser/backup/wire-backup.sh
You can then manually run the file with:
./home/myuser/backup/wire-backup.sh
As a test, run the script manually to make sure it actually properly does its job without any errors.
Then, you can add this to your cron file (by running crontab -e) to make sure this commands gets executed on a regular basis (here we will use an hourly backup):
# Edit this file to introduce tasks to be run by cron.
#
# m h dom mon dow command
@hourly /home/myuser/backup/wire-backup.sh
Your backup should now happen every hour automatically, ensuring you always have a backup at least an hour fresh in case of an emergency.
There are ways to have incremental backups and to do more complex/refined backup procedure, but those are beyond the scope of this document. You should check out Borg.
You should also set up your monitoring software (for example Nagios) to check whether your backup file is older than an hour, and if it is (meaning something went wrong), warn you immediately.
Recovery procedure
If the worse has happened, and you need to recover/restore your Wire installation, you will need to do the following:
Create a new Wire installation from scratch (following this website or other customer-specific on-premise instructions you used the first time around).
While creating this new wire installation, instead of using a fresh/empty
wire-serverfolder, you use thewire-serverfolder you backed up above (as it contains yourvaluesandsecretfiles among other important files).Restore your Cassandra backup files over your new Cassandra installation.
Restore your MinIO backup files over your new MinIO installation.
Restart all services.
Ensure all services are functioning correctly
Restoring Wire-Server from a backup
If you correctly backup up your wire-server installation, you should have access to a file named wire-server-backup.tar.gz (see above).
You can extract this file to the remote machine with the following command:
cat wire-server-backup.tar.gz | ssh user@my-wire-server.your-domain.com "cd /path/to/my/folder/for/wire-server/ && tar zxvf -"
Where :
wire-server-backup.tar.gzis the name of the file in which your wire-server folder has been storedmy-wire-server.your-domain.comis the domain name or IP address for the server with your Wire installuseris the user you used to install Wire on this server, typicallywireorroot/path/to/my/folder/for/wire-server/is the (absolute) path, on the server, where your wire-server folder is located
Now all files should be in their proper place, with the proper values, and you should be able to run the required ansible/helm commands that will result in a full installation of Wire.
Restoring Cassandra from a backup
If you correctly backed up your Cassandra database, you should have a series of snapshot files, which you now need to restore over your “fresh” (ie. empty) Cassandra installation.
This page from the Cassandra documentation goes over how to restore from snapshots.
You should have lots of snapshots from the backup process. This example will go over how to restore one of the snapshots, but you should do this process for each and every snapshot you backup up.
First, transfer and extract the snapshot to the Cassandra Virtual Machine:
cat cassandra-catalogkeyspace-journal-backup.tar.gz | ssh user@cassandra-vm.your-domain.com "cd /mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/ && tar zxvf -"
Now the folder /mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/snapshots/ should contain your snapshot.
Next, ssh into the Cassandra Virtual Machine and cd to the snapshot folder:
ssh user@cassandra-vm.your-domain.com
cd /mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/
Next run
ls -lh
This will show you a list of the snapshots available:
drwxr-xr-x 2 cassandra cassandra 4.0K May 26 14:45 1653752714460
drwxr-xr-x 2 cassandra cassandra 4.0K May 28 17:46 1653752759667
Select the most recent one (here it is 1653752759667).
Finally, use the sstableloader file to load the snapshot to Cassandra:
sstableloader -d localhost /mnt/cassandra/data/data/catalogkeyspace/journal-296a2d30c22a11e9b1350d927649052c/snapshots/1653752759667/
This should load the snapshot.
Repeat this for all snapshots you saved, and you should have a fully restored Cassandra database.
Finally, restart Cassandra:
sudo service cassandra restart
And ensure that it is working properly.
Restoring MinIO from a backup
Restoring MinIO from a backup is as simple as extracting the files (minio-server1-backup.tar.gz and minio-server2-backup.tar.gz which you have backup up in the backup procedure and should have access to) to the correct remote host (MinIO Virtual Machine), and restarting MinIO:
First run:
cat minio-server1-backup.tar.gz | ssh user@my-minio-server.your-domain.com "cd /var/lib/ && tar zxvf -"
cat minio-server2-backup.tar.gz | ssh user@my-minio-server.your-domain.com "cd /var/lib/ && tar zxvf -"
Where:
useris the user you used to install Wire on this server, typicallywireorrootmy-minio-server.your-domain.comis the domain name or IP address for the MinIO Virtual Machineminio-server[number]-backup.tar.gzis the name of the file in which each minio data folder was be stored and backup up
Finally SSH into the server and restart MinIO:
ssh user@minio-vm.your-domain.com
sudo service minio restart
Where:
useris the user you used to install Wire on this server, typicallywireorrootminio-vm.your-domain.comis the domain name or IP address for the server with your MinIO node