Upgrade and backup

Keep ThirtyFile up to date, and keep a copy of your data somewhere safe.

Upgrading

Back up first. Your files and settings are kept, and the database is updated when the new version starts. Before it changes the database, ThirtyFile saves a copy of it in data/backups/drive-before-<version>.db (the last three are kept).

docker run

docker pull ghcr.io/thirtyfile/thirtyfile:latest
docker stop thirtyfile && docker rm thirtyfile
docker run -d --name thirtyfile --restart unless-stopped …   # the same options as when you installed it

Docker Compose

docker compose pull && docker compose up -d

The compose.yaml from the releases page follows the release line it came with, such as ghcr.io/thirtyfile/thirtyfile:0.4: this brings that line's repairs. To move to a newer line, read its release notes, then change the tag on the image: line (to :0.5, say) and run the same commands.

Data from 0.4.0 and later is upgraded in place: start the newer version on the same data and storage folders. Only versions up to 0.3 can't be upgraded, as the database was started over after them: install the new version with empty data and storage folders, then copy your files into it. Started on data from 0.3 or older, a newer version stops at once and says so in the log, before it writes anything into the data or storage folder, so the old data stays as it was. It also stops with a plain message on a database a newer version has changed (see Going back to an older version).

Which version you get

Both ghcr.io/thirtyfile/thirtyfile and Docker Hub's thirtyfile/thirtyfile carry the same verified image and version tags. To switch registries, change only the image name in your command or Compose file, keeping the version tag and data volumes. Signed GitHub provenance remains attached to the GHCR image; verify it there using the matching digest when checking a Docker Hub copy.

Before 1.0.0 there is no major-version tag: 0.4 is the closest, since any 0.x release may change things. The changes in each release are listed on the releases page, which also has compose.yaml and the server as a single program for Linux on amd64 and arm64, with their SHA-256 checksums in SHA256SUMS.

Every released image comes with a list of what it contains (an SBOM) and a signed record of how it was built. To check that an image or a program was built by ThirtyFile's own release workflow, use the GitHub CLI:

gh attestation verify oci://ghcr.io/thirtyfile/thirtyfile:1.2.3 --repo ThirtyFile/ThirtyFile
gh attestation verify thirtyfile-1.2.3-linux-amd64 --repo ThirtyFile/ThirtyFile

The licences of the libraries inside ThirtyFile are in /usr/share/licenses/thirtyfile/THIRD-PARTY-NOTICES in the image, and in THIRD-PARTY-NOTICES on the releases page.

Going back to an older version

An older version doesn't start on a database a newer one has changed. To go back, stop ThirtyFile, replace data/drive.db with the copy made before the upgrade (data/backups/drive-before-<new version>.db), delete drive.db-wal and drive.db-shm if they are there, and start the older image. Changes made since the upgrade (new accounts, files uploaded and their names) are lost; files uploaded since then stay in the storage folder but aren't listed.

Backing up

The data and storage folders together contain everything: files, accounts, settings and logs. The files in storage/company, storage/teams and storage/users are ordinary files, so any backup tool can copy them and they can be restored one by one; share links, permissions, the trash and earlier versions need the database too. The database in data is being written all the time, so don't copy drive.db while ThirtyFile runs; save a copy with the backup command instead, or stop ThirtyFile while copying.

Without stopping it

1. Save a consistent copy of the database (it can't be overwritten, so use a new name each time):

   docker exec thirtyfile thirtyfile backup /data/backups/drive-$(date +%F).db

2. Copy storage, and data without drive.db*, thumbs and tmp (thumbnails are made again, and tmp holds only uploads in progress):

   sudo tar czf thirtyfile-$(date +%F).tar.gz -C /srv/thirtyfile \
     --exclude='data/drive.db*' --exclude=data/secret.key --exclude=data/thumbs --exclude=data/tmp data storage

Files uploaded while the archive is being made may be in the archive without being in the saved database, or the other way round; either way the backup restores. To run it every night, put both commands in a script and add it to cron, for example 0 3 * * * /usr/local/bin/thirtyfile-backup.sh, and delete archives older than a few weeks in the same script.

With ThirtyFile stopped

docker stop thirtyfile
sudo tar czf thirtyfile-backup.tar.gz -C /srv/thirtyfile data storage
docker start thirtyfile

With Docker Compose, use docker compose stop and docker compose start in the folder of compose.yaml, and data storage relative to it.

Copies of a location's files

ThirtyFile can also copy the files of a storage location to another location itself, once with Copy everything to… under Control panel › Storage locations (see Copying everything to another location), or again and again with a backup of files. Replicas keep the files of a location on other locations as they are now, so they can still be read when it fails; they don't keep earlier states, so they don't replace a backup. It is a copy of files, not of the whole server: accounts, settings, share links and logs aren't in it, so keep backing up the data folder as above.

Backups of files to another location

A backup keeps snapshots of the spaces of one storage location on another: a NAS, another disk, an S3 bucket, an SFTP or FTP server. Each snapshot is a restore point: the spaces as they were at one moment, with their folders, files, trash and earlier versions. Older snapshots are kept as long as you choose, so a file deleted or changed by mistake can be brought back as it was. Only administrators see and manage backups.

Making a backup

Under Control panel › Backups, choose New backup… (or Back up… in the menu of a storage location), then:

• Back up and To: the location whose spaces are backed up, and where the backup goes. Locations in the same place, or one inside the other, are refused. One on the same disk or storage service is allowed, and the dialog says it doesn't survive a failure of that disk or service. A location can be backed up to several others, each with a backup of its own.
• Spaces: every space on the location, including spaces added to it later (the default), or the spaces you tick. Earlier versions and the trash are included unless you untick them.
• When:
  • Soon after changes: every change to a space (an upload, an edit, a rename, a move, a deletion, a change of access) is counted the moment it is saved. After a few seconds without more, a snapshot is made. One snapshot is made at a time; changes during it make one more after it. Files are protected a little after they change, not at the same moment: the Backups page shows how far behind a backup is.
  • On a schedule: every so many minutes or hours, every day at a time, or on some days of the week at a time, in a time zone you choose (the next three times are shown). A time that doesn't exist on the day summer time starts isn't run that day; a time that happens twice when it ends is run once. After the server was off, one snapshot catches up, not one for every time missed.
  • Both (the default): soon after changes, and on the schedule as well.
• Keeping and checking: how many days snapshots are kept (30 by default), how many of the newest are always kept (at least 1), after how many hours without a new snapshot administrators are told (48), how often the backup is read back and checked (every 7 days), and a speed limit for copying (none).

The first snapshot is made right away and copies everything; later ones copy only content the backup doesn't hold yet (each content is stored once, whichever snapshots hold it). Snapshots run in the background, one job at a time with copies and restores, and go on when the page is closed or the server restarts.

What a snapshot holds

• Content-store spaces (S3, SFTP, FTP): exactly what the spaces held when their list of files was made. Content deleted after that is kept from deletion until the snapshot has copied it.
• Folder spaces (the storage folder, Local folder locations): the folder is checked for changes other programs made (over SMB, say) first, then each file is read and checked not to have changed while it was read; one that changed is read again. A file another program keeps writing (a log, a mailbox) doesn't hold the snapshot up: after a few tries it is kept as it was last read whole, and the job's details list it as changed while copied (a file never read whole is left out of that snapshot and listed the same way; files of personal spaces are listed without their names). Each file is whole, but files other programs change at different moments aren't one instant together. Changes other programs make are seen by the next check for changes, not the moment they are made.
• Only a complete snapshot counts: its manifest and then its completion marker are written last, after all its content is copied and checked. A snapshot that can't copy everything stops, lists what it couldn't copy (never naming files of personal spaces), and deletes nothing.
• Files, not what people keep about them. A snapshot holds the spaces' folders and files and their earlier versions. Tags, smart folders, favorites, share links, access and activity are kept in ThirtyFile's database instead: restored items come back without them (they take the access of where they go). To keep those too, back up the data folder as well (see Backing up).

Following a backup

Each backup shows how it is doing: Protected (the newest snapshot holds every change), Changes waiting (and for how long), Backing up, Waiting for the location (it can't be reached; tried again every few minutes, and as soon as ThirtyFile sees it working again), Failing (the last snapshot stopped by an error; tried again every hour, or at once with Back up now), Overdue (no new snapshot for longer than the policy allows), Paused, and the time the spaces are protected as of. Administrators get a notification (and an email, when set up) once when a backup starts failing, waits for its location or is overdue, and once when it works again. Details lists the restore points and the jobs, with their errors. Finished jobs are kept for 90 days, at most the newest 1,000 of each backup, and the newest 20 whatever their age; a job that failed stays until it is resumed or cancelled.

Pause backing up stops making snapshots and keeps the ones made. Delete backup stops the backup and deletes its folder, with every snapshot in it, after you confirm; it can be paused, not cancelled. A location holding a backup can't be deleted until the backup is.

How much room it takes

Content several snapshots hold is stored once, so a backup takes about the size of the spaces, plus what changed or was deleted during the days snapshots are kept. After each new snapshot, the ones older than the policy keeps are deleted (never the newest ones), then content none of the others holds. Control panel › Storage usage shows what backups take on each location.

Restoring from a backup

Restoring checks existing content before reusing it. Missing or damaged content in the content store is recovered from the checked backup, preserving files that share it. The choice for a taken name is applied when each file is written, including names taken while the restore was reading its backup.

Select the backup and choose Restore…, then a restore point and a space. For company and team spaces you can pick a folder, or some of its items; a personal space is restored whole, only into its owner's personal space, since administrators don't see its files.

• Into a new folder (the default): everything goes into a folder of its own, Restored <space> <date> in your language (the dialog shows the name, and you can change it), in the space or another one you choose. Nothing there is replaced.
• Back where it was: into the space itself. Folders still there are used, missing ones are made again. Before it starts, the dialog says how many files have something where they go; choose to skip them, keep both (the restored one gets a number), or replace their content (what they have now is kept as an earlier version).
• Each file is checked against its SHA-256 as it comes back. Restored items take the permissions of where they go: the access the backup recorded isn't given again. Earlier versions stay in the backup. Files that can't be restored are listed, with why.

When the server itself is lost

A backup of files doesn't depend on the server that made it: each snapshot's manifest, on the backup's location, lists the spaces, folders and files and the content they had (.thirtyfile-backups/<id>/, see the layout). To restore on a new server after the old one and its database are gone:

1. Install ThirtyFile, and create the accounts whose personal spaces you want back, with the same user names.
2. Add the backup's location as a storage location with the same settings. Its .thirtyfile-location file names the old server's location, so a new one is refused: delete that file (in the folder, or at the top of the bucket or server folder) first, since the server it named is gone.
3. Under Control panel › Backups, choose Find backups on a location… and that location. The backups found are listed and checked in the background; their complete snapshots can then be restored from.

This brings files back, not the server: accounts, groups, access, share links, settings and logs come from a backup of the data folder (see Backing up), and the passwords saved for storage locations and single sign-on need secret.key (see below) or entering again. Keep the settings of the backup's location (keys, passwords) somewhere safe outside ThirtyFile, so you can add it again. Practise it now and then: restore a folder into a new folder, and check a few files.

Restoring

1. Stop ThirtyFile and unpack the archive where the folders belong: sudo tar xzf thirtyfile-backup.tar.gz -C /srv/thirtyfile (with Docker Compose, in the folder of compose.yaml).
2. If the archive was made without stopping, put the saved database in place: copy data/backups/drive-<date>.db to data/drive.db, and delete drive.db-wal and drive.db-shm.
3. Start the same version of ThirtyFile the backup came from (or a newer one), sign in, and open a few files, including one uploaded recently.

The database also holds the secret that signs sessions and unlocked share links: restored together with secret.key (see below), people stay signed in; without the key, everyone signs in again, and storage locations and single sign-on need their passwords entered again. Share links themselves keep working either way. Moving to a new server works the same way: install ThirtyFile there, restore the backup, and change the Site URL if the address changed.

Test a restore now and then on another computer: a backup is only as good as the last time it was restored.

The size of the database

The database (data/drive.db) and its log of recent writes (drive.db-wal) give back the room they no longer use when ThirtyFile starts and once a day: the log is emptied, and once a long read is over (a backup writing its list of files, say), it doesn't stay larger than 64 MB. New databases also give back the room of what was deleted, such as old logs, keeping a few megabytes for the writes to come. A database created by an earlier version of ThirtyFile keeps that room for its own later writes instead: to shrink it, and have it give room back from then on too, stop ThirtyFile and run docker compose run --rm thirtyfile compact (or docker run --rm -v …same volumes… ghcr.io/thirtyfile/thirtyfile compact). It rewrites the whole database, so it needs about as much free disk room as the database takes.

The key for saved passwords

The passwords and keys of storage locations, the single sign-on client secrets, the two-factor sign-in secrets and the secret that signs sessions and share links are saved in the database encrypted. The key is in data/secret.key (created on first start), or wherever THIRTYFILE_SECRET_KEY or THIRTYFILE_SECRET_KEY_FILE point. A copy of the database alone then gives none of them away.

• Back up the key separately, for example in a password manager, and leave it out of the regular backup (as in the command above). Without it, restoring works, but the storage locations and single sign-on need their passwords entered again, and everyone has to sign in again.
• To keep the key out of the data folder entirely, create one (openssl rand -hex 32) and give it with THIRTYFILE_SECRET_KEY_FILE (a Docker secret) or THIRTYFILE_SECRET_KEY. An existing key file's content can be used the same way.
• A new key, for example after a backup that included it went astray: stop ThirtyFile and run docker compose run --rm thirtyfile rotate-secret-key (or docker run --rm -v …same volumes… ghcr.io/thirtyfile/thirtyfile rotate-secret-key). It re-encrypts everything and replaces secret.key. With THIRTYFILE_SECRET_KEY, give the new key in THIRTYFILE_NEW_SECRET_KEY for that run, then switch the setting to it.

Keep backups safe

A backup contains your files, the accounts, and the logs with visitors' IP addresses; with secret.key in it, also everything needed to read the saved passwords. Treat it like a password: store it where only administrators can reach it.