TITLE=Backup
ROUTE=/infrastructure/backup
LOCALE=en
MODIFIED=1792352739
URL=https://wiki.teletypegames.org/infrastructure/backup


q│ Two chains: a Makefile-based backup for database dumps and Gitea repository

q│ mirrors, followed by an rsync-based mirror of the whole file tree to an

q│ off-site NAS. This has been the setup since 2026-10-16. Before that the

q│ second chain was restic; we retired it: the backup is simpler now, at the

q│ cost of having no versioning.


1Daily schedule

1══════════════


t┌───────┬─────────────────────────────────┬──────────────────────────────────┐

t│
h Time  
t│
h What                            
t│
h Tool                             
t│

t├───────┼─────────────────────────────────┼──────────────────────────────────┤

t│
0 04:00 
t│
0 Redmine (MariaDB), catalog and  
t│
0 
cmake backup
0 (cron,               
t│

t│
0       
t│
0 Gitea (MySQL) dumps → mirror    
t│
0 
cinfra/backup-scripts
0)            
t│

t│
0       
t│
0 every Gitea repository → delete 
t│
0                                  
t│

t│
0       
t│
0 dumps older than 7 days         
t│
0                                  
t│

t├───────┼─────────────────────────────────┼──────────────────────────────────┤

t│
0 05:00 
t│
0 Matomo, ResourceSpace and       
t│
0 
cvps-backup-sync
0 (systemd timer,  
t│

t│
0       
t│
0 Uptime Kuma dumps → Woodpecker  
t│
0 
cinfra/host-config
0)               
t│

t│
0       
t│
0 and Grafana SQLite copies →     
t│
0                                  
t│

t│
0       
t│
0 package list → full file mirror 
t│
0                                  
t│

t│
0       
t│
0 to the NAS                      
t│
0                                  
t│

t└───────┴─────────────────────────────────┴──────────────────────────────────┘

The mirror runs an hour later so it carries the fresh dumps and repository
mirrors too. The 04:00 chain does not sync on its own; everything reaches the
NAS with the 05:00 mirror.


1Chain 1: dumps and repository mirrors (Makefile)

1════════════════════════════════════════════════

Environment variables in the 
cMakefile
0:


t┌───────────────────────────┬────────────────────────────────────────────────┐

t│
h Variable                  
t│
h Meaning                                        
t│

t├───────────────────────────┼────────────────────────────────────────────────┤

t│
0 
cBACKUP_BASE
0               
t│
0 Root of all backups (default 
c/srv/backup
0)      
t│

t├───────────────────────────┼────────────────────────────────────────────────┤

t│
0 
cBACKUP_KEEP_DAYS
0          
t│
0 Retention for DB dumps (default 
c7
0)             
t│

t├───────────────────────────┼────────────────────────────────────────────────┤

t│
0 
cDATABASE_BACKUP_BASE
0      
t│
0 Where SQL dumps go                             
t│

t├───────────────────────────┼────────────────────────────────────────────────┤

t│
0 
cREPOSITORY_BACKUP_BASE
0    
t│
0 Where mirrored repos go                        
t│

t├───────────────────────────┼────────────────────────────────────────────────┤

t│
0 
cGITEA_SERVER
0, 
cGITEA_TOKEN
0 
t│
0 Gitea API for listing repositories; the token  
t│

t│
0                           
t│
0 comes from an 
c.envrc
0 file the Makefile reads   
t│

t│
0                           
t│
0 itself (
c-include
0), so it works under cron too  
t│

t└───────────────────────────┴────────────────────────────────────────────────┘


2Targets

2───────


p make backup                          # db_redmine -> db_catalog -> db_gitea »

p -> repos -> db_cleanup                                                       

p make db_redmine db_catalog db_gitea  # databases only                        

p make repos                           # update the git mirrors                

p make db_cleanup                      # drop expired dumps                    


2Database backups

2────────────────

Each dump runs inside the service's container. Dumps are timestamped into

c/db_backup
0 in the container, which is mounted to the host's

cDATABASE_BACKUP_BASE
0.


t┌────────────────────────┬─────────────────────────┬─────────────────────────┐

t│
h Service                
t│
h How                     
t│
h Note                    
t│

t├────────────────────────┼─────────────────────────┼─────────────────────────┤

t│
0 Redmine (MariaDB)      
t│
0 
cdatabase/mariadb_
0       
t│
0 
cMARIADB_USER
0,           
t│

t│
0                        
t│
0 
cbackup.sh
0               
t│
0 
cMARIADB_PASSWORD
0,       
t│

t│
0                        
t│
0                         
t│
0 
cMARIADB_DATABASE
0 in the 
t│

t│
0                        
t│
0                         
t│
0 container               
t│

t├────────────────────────┼─────────────────────────┼─────────────────────────┤

t│
0 Catalog (MySQL,        
t│
0 
cdatabase/mysql_backup.
0  
t│
0 
cMYSQL_ROOT_PASSWORD
0,    
t│

t│
0 
csoftwares
0)             
t│
0 
csh
0                      
t│
0 
cMYSQL_DATABASE
0 in the   
t│

t│
0                        
t│
0                         
t│
0 container               
t│

t├────────────────────────┼─────────────────────────┼─────────────────────────┤

t│
0 Gitea (MySQL, 
cgitea
0)   
t│
0 
cdb_gitea
0 in the         
t│
0 
bSince 2026-10-16.
0       
t│

t│
0                        
t│
0 Makefile: 
cmysqldump
0     
t│
0 Before that the Gitea   
t│

t│
0                        
t│
0 
c--single-transaction
0    
t│
0 database (users,        
t│

t│
0                        
t│
0 
c--routines --triggers
0   
t│
0 permissions, issues,    
t│

t│
0                        
t│
0                         
t│
0 PRs, OAuth apps,        
t│

t│
0                        
t│
0                         
t│
0 registry metadata) was  
t│

t│
0                        
t│
0                         
t│
0 not backed up anywhere. 
t│

t├────────────────────────┼─────────────────────────┼─────────────────────────┤

t│
0 Wiki (PostgreSQL)      
t│
0 
cdatabase/pg_backup.sh
0   
t│
0 
bRetired 2026-09-11
0 –    
t│

t│
0                        
t│
0                         
t│
0 the Wiki.js database is 
t│

t│
0                        
t│
0                         
t│
0 gone; the script is     
t│

t│
0                        
t│
0                         
t│
0 legacy.                 
t│

t└────────────────────────┴─────────────────────────┴─────────────────────────┘


2Repository backups

2──────────────────


cmake repos
0 lists every private and public repository from the Gitea API, then
per repo: 
cfetch --all --prune
0 + 
cpull
0 when it exists locally, 
cgit clone
0 when it
does not.


2Cleanup

2───────


cmake db_cleanup
0 groups dumps by database prefix and deletes files older than

cBACKUP_KEEP_DAYS
0. A safety check keeps at least one backup per database, even
when it is older than the retention window.


1Chain 2: file mirror (vps-backup-sync)

1══════════════════════════════════════

The 
c/usr/local/sbin/vps-backup-sync
0 script (source in the 
cinfra/host-config
repository) runs as root, started by 
cvps-backup-sync.timer
0.


bDumps at the start of the script:
0 Matomo (
cmariadb-dump
0), ResourceSpace
(
c--all-databases
0), Uptime Kuma (
csqlite3 .backup
0). For the live Woodpecker and
Grafana SQLite databases, 
csqlite3 .backup
0 writes a consistent copy next to the
file (
cwoodpecker-backup.sqlite
0, 
cgrafana-backup.db
0). The list of installed
packages goes to 
c/srv/backup/system/
0 (
cdpkg --get-selections
0, 
capt-mark

cshowmanual
0).


bWhat it mirrors:
0 
c/srv/stacks
0, 
c/srv/stacks_disabled
0, 
c/srv/backup
0,

c/srv/host-config
0, 
c/srv/presentation
0, 
c/srv/dockge
0, the host's 
c/etc
0, 
c/root
0,

c/home
0, 
c/usr/local/sbin
0, 
c/var/spool/cron
0 and 
c/var/www
0, plus the ResourceSpace
filestore and Matomo Docker volumes. About 27 GB in total.


bWhat it skips:
0 live database directories (the dumps stand in for them, since a
running DB's files are not consistent), 
cnode_modules
0, 
ctmp
0, 
clog
0, Gitea search
indexes, caches.


bHow:
0 
crsync -aH --numeric-ids --relative --delete
0 into a mode-700 directory on
the NAS, with absolute paths. A non-root user receives on the NAS, so owner,
permissions and device data are stored in xattrs (
c-M--fake-super
0). Copy back
with the same option, otherwise every file ends up owned by the NAS user.


bNo versioning.
0 The NAS always holds the latest state only. If something is
deleted or corrupted on the server, it is deleted or corrupted on the NAS the
next night too. For databases, the 7-day dump rotation gives some history.


1Off-site mirror

1═══════════════

The target is an external Synology NAS (DS212j, DSM 6.2, ext4). It only
accepts 
cssh-rsa
0 keys, has no SFTP subsystem and cannot do Btrfs snapshots, so
NAS-side versioning would only be possible with hardlinked copies.


bOpen question:
0 the mirror is writable from the server, so a compromised host
could delete the backup too. Secrets (
c/root
0, 
c/etc/shadow
0, 
c.env
0 files,
certificate keys) sit unencrypted on the NAS.


1Restoring to a new server

1═════════════════════════

The exact commands are in the 
cinfra/host-config
0 README, section „Mentés és
visszaállítás”. In short:

1. Copy the whole tree back as root, with 
c-M--fake-super
0, into a temporary
   directory.
2. Install packages from 
c/srv/backup/system/apt-manual.txt
0, then put 
c/etc
0 back
   selectively and restore 
c/srv
0, 
c/home
0, 
c/root
0, 
c/var/www
0, 
c/var/spool/cron
0,
   
c/usr/local/sbin
0.
3. 
csudo /srv/host-config/install.sh
0 for the host's own files.
4. Start the stacks with empty databases, then load the latest dumps
   (
credmine_production
0, 
csoftwares
0, 
cgitea
0, 
cmatomo
0, 
cresourcespace
0).
5. SQLite: move the 
c*-backup.*
0 copies into place with the stack stopped.
6. Docker volumes: restore the ResourceSpace filestore and Matomo volume
   contents.


1File structure

1══════════════

The backup repository (
cinfra/backup-scripts
0):


p .                                                                            

p ├── Makefile                # every chain-1 task                             

p └── database/                                                                

p     ├── mariadb_backup.sh                                                    

p     ├── mysql_backup.sh                                                      

p     └── pg_backup.sh        # legacy, no longer runs                         

Chain 2 lives in the 
cinfra/host-config
0 repository:

cfiles/usr/local/sbin/vps-backup-sync
0,

cfiles/etc/systemd/system/vps-backup-sync.{service,timer}
0,

cfiles/etc/logrotate.d/vps-backup-sync
0. Install with 
csudo ./install.sh
0.

On the server (
cBACKUP_BASE=/srv/backup
0):


p /srv/backup/                                                                 

p ├── database/                     # dumps, 7 days                            

p │   ├── gitea-2026-10-16-02-00.sql                                           

p │   ├── redmine_production-2026-10-16-02-00.sql                              

p │   ├── softwares-2026-10-16-02-00.sql                                       

p │   ├── matomo-2026-10-16-05-03.sql                                          

p │   └── resourcespace-2026-10-16-05-03.sql                                   

p ├── repositories/                 # mirrors of every Gitea repository        

p ├── system/                       # package list (dpkg, apt-mark)            

p └── archive-2026-10-15/           # tar.gz archives of volumes removed durin»

p g cleanup                                                                    


1Verification

1════════════

• The mirror log on the host (
c/var/log/vps-backup-sync.log
0): every step ends
  with an 
cOK
0 or 
cHIBA
0 (error) line.
• After an error-free run the script sends an Uptime Kuma push; if none
  arrives for 26 hours, an alert goes to the Discord Alert channel.
• Grafana: the 
bVPS: Backup
0 dashboard shows the last run state, dump age and
  size, mirror size and the backup log lines. The script writes the metrics
  for node-exporter's textfile collector
  (
c/srv/stacks/observability/textfile/vps_backup.prom
0); the log lines are in
  Loki (
c{service_name="cron-jobs", program="vps-backup-sync"}
0). Grafana:
  https://grafana.teletypegames.org (sign in with your Gitea account).
• If the newest dump of a database gets old, Grafana's 
bDatabase dump stale
  alert goes to Discord.
• Every database must have a fresh file for every day in the dump directory;
  if a prefix disappears or stops, the chain is stuck.


1Repositories

1════════════

• https://git.teletypegames.org/infra/backup-scripts
• https://git.teletypegames.org/infra/host-config