Every File Is Backed Up, but I Still Can't Rebuild the Server

I was saving a snapshot of a server to my own machine. After checking the files, I idly asked myself how many minutes it would take for the site to come up if I put these three files on a blank machine with a fresh OS. When I actually walked through each step in my head, I found that the answer was it would not come up, even with one hundred percent of the data intact.

The backup holds the database, code, and secrets. More than ten other things that actually make that machine work had never been saved at all. They lived in files scattered across /etc, across ~/.config, and in tables nobody had written down. So I collected them directly from the live machine over SSH and wrote a document for rebuilding the entire system in one go. This article is the substance of that document.

The usernames, domains, paths, and reference IDs in this article are examples. The structure and pitfalls are all real.

First — know what is "not" in the backup

The first thing I need to be clear about is scope. Some things are deliberately left out because they can be rebuilt, which is fine, but I need to know what they are and how long rebuilding them takes.

The item on the right that worries me most is not the largest one. It is the python job's virtualenv, because rebuilding it means downloading the whole set of large libraries again. If the connection is bad that day or the versions on the index have changed, a job that should take ten minutes becomes half a day. If you have something like this, I recommend pinning all the versions in a file. Do not leave them floating.

What disappears with the machine, not with the files

I collected these items one by one from the live machine, because the old documentation I had written when setting it up was completely out of date. The resulting list fell into four groups.

The first group is the Apache vhost and its connection to php-fpm. The file lives in /etc/apache2/sites-enabled/, so it could never be included in the project's tar.

# /etc/apache2/sites-enabled/app.conf  (สาระที่ต้องมี)
ServerName app.local
DocumentRoot /opt/appsrv/app/public

SSLCertificateFile /etc/ssl/app/cert.pem      # self-signed — ปลาย TLS จริงอยู่ที่ขอบนอก

<Directory /opt/appsrv/app/public>
    AllowOverride All            # ให้ .htaccess ทำงาน เหมือนตอนอยู่ shared host
    Require all granted
    Options -Indexes +FollowSymLinks
</Directory>

<FilesMatch \.php$>
    SetHandler "proxy:unix:/run/php/php8.4-fpm.sock|fcgi://localhost"
</FilesMatch>

That single AllowOverride All line matters more than it looks, because this app was moved from a shared host where .htaccess was active. If I forget this line when restoring to a new machine, the home page loads normally, but every other URL immediately returns 404. That is a very difficult symptom to track down unless it occurs to me at the start.

Two levels of systemd — and the line that takes the whole site down if forgotten

The second group is services, which exist at two levels. System services live in /etc/systemd/system/, and user services live in ~/.config/systemd/user/. While checking them, I found that the most important service on the entire machine happened to be a user service: the tunnel that provides the only way in from outside.

# ~/.config/systemd/user/tunnel.service
[Unit]
Description=Outbound tunnel (ทางเข้าเดียวจากภายนอก)
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/opt/appsrv/.private/tunnel.env
ExecStart=/home/appsrv/.local/bin/tunneld tunnel run
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

systemctl --user enable --now tunnel
loginctl enable-linger appsrv        # <- บรรทัดที่ขาดไม่ได้

At the system level, three or four services need to be enabled again. The one worth recording is the queue worker, because its arguments were chosen deliberately and are not defaults.

# /etc/systemd/system/app-queue.service — สาระของ ExecStart
ExecStart=/usr/bin/php /opt/appsrv/app/artisan queue:work redis \
          --sleep=3 --tries=3 --timeout=240 --max-jobs=200 --max-time=3600
User=appsrv
Restart=always

--max-time=3600 tells the worker to exit every hour and let systemd restart it, preventing memory leaks from accumulating. The related detail is that the healthcheck must know about this too. Otherwise, it can catch the service just as it is restarting and report an outage. On this machine, I addressed that by checking again eight seconds later before deciding.

Three cron entries nobody thinks about

The third group is the smallest and the easiest to forget: crontab -l for a single user, three entries, and no cron jobs for root at all. Without a record, the site will come up beautifully after recovery and everything will look normal, until three days later someone asks why there are no backups.

0 2 * * *   /opt/appsrv/app/scripts/backup-all.sh   >> /opt/appsrv/.private/backups/backup.log 2>&1
*/5 * * * * /opt/appsrv/app/scripts/healthcheck.sh  >> /opt/appsrv/.private/backups/health.log 2>&1
* * * * *   cd /opt/appsrv/app && /usr/bin/php artisan schedule:run >> /dev/null 2>&1

The same applies to the service list. I added systemctl list-unit-files --state=enabled and systemctl --user list-unit-files --state=enabled to that same file. It is only a few kilobytes of text, but during recovery it is the difference between knowing and guessing.

External access, and a prohibition that needs large print

The fourth group is access. This machine has no public IP of its own. Everything goes out through a single tunnel, with the provider's edge handling incoming requests. This tunnel's ingress settings are centrally managed, which means the daemon on the machine fetches its own config at startup instead of reading a local file.

# หน้าตาของ ingress (แก้ผ่าน API หรือหน้าเว็บ ไม่ได้อยู่บนเครื่อง)
app.example.com       -> https://localhost:443
hook.example.com      -> http://localhost:8787
ssh.example.com       -> ssh://localhost:2244
(catch-all)           -> http_status:404

The advantage is that restoring to a new machine requires no ingress changes at all. I just put the original tunnel credential in place and it connects as before. But that very advantage becomes the biggest recovery trap.

As for sshd, it uses a different port and has all password authentication disabled. This needs to be recorded because if I forget it during recovery, I will not be able to access the machine until I sit down at its console.

# /etc/ssh/sshd_config — สาระ 4 บรรทัด
Port 2244
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

rclone.conf and secrets outside the project folder

I found this just as I was about to wrap up. The backup included the project's entire secrets folder and .env file, but the rclone config used to upload files to off-machine storage lived in the user's ~/.config/rclone/rclone.conf, outside the project folder.

That means if the machine really disappears, I can restore everything, only to discover that the tool meant to store the next backup has no config to use. I would have to retrieve a token and convert it into an S3 credential again myself, a procedure that is fairly hard to remember.

# ต้องทำเองหลังกู้ — ไฟล์นี้ไม่มีใน backup
mkdir -p ~/.config/rclone
cat > ~/.config/rclone/rclone.conf <<'EOF'
[r2]
type = s3
provider = Cloudflare
access_key_id = <TOKEN_ID>
secret_access_key = <SHA256_OF_TOKEN>
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
no_check_bucket = true
EOF
chmod 600 ~/.config/rclone/rclone.conf

# และอย่าใช้ rclone ของ apt — เก่าเกินไป ปลายทางจะฟ้อง 501
/usr/local/bin/rclone version | head -n1

My long-term fix was to have the backup script copy rclone.conf into the secrets folder before every tar, so it automatically goes into the third bundle. If you are concerned about bundling secrets with other secrets, that is your call, but that bundle already contains .env, which holds much more sensitive material.

The actual recovery sequence, and the line never to type

Now I can put it all in order. The important point is to always restore secrets before rebuilding anything regenerable, because several tools read .env while they run.

1. Install the OS, then create the user with the original uid, disable sleep, and set the time correctly.
2. Install the full stack — web server, database, cache, and php at the versions actually in use — then enable the required modules.
3. Restore the code from the second tar, put the bare repo back in its original location, then clone it into a working tree.
4. Restore the secrets from the third tar, then restore the original permissions on every file.
5. Create the database and user, then import the first dump, reading the password from the newly restored .env.
6. Rebuild what can be regenerated — composer, npm build, storage link, and clear every type of cache.
7. Put the vhost in place, create the certificate, enable the site, then reload the web server.
8. Put the units in place at both levels, enable all of them, then run enable-linger for the user.
9. Add the three cron entries, configure sshd, then reconnect the tunnel — after confirming the old machine is fully shut down.

In step six, there is one line I must never type, and it is exactly the line my hands will type out of habit.

cd /opt/appsrv/app
composer install --no-dev -o
npm install && npm run build
php artisan config:clear && php artisan route:clear && php artisan storage:link
# php artisan key:generate    <- ห้าม ถ้ากู้ .env เดิมมาแล้ว

Verify success with commands that return numbers

I dislike the method of "open a browser and it feels normal," because it easily conceals problems that emerge later. So I put together four lines of checks that return only numbers.

# 1) แอปตอบจากในเครื่องเอง ตัดตัวแปรเรื่องเน็ตกับ tunnel ออกก่อน
curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: app.example.com' https://127.0.0.1/login
# ต้องได้ 200

# 2) แล้วค่อยตรวจจากข้างนอกจริง ๆ ผ่าน tunnel
curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/
# ต้องได้ 302

# 3) ชุดเฝ้าระวังของเครื่องเองต้องผ่านทุกข้อ
/opt/appsrv/app/scripts/healthcheck.sh

# 4) ที่เก็บนอกเครื่องยังคุยได้ (ข้อนี้คนลืมบ่อยสุด)
/usr/local/bin/rclone lsd r2:app-backup

Old documentation on the machine is dangerous

I will finish with what I think was the most important part of this work. At the start, I intended to read the old documentation on the machine and update it. Ten lines in, I had to stop, because every number in it was wrong. The recorded route count was off by a factor of two, the migration count did not match, the module list was missing nearly half its entries, and some sections said "not finished yet" about work that had been done long ago.

The file named as a guide to setting up a new machine was unusable too, not because it was wrong, but because it was not enough. It covered only cloning the code and installing dependencies, the easiest parts. Not a single line mentioned vhost, unit, linger, cron, ingress, or sshd.

So I set myself a rule: recovery documentation must be collected directly from the live machine. Not a single line may be copied from old documentation. If I am going to write down the database name, I have to run SHOW DATABASES and see it for myself. If I am going to list the services, I have to run list-unit-files and paste the output. Documentation written from memory silently rots every time someone changes the system without going back to update the documentation, which is every time.

The cheaper approach is to make it collect itself automatically. Write a short script that sends the output of the entire set of inspection commands to one file, then have cron run it weekly and save it in a folder the backup already covers.

#!/bin/bash
# scripts/snapshot-machine-state.sh — รันสัปดาห์ละครั้ง เก็บลงที่ที่ backup กวาดถึง
out=/opt/appsrv/.private/machine-state.txt
{
  echo "=== เก็บเมื่อ $(date -Is) ==="
  echo "--- crontab ---";        crontab -l
  echo "--- units (system) ---"; systemctl list-unit-files --state=enabled --no-pager
  echo "--- units (user) ---";   systemctl --user list-unit-files --state=enabled --no-pager
  echo "--- linger ---";         loginctl show-user "$USER" -p Linger
  echo "--- vhost ---";          cat /etc/apache2/sites-enabled/*.conf
  echo "--- php ---";            php -v | head -n1; php -m | tr '\n' ' '
  echo "--- ports ---";          ss -tlnp 2>/dev/null
  echo "--- versions ---";       mysql -V; /usr/local/bin/rclone version | head -n1
} > "$out"
chmod 600 "$out"

This file is less than a hundred kilobytes, but it is the difference between finishing a server recovery in two hours and guessing all day. Feel free to add any other commands you need. I have not yet included firewall and sysctl settings in mine; I will come back and add them later.