Backups Succeeded Nightly, but Offsite Uploads Failed for 3 Weeks

I needed to save a snapshot of a server to my own machine in case I had to rebuild it. When I checked the date of the latest files in offsite storage, I found that the folder for that month contained just one set, even though cron had been configured to upload every night for a long time. Looking further back, I discovered that offsite uploads had failed for every file for three straight weeks, without anything in the system raising an alert even once.

What hurt more was that local rotation was set to 14 days. By the time I noticed, the backups from the first eight nights of failed uploads had already been deleted from the machine too. Neither location had a copy left. I wrote this article as a postmortem: where a system that seemed complete could fall short, and how to make it raise an alert next time before the backups disappear.

All usernames, paths, bucket names, and IP addresses in this article are examples. The structure and symptoms are real.

What a system I should be able to trust looks like

The setup was straightforward, and I maintain that there was nothing wrong with its structure. One script ran at two in the morning, produced three files, and pushed them to the destination object storage.

The three files were a full database dump, a tar archive of the secrets directory and .env files, and a tar archive of the git bare repo and directories outside git. What I liked about this design was that customer documents were stored as blobs in the database, rather than scattered across files. That meant a single database dump covered the entire document archive, with no separate layer of file backups to manage.

# cron ของ user (3 บรรทัด ไม่มี cron ของ root เลย)
0 2 * * *   /opt/appsrv/app/scripts/backup-all.sh   >> /opt/appsrv/.private/backups/backup.log 2>&1
*/5 * * * * /opt/appsrv/app/scripts/healthcheck.sh  >> /opt/appsrv/.private/backups/health.log 2>&1
* * * * *   cd /opt/appsrv/app && /usr/bin/php artisan schedule:run >> /dev/null 2>&1

A healthcheck ran every five minutes, checked six things, and sent an alert by email if it found a problem. On paper, this was more thorough than most machines I had come across. Yet the backups still disappeared.

Gap one — 2>/dev/null discarded the most essential messages

The line responsible was very short.

# สาระของบรรทัดที่ผิด
if rclone copy "$f" "r2:app-backup/$YEAR/$MONTH/" 2>/dev/null; then
    log "uploaded $(basename "$f")"
else
    log "upload failed: $(basename "$f")"
fi

The author's intention (that author being me on another day) was to keep the log clean, without rclone progress bars and minor warnings cluttering it. The result was that the log could say only upload failed, and nothing more. There was no clue why it had failed. When I went back through three weeks of logs, every line looked the same.

The pattern I use instead captures stderr first, then decides whether to print it. That keeps logs clean during normal operation and preserves the full reason when something breaks.

upload() {
    local f="$1" err rc
    err="$(rclone copy "$f" "r2:app-backup/$YEAR/$MONTH/" 2>&1 >/dev/null)"
    rc=$?
    if [ "$rc" -ne 0 ]; then
        log "UPLOAD FAIL rc=$rc $(basename "$f")"
        printf '%s\n' "$err" | tail -n 20 | while read -r l; do log "    | $l"; done
        return 1
    fi
    log "uploaded $(basename "$f")"
}

The order in 2>&1 >/dev/null matters. The first redirection sends stderr wherever stdout currently goes (the pipe that captures output into the variable). The second then discards the actual stdout. Reversing them has the opposite effect: it swallows stderr too. At a glance, the reversed version looks very similar.

Gap two — a healthcheck that never touched the destination

The second gap hurt more than the first, because this was the layer I had put in place specifically to catch problems like this. The healthcheck checked six things: the web page returned 200, the database accepted connections, two services were still running, the process handling external access was still there, disk usage was no higher than 90%, and this final check.

# ข้อที่หกของ healthcheck.sh — เช็กแค่ไฟล์ในเครื่อง
newest=$(find /opt/appsrv/.private/backups -name '*.sql.gz' -mmin -2160 | head -n1)
[ -z "$newest" ] && problem "backup ล่าสุดเก่ากว่า 36 ชั่วโมง"

This condition was true every day, because the local dump really did succeed every night. The failed stage was the next one, which not a single line checked. The entire monitoring system therefore reported green for all three weeks.

My conclusion was that a healthcheck that checks only the source is not a healthcheck for the backup system. It merely checks that the dump command still runs, which is a different question from whether I actually have a backup.

A 14-day rotation is a countdown, not a retention policy

When I designed this, 14 days locally and 90 days at the destination sounded very reasonable. Local storage kept enough for urgent restores; offsite storage kept backups longer in case of a major incident. But once the second stage died, those 14 days immediately changed meaning. They became a countdown of how much time I had before backups disappeared permanently.

Just subtract the two numbers: three weeks minus fourteen days leaves one week of backups missing from both locations. The backups from those first eight nights no longer existed anywhere in the world. I stopped treating the rotation interval as a way to save space and started seeing it as a time budget for detecting failures. If my checking interval exceeds that budget, I am effectively choosing to find out after the backups have disappeared.

There are two ways to fix this. Either one is enough, but doing both is better. The first is to retain local backups longer than the interval at which a human will actually take a look. The second is to automate destination checks, which is cheaper and addresses the problem more directly.

So what actually caused the failure?

Here I will stick to what I could prove, without drawing conclusions beyond the evidence. By the time I investigated, rclone was working again, without anyone having changed the script. The strongest explanation was that the API token used for credentials had an IP allowlist, while this machine sat behind a residential internet connection whose public IP changed periodically. When the IP changed, the token was rejected across the board.

In this case, the symptom would be a provider-specific error code, rather than a normal S3 error. And that was exactly the message 2>/dev/null had swallowed for three weeks.

# ตรวจไอพีขาออกจริงของเครื่อง แล้วเทียบกับ allowlist ของ token
curl -4 -s ifconfig.me
# 203.0.113.44

# ยืนยัน token ว่ายังใช้ได้ — account token ต้องยิงที่ /accounts/{id}/tokens/verify
curl -s -H "Authorization: Bearer $CF_TOKEN" \
     "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/tokens/verify" | head -c 300

Traps when setting up rclone with S3-compatible object storage

While investigating this case, I found two more traps that could send someone setting this up for the first time down a long detour. I am recording them here for anyone about to do it.

The first is credentials. This storage service lets you create an Access Key pair through its web interface, but that was not the pair this system actually used. It used a pair derived directly from the API token: the Access Key was the token's id, and the Secret was the sha256 hash of the token itself. Using the wrong pair produces AccessDenied for both ListBuckets and ListObjects, which reads like insufficient permissions rather than the wrong keys.

# ได้ secret สำหรับ S3 จากค่า token
printf '%s' "$CF_TOKEN" | sha256sum | cut -d' ' -f1

# /home/appsrv/.config/rclone/rclone.conf  (chmod 600)
[r2]
type = s3
provider = Cloudflare
access_key_id = <TOKEN_ID>
secret_access_key = <SHA256_OF_TOKEN>
endpoint = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
no_check_bucket = true

The second is the rclone version. The version supplied by apt on the stable distro was 1.60, which was too old for this destination. The result was a puzzling 501 response. I had to download a binary of version 1.74 or later and place it in /usr/local/bin/ myself.

rclone version | head -n1
# rclone v1.74.4

which -a rclone
# /usr/local/bin/rclone
# /usr/bin/rclone        <- ตัวของ apt ที่เก่าเกิน อย่าให้ cron ไปเจอตัวนี้ก่อน

Another point to know is that rclone's config lives at ~/.config/rclone/rclone.conf under the user running cron, not in the project directory. That means this file is not included in the backup that archives the project's secrets directory with tar. When restoring to a new machine, I have to recreate it manually. I will write a separate article about things the backup does not capture.

Checking the destination in a way I can trust

The key to checking is knowing which "time" I am reading. This fooled me once.

rclone lsl displays the mtime attached to the object, which is the source file's mtime when it was uploaded. By contrast, LastModified obtained directly from the S3 API is the time the object was actually written at the destination. These are different values, and when investigating "when exactly did this file get uploaded?", I need the latter.

# ตรวจว่าไฟล์ถูกเขียนขึ้นปลายทางจริงเมื่อไหร่ (ไม่ใช่ mtime ของต้นทาง)
import boto3, os, datetime

s3 = boto3.client(
    "s3",
    endpoint_url=f"https://{os.environ['ACCOUNT_ID']}.r2.cloudflarestorage.com",
    aws_access_key_id=os.environ["S3_KEY"],
    aws_secret_access_key=os.environ["S3_SECRET"],
)

now = datetime.datetime.now(datetime.timezone.utc)
resp = s3.list_objects_v2(Bucket="app-backup", Prefix="2569/09/")
for o in sorted(resp.get("Contents", []), key=lambda x: x["LastModified"]):
    age_h = (now - o["LastModified"]).total_seconds() / 3600
    print(f"{o['LastModified']:%Y-%m-%d %H:%M}  {o['Size']:>12,}  {age_h:6.1f}h  {o['Key']}")

The results were clear. Not a single file that should have been there from those three weeks was present. And the only latest set had a LastModified time of nine fifty in the evening, rather than the two in the morning when cron ran. That meant someone had uploaded that set manually later; it was not the work of the automated system. Looking only at rclone lsl, I would never have seen this distinction, because it would neatly display two in the morning from the source files' mtime.

Fixing the whole path — let the destination decide

I changed three things, in order of importance.

1. Stop swallowing stderr from the stage that talks to the destination. Capture the messages and print them on failure along with the exit code, so I can tell whether the problem is permissions, the network, or the version.
2. Have backup-all.sh verify immediately after upload that the files just sent actually exist at the destination and match the source sizes. A mismatch is a failure, not a warning.
3. Add a seventh healthcheck to ask the destination how old its newest object is. Once it exceeds 36 hours, send an alert just as for a website outage.

The post-upload verification can be written briefly like this, using rclone size to compare bytes directly without relying on another tool.

verify_remote() {
    local f="$1" name local_size remote_size
    name="$(basename "$f")"
    local_size=$(stat -c %s "$f")
    remote_size=$(rclone size --json "r2:app-backup/$YEAR/$MONTH/$name" 2>/dev/null \
                  | sed -n 's/.*"bytes":\([0-9]*\).*/\1/p')
    if [ "$local_size" != "$remote_size" ]; then
        log "VERIFY FAIL $name local=$local_size remote=${remote_size:-none}"
        return 1
    fi
    log "verified $name ($local_size bytes)"
}

And here is the seventh healthcheck, the line that should have been there from day one.

# ข้อ 7 — ของที่ปลายทางต้องใหม่พอ ไม่ใช่แค่ของในเครื่อง
newest_remote=$(rclone lsjson --recursive "r2:app-backup/$(date +%Y)/$(date +%m)/" 2>&1 \
                | sed -n 's/.*"ModTime":"\([^"]*\)".*/\1/p' | sort | tail -n1)
if [ -z "$newest_remote" ]; then
    problem "อ่านรายการที่ปลายทางไม่ได้เลย"
else
    age=$(( ( $(date +%s) - $(date -d "$newest_remote" +%s) ) / 3600 ))
    [ "$age" -gt 36 ] && problem "ไฟล์ใหม่สุดที่ปลายทางอายุ ${age} ชั่วโมง"
fi

What to check before trusting a backup

Finally, I made a short checklist for myself. Anyone is welcome to use it. Every item answers the same question: "Which side provides the evidence that the backups actually exist?"

# 1) ปลายทางมีของกี่ก้อน ใหม่สุดวันไหน
rclone lsl r2:app-backup/2569/09/ | sort -k2 | tail -n5

# 2) ไฟล์ที่บีบไว้เปิดออกได้จริงไหม (แค่มีไฟล์ไม่พอ)
gzip -t /opt/appsrv/.private/backups/appsrv_main-*.sql.gz && echo "gzip ok"
tar -tzf /opt/appsrv/.private/backups/code-*.tar.gz > /dev/null && echo "tar ok"

# 3) dump มีเนื้อจริงไม่ใช่ไฟล์เปล่าที่บีบมาสวย ๆ
zcat /opt/appsrv/.private/backups/appsrv_main-*.sql.gz | grep -c '^INSERT INTO'

# 4) ขนาดสองฝั่งตรงกันเป็นไบต์
stat -c '%s %n' /opt/appsrv/.private/backups/*.gz
rclone size r2:app-backup/2569/09/

Items two and three matter more than they might seem. A dump that fails halfway still produces a .sql.gz file; it is just missing half its contents. A reassuringly large file does not mean the contents are complete. I therefore like to record the number of INSERT lines as well, for comparison with the previous night. An unusually sharp drop means something is wrong.

What I took away from this was no new technique. It was accepting that any stage whose outcome goes unverified will eventually fail silently, and stay silent until someone happens to look. If your backup system still checks only local files, try adding a line that queries the destination. It takes less than ten minutes, but it is the difference between finding out within a day and finding out a week after the backups have disappeared. If I have left anything out or made a typo, I will come back and correct it later.