#Disaster Recovery
A 25-Hour Power Outage Left Scheduled Jobs Missing
The machine running my automated jobs lost power three times in three days, with the longest outage lasting 25 hours. When it came back, I found that jobs scheduled directly through Task Scheduler had skipped their runs entirely, while jobs managed by a dispatcher that checked the time itself had all caught up. This article covers the setting to enable, what to avoid during recovery, and the traps that led me to misread the status.
Every File Is Backed Up, but I Still Can't Rebuild the Server
The database dump is complete, the code tar is complete, and all the secrets are there, but put them on a blank machine and the site will not come up, because the things that actually make the server work were never in the backup files. This article lists what disappears with the machine and the recovery sequence I wrote for my future self.
Backups Succeeded Nightly, but Offsite Uploads Failed for 3 Weeks
cron ran every night, local files were complete every day, and the log said done, but offsite storage had not changed for 3 weeks. The first 8 nights of backups were already gone for good because local rotation had deleted them first. This article examines the two gaps that let the failure stay silent for so long.
Recover MariaDB Redo Log Corruption with force_recovery and Reinit
MariaDB would not start after an unclean shutdown — a corrupted InnoDB redo log took the mysql.global_priv system table down with it. This is the recovery sequence I actually used, starting with innodb_force_recovery to make the data readable, then dumping each db before reinitializing the whole system, with one rule never to raise force_recovery higher than necessary.
Set DirectAdmin Admin Backup Cron from the CLI Without the Panel
CMD_ADMIN_BACKUP takes five steps to set up a cron job, but DirectAdmin actually just appends one line to backup_crons.list and lets dataskq pick it up. Here is how to write directly to the file from the CLI, including common traps with when=now, base64, and append_to_path.
Recovering Old Website Images with Wayback Machine CDX API + Python
A website over a decade old had broken images throughout because it hotlinked to a dead CDN — I wrote a Python script to list images from the Wayback Machine CDX API and restore them to their original paths without touching a single character of HTML, with the full script I actually used.
Off-Site Backups with Pull Architecture + rclone/rsync
Add off-site Layer 3 to complete 3-2-1 with pull architecture (the destination host fetches the data), rclone + rsync, and lessons from real pitfalls involving NAT, RSA host keys, cwRsync, and PID recycling.
Rebuilding DirectAdmin Backups with NFS and POSIX ACL in Production
A record of rebuilding backups on a DirectAdmin server after migrating to a new machine — from FTP push that broke at three layers to an NFSv4 hard mount + POSIX ACL, with kernel-level lessons learned along the way.