#Runbook
Reads from a Replica Must Show How Many Minutes Old the Data Is
replication can report a lag of 0 seconds while the data is stuck at yesterday, because these two signals measure different things — how to set up a monitor that checks both Seconds_Behind_Master and business data freshness, catches anyone writing directly to the slave, and has another check watching the monitor itself
The Service Is Running but Dead — systemd restart Cannot Help
A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.
Every File Is Backed Up, but I Still Can't Rebuild the Server
The database dump is complete, the code tar is complete, and all the secrets are there, but put them on a blank machine and the site will not come up, because the things that actually make the server work were never in the backup files. This article lists what disappears with the machine and the recovery sequence I wrote for my future self.