#Incident Response
The Monitor That Hammered Devices Until the Whole Farm Went Down
A monitoring loop that runs every 30 seconds and tries to recover every device that fails a ping sounds useful, until one device hangs and gets hammered repeatedly until the whole bus goes down. This article walks through ruling out hardware, tracking what was accessing the devices, and the settings that fixed it.
Reads from a Replica Must Show How Many Minutes Old the Data Is
replication can report a lag of 0 seconds while the data is stuck at yesterday, because these two signals measure different things — how to set up a monitor that checks both Seconds_Behind_Master and business data freshness, catches anyone writing directly to the slave, and has another check watching the monitor itself
The Service Is Running but Dead — systemd restart Cannot Help
A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.
Mailbox Compromised or Just Password Guessing? Prove It First
Telling a customer their account has been compromised is an allegation that needs decisive evidence first. This article covers what counts as decisive evidence, two cases with opposite outcomes, and the case I nearly reported incorrectly.
Shared Neutrals Across Transformers: Why Phase Balancing Is Not Enough
A case involving two transformers with interconnected neutrals and houses tapped into one side's main feeder, leaving phase A 30-40 A higher even on days off, plus a meter-reading trap that can suggest a 200 A difference if the CT ratio is not verified first, and a repair sequence that must be followed in order.
Stolen Mailbox Password Sent 1,786 Spam Emails Unseen by CSF/LFD
A real case of a stolen mailbox password used to send 50-75 spam emails a day, below a limit of 200, for 4 straight weeks without a single response from CSF/LFD/BFM because they only count failed authentication attempts, with the decisive question of whether an account was compromised or merely under attack, exim forensic commands, and a blocklist removal playbook
nginx heap overflow spans 15 years—version checks aren't enough
CVE-2026-42533 affects nginx versions 0.9.6 through 1.31.2, spanning 15 years, but actual exposure depends on the config, not the version. The problem is that news reports describe narrower conditions than the real ones. Checking against those reports can return "safe" even when the config still meets the conditions.
Read-only Debian shared-host security audit — avoid banning yourself
Audit the security of a Debian server running DirectAdmin in read-only mode first — but the first trap is that scan traffic itself triggers CSF's brute-force rules and gets your own IP banned. This article lays out a safe audit sequence + areas to check in 2024-2026