#systemd
The Monitor That Hammered Devices Until the Whole Farm Went Down
A monitoring loop that runs every 30 seconds and tries to recover every device that fails a ping sounds useful, until one device hangs and gets hammered repeatedly until the whole bus goes down. This article walks through ruling out hardware, tracking what was accessing the devices, and the settings that fixed it.
The Service Is Running but Dead — systemd restart Cannot Help
A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.
One Codebase, Two Roles: Hosted Web, Worker at Home
The whole system used to run on one machine behind a home internet connection. When that machine went down, the team could not use anything. I split it into two roles using the same codebase — the web runs on an always-on host, while heavy work stays on the home machine, which pulls jobs itself, with an atomic job-claim table and an 18-step cutover plan with a rollback path.
One Tunnel Is the Only Way In—and a Single Point of Failure
The machine sits behind NAT with no ports open to the internet. Every way in runs through one cloudflared tunnel — web, SSH, and remote desktop. This article covers the four ingress routes I set up, how to read the signs when it goes down, and the trap that keeps the tunnel from starting at boot.
Tell a Broken Link from an ISP Outage Before Restarting Anything
A typical watchdog restarts NetworkManager when ping fails, which is wrong half the time because no internet access may mean the ISP is down, not that the adapter has failed. This article distinguishes three states using nmcli, then escalates from gentle recovery to PCI remove and rescan without rebooting the machine.
Every File Is Backed Up, but I Still Can't Rebuild the Server
The database dump is complete, the code tar is complete, and all the secrets are there, but put them on a blank machine and the site will not come up, because the things that actually make the server work were never in the backup files. This article lists what disappears with the machine and the recovery sequence I wrote for my future self.
Laravel Queues on Shared Hosting — Don't Share Redis with Cache
If you're enabling queues for Laravel on shared hosting and planning to separate the queue from cache with different REDIS_DB numbers, that won't help. maxmemory and eviction apply to the whole instance, not individual DBs. This article explains why I chose the database driver instead, with complete systemd units to replace Supervisor.
Per-user Redis on shared hosting—and the maxmemory bug that hung sites
Designing one Redis instance per user on shared hosting with a systemd template + cgroup — and lessons from a maxmemory bug where setting it above the cgroup limit caused intermittent WooCommerce hangs, plus the 80% rule and a guard script to prevent recurrence