#Alerting
The NVR Kept Beeping, Then Fell Silent When the Disks Failed
The NVR's all-day beeping was not videoloss as everyone thought. It was ipconflict from cameras using DHCP with conflicting IP addresses. Someone got annoyed and muted the whole unit, so when both disks failed together, there was not a single beep. This article fixes the policy one event at a time through ISAPI.
Reads from a Replica Must Show How Many Minutes Old the Data Is
replication can report a lag of 0 seconds while the data is stuck at yesterday, because these two signals measure different things — how to set up a monitor that checks both Seconds_Behind_Master and business data freshness, catches anyone writing directly to the slave, and has another check watching the monitor itself
The Service Is Running but Dead — systemd restart Cannot Help
A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.
Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read
In one day, 28 events became 56 messages, and 57 percent of all events were repeat alerts every 30 minutes. This article covers recipient tiers, merging overlapping cycles, state that survives restarts, and why you must not add keepalive to a chat app session.
Tell a Broken Link from an ISP Outage Before Restarting Anything
A typical watchdog restarts NetworkManager when ping fails, which is wrong half the time because no internet access may mean the ISP is down, not that the adapter has failed. This article distinguishes three states using nmcli, then escalates from gentle recovery to PCI remove and rescan without rebooting the machine.
A Telegram Bot for DA Server Monitoring with a Small Blast Radius
A guide to setting up a Telegram bot as a lightweight control surface for monitoring a DirectAdmin server — read-only commands plus IP blocking/unblocking only, with no open ports, no added services, and a blast radius limited by least privilege at every layer