#Monitoring
Passing the Sandbox Doesn't Mean It Works — 7 False Signals
The automated ordering system I wrote passed every command in a provider's sandbox, but almost everything the sandbox told me differed from reality, from response times and charges to accepted names and items that were created successfully but could not be found. Here are 7 false signals and how to write code that does not fall for them.
The Monitor That Hammered Devices Until the Whole Farm Went Down
A monitoring loop that runs every 30 seconds and tries to recover every device that fails a ping sounds useful, until one device hangs and gets hammered repeatedly until the whole bus goes down. This article walks through ruling out hardware, tracking what was accessing the devices, and the settings that fixed it.
The NVR Kept Beeping, Then Fell Silent When the Disks Failed
The NVR's all-day beeping was not videoloss as everyone thought. It was ipconflict from cameras using DHCP with conflicting IP addresses. Someone got annoyed and muted the whole unit, so when both disks failed together, there was not a single beep. This article fixes the policy one event at a time through ISAPI.
No kWh Meter? Estimate Daily Electricity Costs from Current
Notes on converting CT current readings into energy usage and daily time-of-use electricity costs, from the three-phase power formula and the sampling interval built into it, through skipping outages and separating peak from off-peak with epoch arithmetic, to a daily summary table that made the web page seven times faster.
Reading an Unknown Modbus Device: Verify Before Interpreting
My notes on tracking down an undocumented device on an RS485 bus, from scanning baud rates and addresses one combination at a time to proving which registers hold live values and which are dead, and the lesson that guessing a device's type from number patterns alone can lead to months of misinterpretation.
Reads from a Replica Must Show How Many Minutes Old the Data Is
replication can report a lag of 0 seconds while the data is stuck at yesterday, because these two signals measure different things — how to set up a monitor that checks both Seconds_Behind_Master and business data freshness, catches anyone writing directly to the slave, and has another check watching the monitor itself
The Service Is Running but Dead — systemd restart Cannot Help
A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.
Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read
In one day, 28 events became 56 messages, and 57 percent of all events were repeat alerts every 30 minutes. This article covers recipient tiers, merging overlapping cycles, state that survives restarts, and why you must not add keepalive to a chat app session.
Shared Neutrals Across Transformers: Why Phase Balancing Is Not Enough
A case involving two transformers with interconnected neutrals and houses tapped into one side's main feeder, leaving phase A 30-40 A higher even on days off, plus a meter-reading trap that can suggest a 200 A difference if the CT ratio is not verified first, and a repair sequence that must be followed in order.
Diagnosing Feeder Overload from Melted Insulation and Duty Cycles
My investigation of a 95 mm² main cable with insulation melting at scattered points even though the system seemed to work normally, from the nameplates of 6 washers and dryers totaling 234.7 kW to a minute-by-minute current model explaining why a 482 A load on a cable rated for 180 A had not caused a visible system failure.
News Stuck for Three Days Because the Feed Wasn't Sorted by Date
The news site's homepage showed the same day's stories for three days straight, even though scheduled jobs ran, translations finished, and every step reported success. The causes were an RSS feed sorted by relevance rather than date, string sorting of pubDate that actually sorted by weekday name, and requesting too few stories, leaving nothing after filtering before the rest even got a look.
CDP Still Responds but Commands Fail After Chrome Updates Itself
I run several automation jobs on one machine that need a real browser, and ran into two problems at once — jobs fighting over tabs and getting each other's results, and Chrome overwriting its own files during an update while the process was still running, leaving the port responsive but Runtime.evaluate unusable. This article covers how I isolate jobs, detect the failure, and repair it automatically.
A 25-Hour Power Outage Left Scheduled Jobs Missing
The machine running my automated jobs lost power three times in three days, with the longest outage lasting 25 hours. When it came back, I found that jobs scheduled directly through Task Scheduler had skipped their runs entirely, while jobs managed by a dispatcher that checked the time itself had all caught up. This article covers the setting to enable, what to avoid during recovery, and the traps that led me to misread the status.
Tell a Broken Link from an ISP Outage Before Restarting Anything
A typical watchdog restarts NetworkManager when ping fails, which is wrong half the time because no internet access may mean the ISP is down, not that the adapter has failed. This article distinguishes three states using nmcli, then escalates from gentle recovery to PCI remove and rescan without rebooting the machine.
Managing Site Switches Remotely with UniFi's Cloud Connector Proxy
My workstation is in a different location from the site, with no VPN and no inbound ports open, but I still need to change VLANs per port, reserve fixed IPs, and track down failing cables. This article explains how Site Manager's connector proxy reaches the legacy API for both reads and writes without requesting a CSRF token, and the discipline needed when holding a key this powerful.
Using an NVR to Reach Inaccessible Cameras with ISAPI and SDK
The cameras sit behind the NVR's second interface, and none are directly reachable. This article works through what the NVR's ISAPI can actually do, with 26 of 38 tested endpoints working, retrieves CCTV snapshots and RTSP through the NVR, finds when each channel last recorded, and falls back to HCNetSDK through ctypes to read device logs when ISAPI runs out of options.
Every Layer Reported Success While the Output Was Garbage
In one day, four layers of the same automated pipeline failed together, but every layer reported success without a single error line. The person who caught it was a user who said the site was not updating. These are the rules I wrote afterward so that failing stages could speak up for themselves.
Backups Succeeded Nightly, but Offsite Uploads Failed for 3 Weeks
cron ran every night, local files were complete every day, and the log said done, but offsite storage had not changed for 3 weeks. The first 8 nights of backups were already gone for good because local rotation had deleted them first. This article examines the two gaps that let the failure stay silent for so long.
Auditing CCTV Directly with ISAPI and ffprobe RTSP, Bypassing the NVR
The NVR screen only tells you a camera is online. It does not tell you its model, how old its firmware is, or the actual stream resolution. In this article, I write a script to query ISAPI deviceInfo on each camera directly, then use ffprobe to read values from the actual RTSP stream to build an inventory and check the health of the camera system I maintain.
Planning 2.4GHz WiFi Channels 1/6/11 and Multi-Floor Heatmaps
A five-story building with seven APs per floor needs a 2.4GHz channel plan that avoids overlap both horizontally and vertically. This article covers the 1/6/11 principle, per-floor channel tables, the actual Tx power and RSSI threshold settings, and a Python script that labels floor plans with channels using PIL before a walkthrough survey to create a heatmap.
Inventory Switch and AP Ports with SNMP Walk and MAC Tables
You can build your own Layer 2 inventory without a controller or NMS. Write an SNMP walk with pysnmp asyncio to retrieve sysDescr, ifTable, and the bridge FDB, then convert OID indexes back into MAC addresses to find which device connects to which port on which switch.
Building a Single-Screen Ping Monitor with C++, ImGui and DirectX 11
A small app that pings every device on the network at once and puts the results on one screen. It sends ICMP directly through iphlpapi without calling ping.exe, runs one thread per host, communicates with the UI through std::atomic, renders with Dear ImGui on DirectX 11, and builds with CMake and a single PowerShell script.
Read Device Uptime Without SNMP/SSH/API Using TCP Timestamps
Many IP cameras, door controllers, and embedded devices have no SNMP, SSH, or API for querying uptime. But if they return TCP timestamps, I can read their boot time remotely without logging in even once.
MariaDB on defaults — a 128M buffer pool on a 55 GB RAM server
A 16 core server with 55 GB RAM and an SSD, but MariaDB still had the default 128 MB buffer pool. The result was 50 million disk reads in 5 days. The full before-and-after settings, service name and config path traps on DirectAdmin/Debian, rollback steps, and another 500 MyISAM tables waiting for conversion.
One Query Hung for 54 Minutes, Causing Sitewide 504s via MyISAM Locks
A single 30 MB table caused 504s on every page across the site, even with 9.5 GB of RAM still free — I traced information_schema.PROCESSLIST to a single SELECT that had been stuck for 54 minutes on a MyISAM table, cascading into metadata locks across the server, with an immediate fix and ways to prevent it from happening again.
Inferring Cold Room Defrost Timing from Temperature Slopes
A cold room used to dehydrate glass noodles has no humidity sensors, but frost on the coil always flattens the temperature drop slope (dT/dt). This article uses the lumped-capacitance equation and time constant τ to detect frost and trigger demand defrost with working Python code, along with calculations for R404A COP and water removed per cycle.
reCamera + Gimbal: RISC-V Cross-Compilation, CAN, and Node-RED
reCamera is an AI camera on a RISC-V chip mounted on two brushless axes, with rotation controlled over CAN bus at a resolution of 0.01 degrees. This article covers cross-compiling C for riscv64 musl, uploading to the device with scp, the CAN frame structure for motor control, reading back the 15-bit encoder, finding motor zero using current, a Node-RED dashboard + PID + auto-track, and STP files for 3D-printing the housing.
MDES 2564 Government CCTV Price Limits and a Hikvision PoE BOM
Read the full MDES June 2564 government CCTV reference pricing and basic specifications before setting a budget, then design a Hikvision IP/PoE BOM that meets the minimum specifications, reference price ceilings, 30-day recording retention rule, and requirement to allow at least 3 brands to compete.
Reading a DTSU666 3-Phase Meter with Raspberry Pi over Modbus RS485
Notes on connecting a DTSU666 Smart Power Sensor to a Raspberry Pi over RS485, reading 3-phase electrical measurements from multiple meters using Modbus RTU, decoding 32-bit floats correctly, and storing them in SQLite/MySQL before exporting through MQTT and a REST API, including a negative-value bug I ran into myself
Reading Modbus RTU over an RS485 Gateway Without Phantom Values
Connecting multiple Modbus RTU meters/sensors on an RS485 bus through a serial-to-Ethernet gateway can produce phantom values, incorrect readings, or devices that do not actually exist — because a transparent gateway uses RTU-over-TCP, which requires assembling frames and calculating CRC yourself, and validating every reply. This article covers the rules I learned from doing it.
Mail Checks Ban the Whole Office—Fixing It with Login Auto-Whitelist
Real mail clients behave exactly like bots — office machines keep retrying old passwords until CSF bans the whole office. This article covers an auto-whitelist system that watches for successful logins + Thai GeoIP, then grants a temporary allow for 24 hours, along with 5 pitfalls I learned the hard way.
Per-user Redis on shared hosting—and the maxmemory bug that hung sites
Designing one Redis instance per user on shared hosting with a systemd template + cgroup — and lessons from a maxmemory bug where setting it above the cgroup limit caused intermittent WooCommerce hangs, plus the 80% rule and a guard script to prevent recurrence
A Telegram Bot for DA Server Monitoring with a Small Blast Radius
A guide to setting up a Telegram bot as a lightweight control surface for monitoring a DirectAdmin server — read-only commands plus IP blocking/unblocking only, with no open ports, no added services, and a blast radius limited by least privilege at every layer