The Monitor That Hammered Devices Until the Whole Farm Went Down

I had taken over a gateway farm from its previous maintainer: a single host with several devices connected through USB serial ports, each serving as an outbound connection for a separate set of jobs. The symptom that brought me in was, "Late in the morning, every device disconnects at once, then they come back on their own." This happened two or three times a day. The previous team had concluded that the hub was faulty, but replacing it twice had made no difference. Once I investigated, the cause turned out not to be hardware at all. It was the monitor the previous team had written.

The hostnames, IP addresses, and service names in this article are examples. The commands, kernel messages, and settings that fixed the problem are real.

Symptoms: one device hangs, then the whole farm goes down

The first thing I had to do was set aside the previous team's conclusion and look at the actual timeline to see what happened first. I pulled the kernel logs from the incident window and put them in chronological order, and the picture became clear immediately.

# ดูเฉพาะช่วงเหตุการณ์ ไม่ใช่ไล่ทั้งวัน
journalctl -k --since "2026-11-02 09:40" --until "2026-11-02 09:50" -o short-precise \
  | grep -Ei 'usb|tty|disconnect|reset'

One device disconnected roughly forty seconds before the others. That was followed by repeated port resets in quick succession, then a while later the remaining devices started disconnecting one by one. Eventually, the entire bus went down.

usb 1-1.3: USB disconnect, device number 14
usb 1-1.3: new full-speed USB device number 15 using ehci-pci
usb 1-1.3: device descriptor read/64, error -110
usb 1-1.3: new full-speed USB device number 16 using ehci-pci
usb 1-1-port3: Cannot enable. Maybe the USB cable is bad?
hub 1-1:1.0: cannot reset port 3 (err = -32)

There was only one way to read that sequence: something was issuing repeated resets before the same device had finished registering itself. This was not the pattern of a loose cable, which would normally disconnect and reconnect irregularly. There was a very clear rhythm here, and it was exactly every thirty seconds.

Rule out hardware first

Even though I already suspected software, I still worked through the hardware possibilities first. If the hub really could not supply enough power, no script fix would make the system reliable. The numbers to check were the power the hub reported it could supply against the combined requests from all the devices.

lsusb -t                                   # ดูผังว่าตัวไหนห้อยอยู่ใต้ฮับตัวไหน
lsusb -v 2>/dev/null | grep -E 'MaxPower|bNumInterfaces|iProduct' | head -40

The hub reported zero available power, which meant it was powered only by its upstream port, with no separate adapter. Each device requested several hundred milliamps, so the combined demand was definitely more than a single port could supply. This was a real problem that needed fixing, but it still was not the cause of the regularly timed failures.

Track what is accessing the devices

Once I suspected something was issuing resets, the next question was what. A machine inherited from someone else often has more running on it than the documentation says. I wrote a simple tracker that took a snapshot every five seconds of which processes had device files open and whether the list of devices visible to the system had changed.

#!/usr/bin/env python3
"""flap-detector: ถ่ายภาพนิ่งทุก 5 วิ ว่ามีใครแตะอุปกรณ์ และรายชื่อเปลี่ยนไหม"""
import glob, os, subprocess, time, json, datetime

def holders(path):
    """คืนรายชื่อโปรเซสที่เปิดไฟล์อุปกรณ์นี้ค้างไว้"""
    p = subprocess.run(["fuser", "-v", path], capture_output=True, text=True)
    out = []
    for line in (p.stderr or "").splitlines()[1:]:
        f = line.split()
        if len(f) >= 3:
            out.append("%s/%s" % (f[1], f[-1]))      # pid/ชื่อคำสั่ง
    return out

prev = None
while True:
    devs = sorted(glob.glob("/dev/ttyUSB*"))
    snap = {d: holders(d) for d in devs}
    if snap != prev:
        rec = dict(ts=datetime.datetime.now().isoformat(timespec="seconds"),
                   count=len(devs), snap=snap)
        print(json.dumps(rec, ensure_ascii=False), flush=True)
        prev = snap
    time.sleep(5)

After half an hour, I had the answer. Two things were accessing the same devices. The first was the monitoring service running on a schedule. The second was a button on the management web page that the previous team had built for manual device recovery. Neither knew about the other, and there was no lock coordinating them.

# ยืนยันอีกทางว่าใครถืออุปกรณ์อยู่ตอนนี้
sudo lsof /dev/ttyUSB* 2>/dev/null | awk 'NR==1 || $0 ~ /ttyUSB/'
systemctl list-timers --all | grep -i gw

The cause: a loop with no cooldown and no lock

When I opened the original monitoring script, the logic was very straightforward, and that was the problem. Every thirty seconds, it went through every device and immediately attempted recovery for any device that could not ping the destination. It did not count consecutive failures, remember when it had last attempted recovery for a device, or limit how many devices it could recover at once.

# ตรรกะของตัวเดิม ย่อมาแต่ใจความครบ
while True:
    for dev in devices:
        if not ping(dev["ip"], timeout=2):
            usb_reset(dev["bus_path"])     # สั่งทันที ไม่มีเงื่อนไขอะไรเลย
    time.sleep(30)

The problem was that recovery through a port reset always took longer than thirty seconds. The device had to disconnect, register again, and request its connection settings again, taking more than a minute in total. When the next loop arrived before the device had finished registering, the ping would still fail. The script would issue another reset, and the cycle would continue indefinitely. The hung device could never come back, and the frequent resets on the shared bus dragged the other devices down with it.

The frightening part of this case was that the worse the system got, the harder the monitor worked, because more and more devices met its recovery condition. It was not just failing to help; it was accelerating the failure. Nobody had suspected it for the past two weeks because everyone saw it as something that was there to help.

Rewrite recovery: one at a time, with a cooldown and a limit

Four settings made the symptoms disappear. I rewrote the whole thing, but the core logic was short. The key was that all four conditions had to be met before attempting recovery.

import json, os, time, subprocess

STATE = "/var/lib/gw-watchdog/state.json"
INTERVAL   = 60      # รอบละ 60 วิ ยาวกว่าที่การกู้ใช้เวลาจริง
NEED_FAIL  = 3       # ต้องล้มติดกัน 3 รอบก่อน ไม่ใช่พลาดครั้งเดียวแล้วลุย
COOLDOWN   = 300     # กู้ตัวเดิมซ้ำได้เร็วสุดทุก 5 นาที
MAX_PER_HR = 6       # เพดานต่อตัวต่อชั่วโมง เกินแล้วหยุดและแจ้งคน

def load():
    try:
        return json.load(open(STATE))
    except Exception:
        return {}

def save(s):
    os.makedirs(os.path.dirname(STATE), exist_ok=True)
    json.dump(s, open(STATE, "w"))

def should_recover(dev, st, now):
    d = st.setdefault(dev, {"fail": 0, "last": 0, "hits": []})
    if d["fail"] < NEED_FAIL:
        return False, "ยังล้มไม่ครบ %d รอบ" % NEED_FAIL
    if now - d["last"] < COOLDOWN:
        return False, "อยู่ในคูลดาวน์อีก %ds" % int(COOLDOWN - (now - d["last"]))
    d["hits"] = [t for t in d["hits"] if now - t < 3600]
    if len(d["hits"]) >= MAX_PER_HR:
        return False, "ชนเพดาน %d ครั้ง/ชม. ต้องให้คนเข้ามาดู" % MAX_PER_HR
    return True, "ok"

def tick(devices):
    st, now = load(), time.time()
    recovered = 0
    for dev in devices:
        d = st.setdefault(dev["id"], {"fail": 0, "last": 0, "hits": []})
        if ping(dev["ip"], timeout=2):
            d["fail"] = 0
            continue
        d["fail"] += 1
        ok, why = should_recover(dev["id"], st, now)
        if not ok:
            log("skip %s: %s" % (dev["id"], why))
            continue
        if recovered >= 1:                      # ทีละตัวต่อรอบเท่านั้น
            log("skip %s: รอบนี้กู้ไปแล้วหนึ่งตัว" % dev["id"])
            continue
        log("recover %s (fail=%d)" % (dev["id"], d["fail"]))
        usb_reset(dev["bus_path"])
        d["last"] = now
        d["hits"].append(now)
        d["fail"] = 0                            # เริ่มนับใหม่ ให้โอกาสมันฟื้น
        recovered += 1
    save(st)

The condition I consider most important is the hourly limit, because it changes how the system behaves when it encounters a problem it cannot fix on its own. The old system would keep trying forever. The new one tries six times, then gives up and calls for human help, which is the more appropriate behavior. A device that has not come back after six recovery attempts in 1 hour needs a person, not a seventh attempt.

Put the web interface behind the same lock

Fixing the service alone was not enough, because the button on the web page could still issue the old command, and staff who saw a disconnected device would instinctively click it repeatedly. I required every path that accessed the devices to go through the same lock file. While one held the lock, the others had to wait or back off.

#!/bin/bash
# ตัวห่อเดียวที่ได้รับอนุญาตให้สั่งกู้ ทั้งบริการและหน้าเว็บต้องเรียกผ่านตัวนี้
exec 9>/var/lock/gw-recover.lock
if ! flock -n 9; then
  echo '{"ok":false,"err":"มีการกู้อุปกรณ์อื่นอยู่ กรุณารอสักครู่"}'
  exit 75
fi
/usr/local/bin/gw-recover.py "$@"

[Unit]
Description=Gateway watchdog
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/gw-watchdog.py
Restart=on-failure
RestartSec=30
# กันตัวเองพังแล้ววนรีสตาร์ตรัวจนกลายเป็นลูปกระแทกอีกแบบ
StartLimitIntervalSec=600
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

Make its actions visible before trusting it

What the old system lacked entirely was a record of its actions. It triggered device recovery dozens of times a day without anyone knowing. In the new system, I required a separate log line for every decision, both when it acted and when it decided not to. A line saying "skipped because cooldown is active" is even more valuable than one saying recovery succeeded, because it proves that the safeguard is actually working.

journalctl -u gw-watchdog --since today | grep -c 'recover '
journalctl -u gw-watchdog --since today | grep 'skip ' | awk '{print $NF}' | sort | uniq -c

After the fix, daily recovery attempts dropped from dozens to single digits, and the incidents where the entire set of devices went down together disappeared completely. What remained were individual disconnections that recovered after one attempt, which is normal for this type of device.

My rules for automation that touches hardware

1. The loop interval must always be longer than the actual recovery time. If recovery takes 1 minute, do not set a 30-second interval, because the next loop will hit something that is still in progress.
2. Require several consecutive failed checks before acting. A single missed check is not an outage.
3. Use a cooldown per device and recover only one device per loop. Do not let one incident trigger simultaneous actions across the entire farm.
4. Set an hourly limit. Once it is reached, stop and call for human help instead of continuing indefinitely.
5. Every path that accesses the devices must go through a wrapper that holds a shared lock, including web buttons and manually run scripts.
6. Log both actions and decisions not to act, along with the reasons. Otherwise, I have no way of knowing what it has done.
7. Work through the hardware possibilities alongside this. Insufficient power and a loop hammering devices are separate problems; fixing only one will not resolve both.

I think a monitoring tool with permission to take corrective action needs more careful design than one that can only observe and alert. The worst an alert-only tool can do when it fails is send a false alert, but a tool that can take action can bring down the actual system. My rule is that if it can fix things on its own, it must also be designed to give up, and it must give up before it causes damage. For anyone adapting this structure to recovery work outside hardware, such as restarting a hung service, the same four conditions apply. Only the recovery command needs to change.

Still on my list is collecting statistics on which devices need recovery more often than the others over thirty days, so I can identify which ones really are due for replacement instead of waiting for them to die in place. I also want to try moving recovery to individual ports on a hub that supports switching power off per port, which should be cleaner than resetting at the bus level. If I have left anything out or made any typos, I will come back and correct them later.