Tell a Broken Link from an ISP Outage Before Restarting Anything

I happen to maintain a machine that serves as the central host for another project, sitting at the project owner's office. It connects over WiFi rather than a cable, and nobody sits next to it. The problem was that it would simply disappear in the middle of the night: no ping response, no SSH access. When someone went over to the machine and pressed a button in the morning, it came back on its own. At first I thought it was sleeping, but the logs showed that the machine had been awake the whole time. Only the WiFi had gone away.

The project owner gave me one instruction: frequent reboots are not the answer; manage the device instead. So I wrote a watchdog that escalates from gentle to aggressive recovery and, more importantly, has conditions for doing nothing at all.

The IP addresses, SSID, hostname, and passwords in this article are example values. The commands, module names, and sequence are the actual ones from the machine I maintain.

Typical watchdogs are wrong half the time

Almost every example I found online followed this pattern: ping something outside the network, restart the network service if it fails, done.

The problem is that ping can fail for many reasons, and one of the most common is a provider outage. Restarting anything on our machine at that point does nothing to help. It only makes things worse, because if the internet comes back during the restart, we miss that window and start another cycle. On machines that escalate all the way to an automatic reboot, the result is a machine rebooting itself all night while the provider repairs a cable.

The first trap: missing iw makes every check wrong

Most examples I found used iw to check whether the adapter was associated with the AP. I followed that approach and found the watchdog reporting a broken link every two minutes, even though the internet worked normally.

The reason was that this machine did not have iw installed, and the first version of my script checked the result without distinguishing between a missing command and a command that ran and reported no association. Both cases produced the same "not associated" result.

# ผิด: ถ้าไม่มี iw จะได้ค่าว่างเสมอ แล้วสรุปว่าไม่ associate ทุกครั้ง
iw dev "$IFACE" link | grep -q "Connected to" && echo up

# ถูก: ใช้ของที่มีแน่ และแยกกรณีคำสั่งหายออกมาก่อน
command -v nmcli >/dev/null || { log "ไม่มี nmcli ออก"; exit 0; }
STATE=$(nmcli -t -f GENERAL.STATE device show "$IFACE" 2>/dev/null | cut -d: -f2)
case "$STATE" in
    *connected*) ASSOC=1 ;;
    *)           ASSOC=0 ;;
esac

Clearly distinguish the three states

This is the heart of the script: ask three questions in sequence, then determine which case applies. Notice that the first thing I check is not "Can I reach the internet?" but "Is my link healthy?"

#!/bin/bash
# /usr/local/bin/net-watchdog.sh
set -u
IFACE=${IFACE:-wlan0}
GW=$(ip route | awk '/^default/ {print $3; exit}')
STATE_FILE=/var/lib/net-watchdog/fails
LOG=/var/log/net-watchdog.log
log() { echo "$(date '+%F %T') $*" >> "$LOG"; }

# 1. associate อยู่ไหม
STATE=$(nmcli -t -f GENERAL.STATE device show "$IFACE" 2>/dev/null | cut -d: -f2)
case "$STATE" in *connected*) ASSOC=1 ;; *) ASSOC=0 ;; esac

# 2. มี IP ไหม
HAS_IP=0
ip -4 addr show dev "$IFACE" | grep -q 'inet ' && HAS_IP=1

# 3. gateway ตอบไหม  4. ออกเน็ตได้ไหม
GW_OK=0; NET_OK=0
[ -n "${GW:-}" ] && ping -c2 -W2 -I "$IFACE" "$GW"  >/dev/null 2>&1 && GW_OK=1
ping -c2 -W3 -I "$IFACE" 1.1.1.1 >/dev/null 2>&1 && NET_OK=1

if [ "$NET_OK" = 1 ]; then
    echo 0 > "$STATE_FILE"; exit 0                       # ปกติ ไม่ทำอะไร
fi
if [ "$ASSOC" = 1 ] && [ "$HAS_IP" = 1 ] && [ "$GW_OK" = 1 ]; then
    log "ลิงก์ปกติ gateway ตอบ แต่ออกเน็ตไม่ได้ = ปลายทางล่ม ไม่แตะอะไร"
    echo 0 > "$STATE_FILE"; exit 0                       # ISP ล่ม ปล่อยไว้
fi
FAILS=$(( $(cat "$STATE_FILE" 2>/dev/null || echo 0) + 1 ))
echo "$FAILS" > "$STATE_FILE"
log "ลิงก์พัง รอบที่ $FAILS (assoc=$ASSOC ip=$HAS_IP gw=$GW_OK)"

The line to emphasize is the one that resets the counter to zero during an ISP outage. It must not just do nothing; it must also stop counting. Otherwise, after a two-hour provider outage, the counter will have climbed to the most aggressive step, and the watchdog will remove the adapter just as the internet comes back.

Escalating from gentle to aggressive recovery

I do not let it jump to an aggressive step on the first cycle, because most disconnections clear up on their own within a cycle or two. The counter stored in a file determines what happens on the current cycle.

1. Cycles 1-2 Do nothing except log it. Brief disconnections resolve themselves more often than you might think.
2. Cycle 3 Use nmcli to bring the connection down and back up. This is the gentlest action and the one that works most often.
3. Cycle 6 Restart NetworkManager entirely. Use this when nmcli itself starts responding strangely.
4. Cycle 10 Unload the driver module and load it again. Use this when the adapter's firmware appears to be stuck.
5. Cycle 15 onward Remove the device from PCI and rescan. Keep repeating this step without escalating to a reboot.

IFACE_CONN=$(nmcli -t -f NAME,DEVICE connection show --active \
             | awk -F: -v d="$IFACE" '$2==d {print $1; exit}')

if   [ "$FAILS" -ge 15 ]; then
    log "ขั้น 4: PCI remove + rescan"
    SLOT=$(basename "$(readlink -f /sys/class/net/$IFACE/device)")
    echo 1 > "/sys/bus/pci/devices/$SLOT/remove"
    sleep 5
    echo 1 > /sys/bus/pci/rescan
elif [ "$FAILS" -ge 10 ]; then
    log "ขั้น 3: reload โมดูลไดรเวอร์"
    modprobe -r rtw89_8852bte && sleep 3 && modprobe rtw89_8852bte
elif [ "$FAILS" -ge 6 ]; then
    log "ขั้น 2: restart NetworkManager"
    systemctl restart NetworkManager
elif [ "$FAILS" -ge 3 ]; then
    log "ขั้น 1: nmcli ปิดเปิดการเชื่อมต่อ"
    nmcli connection down "$IFACE_CONN"; sleep 3
    nmcli connection up   "$IFACE_CONN"
fi

Set it up as a timer, not cron

I use a systemd timer instead of cron because its logs live alongside everything else, are visible through journalctl, and it supports Persistent. Most importantly, it does not overlap with itself if the previous cycle has not finished.

# /etc/systemd/system/net-watchdog.service
[Unit]
Description=Network link watchdog
After=network.target

[Service]
Type=oneshot
Environment=IFACE=wlan0
ExecStart=/usr/local/bin/net-watchdog.sh

# /etc/systemd/system/net-watchdog.timer
[Unit]
Description=Run network watchdog every 2 minutes

[Timer]
OnBootSec=3min
OnUnitActiveSec=2min
AccuracySec=10s

[Install]
WantedBy=timers.target

install -d -m 755 /var/lib/net-watchdog
systemctl daemon-reload
systemctl enable --now net-watchdog.timer
systemctl list-timers net-watchdog        # ดูว่ารอบหน้าเมื่อไหร่
journalctl -u net-watchdog.service -n 30 --no-pager
tail -f /var/log/net-watchdog.log

Eliminate known causes before writing a watchdog

This matters more than the entire script put together. A watchdog is a safety net, not a fix. If I already know what causes the disconnections, I need to disable that first.

The WiFi chip in this machine is from a Realtek family that uses the rtw89 driver. Its behavior was very clear: when there was no traffic, it entered power-saving mode and disconnected. From outside, it looked as though the machine was asleep, even though it was awake. The fix was to disable WiFi power saving entirely.

# /etc/NetworkManager/conf.d/wifi-powersave-off.conf
[connection]
wifi.powersave = 2

A value of 2 means off, while 3 means on. That is confusing to read, but that is how it works. After setting this, restart NetworkManager once, then check that the setting has taken effect.

systemctl restart NetworkManager
nmcli -f 802-11-wireless.powersave connection show "$IFACE_CONN"
# ถ้ายังหลุดอีก ค่อยไปดูเรื่อง ASPM ของ PCIe ต่อ อย่าเพิ่งข้ามไปแตะตรงนั้นก่อน

Disable system sleep too, or the symptoms are indistinguishable

The other thing to disable is sleep on the machine itself. A machine serving as a server needs to stay awake all the time, even with the lid closed. This needs to be settled first not just because of network disconnections, but because if the machine really is asleep, the symptoms from outside look exactly like a WiFi disconnection, leaving me unable to tell which problem I am chasing.

systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
# แล้วแก้ /etc/systemd/logind.conf ให้
#   HandleLidSwitch=ignore
#   IdleAction=ignore
systemctl restart systemd-logind

Personally, I think a machine doing important work should not be on WiFi in the first place. Run a cable to it. That costs far less than the time spent tracking down problems like this. But real sites have their constraints: the room has no wired network outlet, the building owner does not allow drilling through walls, and so on. If WiFi really cannot be avoided, at least disable power saving and have a watchdog that knows when to leave things alone. Adding USB-Ethernet as a backup connection would make even more sense, if that works for your setup.

What remains on my plan is to have it send an alert when it reaches the third step, so I know before it reaches the last one, and to find a chance to finish testing the PCI step on a machine within reach. If I have left anything out or made any typos, I will come back and fix them later.