The Service Is Running but Dead — systemd restart Cannot Help

I have a daemon on a server at the factory. Its job is to maintain a long-lived session with an external API and send notifications when events come in from the factory's sensor system. One day, the factory's outbound internet connection went down overnight. By morning, the connection was back and everything was normal, but notifications had gone completely silent. Not a single message went out.

What made this case particularly interesting to me was that when I checked, the system said everything was fine.

systemctl status alert-bridge
# * alert-bridge.service - outbound alert bridge
#      Loaded: loaded (/etc/systemd/system/alert-bridge.service; enabled)
#      Active: active (running) since Tue 2026-10-13 03:12:41 +07; 9h ago
#    Main PID: 21884 (python3)

active (running), with nine hours of uptime. Memory usage was normal, there was no exit code to inspect, and nothing was wrong as far as systemd could see. In reality, though, it had been dead since two in the morning. I think this is worth writing about because it completely changed how I design monitoring.

All service names, paths, and IP addresses in this article are examples.

Tracing the logs back: 194 failures, then silence

journalctl -u alert-bridge --since '2026-10-13 01:00' --until '05:00' | tail -20
# 01:58:11 [login] Temporary failure in name resolution
# 01:58:41 [login] Temporary failure in name resolution
#   ... (วนแบบนี้ 194 รอบ)
# 03:12:41 [boot] starting, restoring session from token
# 03:12:41 [login] Logging in with token...
#   (ไม่มีบรรทัดต่อจากนี้อีกเลย)

The picture became clear immediately. The network outage broke domain name resolution, so every reconnection attempt failed. The code caught the exception and retried, which was the correct behavior. It failed 194 times, with a log entry for every attempt.

Then, on attempt 195, it hung in the middle of a step with no timeout. The library I was using kept a socket open, waiting for a response from an endpoint that was only half alive at that point. It neither rejected the connection nor responded. It just stayed silent. So the process sat there waiting all night, with no exception to catch and no exit.

Why systemd did not restart it — it had no way to know

My service unit used the standard configuration everyone writes.

[Unit]
Description=outbound alert bridge
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=svcapp
ExecStart=/opt/venv/bin/python /opt/alert-bridge/bridge.py
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Restart=on-failure was working exactly as intended. It watches for the process to end with a nonzero exit code, then starts it again. The problem is that in this case, it never ended. The process was still there, consuming zero percent CPU, waiting in the same syscall forever. Even Restart=always would make no difference, because the condition that triggers it is "the process has ended," which never happened.

An appealing approach I rejected: frequent session checks

The first idea everyone reaches for is to have the daemon send a lightweight request to the API endpoint every few minutes. If it gets no response, consider the session dead and reconnect. It sounds reasonable, but I had tried it before and got the opposite result.

Most external services have mechanisms to protect against frequent requests. When we send keepalives too regularly, they interpret that as abnormal and start throttling or dropping the session. The result is more frequent session hangs. We end up creating the very problem we are trying to fix.

If you can change the code, use systemd's WatchdogSec directly

For services I write myself and can modify, the cleanest approach is to have the program check in with systemd periodically. If it stays silent beyond the deadline, systemd considers it failed and handles it. No additional monitor is needed.

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=90
Restart=always
RestartSec=10
ExecStart=/opt/venv/bin/python /opt/alert-bridge/bridge.py

The program must send its notifications from the main loop that does the actual work, not from a separate thread running independently. If an unrelated thread sends the notifications, the work loop can hang while we keep reporting that everything is fine, putting us right back at the original problem.

import socket, os

def notify(state: str) -> None:
    addr = os.environ.get('NOTIFY_SOCKET')
    if not addr:
        return
    if addr.startswith('@'):
        addr = '\0' + addr[1:]
    with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as s:
        s.connect(addr)
        s.send(state.encode())

notify('READY=1')
while True:
    msg = inbox.get(timeout=30)      # ลูปงานจริง
    handle(msg)
    notify('WATCHDOG=1')             # แจ้งจากตรงนี้เท่านั้น

If you cannot change the code: internal status + an external monitor

There are many daemons I cannot modify, or can modify but would rather leave their working main loops alone. My approach is to have them write their internal status to a file or expose an HTTP endpoint listening only on 127.0.0.1, then have a small external monitor read it.

#!/usr/bin/env python3
# /opt/alert-bridge/watchdog.py — อ่านสถานะภายใน ไม่ยิงออกนอก
import json, subprocess, urllib.request, pathlib

STATE = pathlib.Path('/run/alert-bridge-fail')
LIMIT = 3

def healthy() -> bool:
    try:
        with urllib.request.urlopen('http://127.0.0.1:3847/health', timeout=5) as r:
            h = json.load(r)
    except Exception:
        return False
    # อายุการทำงานพอสมควรแล้วแต่ลูปรับข้อมูลไม่เดิน = ค้าง
    return not (h.get('uptime_s', 0) > 180 and not h.get('loop_alive'))

n = 0 if healthy() else int(STATE.read_text() or 0) + 1 if STATE.exists() else 1
STATE.write_text(str(n))
if n >= LIMIT:
    subprocess.run(['systemctl', 'restart', 'alert-bridge'], check=False)
    STATE.write_text('0')

# /etc/systemd/system/alert-bridge-watchdog.timer
[Unit]
Description=check alert-bridge every minute

[Timer]
OnBootSec=3min
OnUnitActiveSec=60s
AccuracySec=5s

[Install]
WantedBy=timers.target

systemctl enable --now alert-bridge-watchdog.timer
systemctl list-timers alert-bridge-watchdog.timer
# NEXT                        LEFT  LAST                        PASSED  UNIT
# Fri 2026-10-17 09:41:12 +07  38s  Fri 2026-10-17 09:40:12 +07  21s ago ...

The point I stress is that checks must fail several times in a row before triggering a restart. A threshold of 3 checks spaced one minute apart means the service must actually be unhealthy for roughly three full minutes before it gets restarted. If we act on a single failed check, we end up with a machine that restarts itself every time the network stumbles for one second, making the system even less stable.

Change the metric from "process alive" to "fresh data"

This is what I consider the most important part of the whole story. Even with a monitor in place, if we watch only whether the process is still alive, we are still answering the wrong question. The right question is how old is the latest piece of data this service is supposed to produce?

So I have every job that runs periodically write a small status file, and have both the web page and the monitor read that same file.

{
  "ts": 1760671283,
  "loop_alive": true,
  "last_sent_ts": 1760671201,
  "last_error": null,
  "queued": 0
}

My criterion is the age of the value, not whether the file exists, because a stale file left behind is exactly what a silent failure looks like.

# ข้อมูลเก่ากว่า 5 นาที = ถือว่าตาย ไม่ว่า process จะยังอยู่หรือไม่
AGE=$(( $(date +%s) - $(jq -r '.ts' /run/alert-bridge/health.json) ))
[ "$AGE" -gt 300 ] && echo "[ERR] stale ${AGE}s" || echo "[OK] fresh ${AGE}s"

The trap that nearly made the monitor fail silently too

I ran into this myself, and I want everyone to watch out for it. The monitor and the status file must belong to the same user. I once accidentally created a status file while testing as a privileged user and left it there. When the scheduled job later ran as an ordinary user, it could not overwrite the existing file because it lacked permission.

The result was a status file that stopped updating but still existed. Its contents were frozen at the last successful write, and the monitor reading it calmly kept reporting the same values over and over. Our monitoring system had itself become a silent failure, which is more dangerous than having no monitoring at all, because now we have a reason to believe everything is fine.

ls -l /run/alert-bridge/health.json
# -rw-r--r-- 1 root root 118 Oct 13 01:57 health.json   <- เจ้าของผิด

install -d -o svcapp -g svcapp -m 0755 /run/alert-bridge
chown svcapp:svcapp /run/alert-bridge/health.json
sudo -u svcapp touch /run/alert-bridge/health.json   # ต้องไม่มี Permission denied

1. Always make the user that actually runs the job the owner of the status file, and test a write as that user once during installation. Do not test with full privileges and call it a pass, because full privileges let you write anywhere anyway.
2. Use /run for status files and declare RuntimeDirectory in the service unit. The system will create the directory with the correct owner on every boot, without relying on a directory someone once left behind with mkdir.
3. Give the monitor a monitor of its own. At a minimum, check that the timer is still working and that its last run was no more than twice the configured interval ago. If it exceeds that limit, send an alert through a channel that does not depend on the service having problems.

Three rules I have applied to every service since

First, systemd's status does not tell you whether a service is doing its job. It only tells you that the process has not ended. To know more, you need a signal from inside the service itself. Second, every network wait must have a time limit, whether connecting, reading, or waiting for a response, because a wait that never expires is how a program dies without dying. Third, measure the output, not the pulse. Ask how many minutes old the latest data is, not whether the process is still breathing.

As for whom to notify once a problem is detected, and at what severity, I wrote about that separately in Tiering alerts so people keep reading them, because a system that detects everything and makes noise all the time will eventually have nobody listening, which amounts to having no monitoring at all. If I have missed anything in the explanation or made a typo, I will come back and fix it later.