Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read

I was building a monitoring system for a refrigeration unit, reading current from a meter into a database and running a script to watch the readings and send messages to a chat app when events occurred. Everyone liked it at first. Two weeks later, recipients started complaining that it was getting things wrong. I went through a full day's logs and found that detection had not made a single mistake. Every event was correct. What was wrong was the message count. I collected these tuning changes as a set because they apply to any alerting system, not just refrigeration monitoring.

I have already covered the physics of inferring whether a unit is defrosting in Inferring Defrost Cycles from the Temperature Slope. This article covers just one thing: once detection is correct, what should be sent to whom?

Count Before Fixing: 28 Events Became 56 Messages in One Day

The first thing I did was count, not change code. I pulled the logs for the day people complained about and counted by event type. Without this step, I would be guessing that major events were the culprit, when it is often small, repeated notifications.

# นับเหตุการณ์ที่ยิงออกไปในวันเดียว แยกตามชนิด
grep "^$(date +%F)" /var/log/example-notify.log \
  | grep -oE '\[event\] [A-Z-]+' | sort | uniq -c | sort -rn

# นับจำนวนข้อความจริงที่ออกไป (หนึ่งเหตุการณ์ส่งหลายคนได้)
grep -c '\[send\] ok=True' /var/log/example-notify.log

These numbers changed how I thought about the whole problem. The events with real value were startup, shutdown, and five defrost cycles, totaling twelve events. The other sixteen events were repeat alerts after defrost ended, saying only how many minutes the unit had been running, and they made up 57 percent of the total. Put simply, recipients had to read messages with no new information more than half the time, every day. Then they stopped reading all of them, including the important ones.

Separate Recipient Tiers First: Halve the Messages Without Changing Logic

The quickest improvement came from stopping the practice of sending everything to everyone. The original system had two recipients who received identical messages, which was wrong from the start because they needed different things. The on-site operator needed to know every transition because they had to walk over and check the unit. The business owner only wanted to know when the unit started and stopped that day, and the summary at the end of each cycle.

# ผู้รับแยกชั้น: (ปลายทาง, สไตล์ข้อความ, ระดับที่รับ)
TIER_ALL   = {"on", "off", "defrost_start", "defrost_end", "reminder", "runtime"}
TIER_BRIEF = {"on", "off", "defrost_end"}          # เจ้าของ: เปิด ปิด และสรุปจบรอบ

RECIPIENTS = [
    ("site-lead",  "casual", TIER_ALL),
    ("owner",      "formal", TIER_BRIEF),
]

def notify(kind, build):
    """build(style) -> ข้อความตามสไตล์ของผู้รับแต่ละคน"""
    for target, style, tier in RECIPIENTS:
        if kind not in tier:
            continue
        try:
            send_message(target, build(style))
        except Exception as e:
            log("[notify] %s: %s" % (target, e))

Recipient tiers also brought differences in wording. I made the message-building function accept style and return different wording, because the on-site operator was comfortable with short, blunt messages while the owner preferred a more polite, formal tone. The important part is that it is the same function, not two separate sets of code. Otherwise, I would inevitably change one and forget the other.

Repeat Alerts: From Six Down to One

The main culprit was the repeat-alert block after defrost ended, configured to send every 30 minutes, up to 6 times. The original intention was sound: I wanted to know whether the room cooled back down quickly enough after defrost. In practice, though, the second through sixth messages added no information. They only said another thirty minutes had passed.

ON_REPORT_EVERY = 1800     # วินาที: จังหวะรายงานระหว่างเครื่องเดิน
PD_INTERVAL     = 1800     # วินาที: จังหวะเตือนซ้ำหลังละลายน้ำแข็งจบ
PD_MAX          = 1        # เดิม 6 -> เหลือ 1 (สรุปหนึ่งครั้งพอ)

now = int(time.time())
if S["state"] == "ON" and not S["def_active"] and S.get("pd_end") and now >= S.get("pd_next", 0):
    stat = read_status(db)
    mins = max(1, round((now - S["pd_end"]) / 60))
    notify("reminder", lambda style: msg_post_defrost(mins, stat, style))

    S["pd_count"] += 1
    S["pd_next"]   = now + PD_INTERVAL
    if S["pd_count"] >= PD_MAX:
        S["pd_end"] = 0          # ปิดชุดนี้ รอรอบละลายน้ำแข็งครั้งหน้า
    save_state()

Another change I made was to prevent running-status reports and post-defrost repeat alerts from overlapping. Originally, each had its own timer, and sometimes they lined up exactly. Recipients would then get two consecutive messages saying the same thing. The fix was to have one push back the other's timer whenever it sent a message: S["on_next"] = now + ON_REPORT_EVERY. This keeps the two from ever colliding.

Overlapping Cycles Must Be Merged, Not Counted Twice

That day's logs showed two defrost cycles just 8 minutes apart, at 10:12 and 10:27, which did not look reasonable. I understood what happened only after tracing the actual current readings: during the first cycle, the heater switched off for about 7 minutes and then came back on, which is normal behavior for this unit. But the grace period I had configured was 120 seconds. Seven minutes is longer than two minutes, so the system had already declared the first cycle finished. When the heater came back on, it counted a second cycle, producing two consecutive sets of start and end messages.

DEF_THR       = 100    # เกณฑ์กระแสฮีตเตอร์ ที่ถือว่าเป็นการละลายน้ำแข็งจริง
DEF_END_GRACE = 900    # เดิม 120 วินาที -> 900 วินาที (ใกล้เคียงความยาวรอบปกติ)

if heater_current > DEF_THR:
    if not S["def_active"]:
        S["def_active"] = True
        S["def_start"]  = ts
        S["pd_end"]     = 0          # รอบใหม่ ยกเลิกชุดเตือนซ้ำของรอบก่อน
    S["def_off_since"] = 0           # ฮีตเตอร์กลับมา = ยังเป็นรอบเดิม ไม่ split
    S["def_last"] = ts

elif S["def_active"]:
    if not S.get("def_off_since"):
        S["def_off_since"] = ts
    if ts - S["def_off_since"] >= DEF_END_GRACE:
        S["def_active"] = False
        S["ndef"] += 1
        notify("defrost_end", lambda style: msg_defrost_end(S, style))
        S["pd_end"]  = S["def_last"]        # เริ่มนับชุดเตือนซ้ำจากตรงนี้
        S["pd_next"] = S["def_last"] + PD_INTERVAL
        S["pd_count"] = 0

One detail needs care: when reporting cycle duration, count up to the last time the heater was seen running, not the time the system decided the cycle had ended. Otherwise, every cycle will appear 15 minutes longer than it actually was, matching the configured grace period. I therefore keep def_last separate from the decision time and always report using def_last.

State Must Survive Restarts, or Counting Starts Over Every Time

That day's timeline also included a service restart during the day at 19:43. The repeat-alert series correctly continued its count because all state was written to a file instead of being kept only in memory. Without this, every restart would reset the counter to zero, and recipients would get another six repeat alerts even though they had just received them.

STATE = "/var/lib/example/notify.state"

S = {"state": "OFF", "last_ts": 0, "on_since": None, "day_start": 0, "ndef": 0,
     "last_active": 0, "def_active": False, "def_start": 0, "def_last": 0,
     "def_off_since": 0, "pd_end": 0, "pd_next": 0, "pd_count": 0, "on_next": 0}

def load_state():
    try:
        S.update(json.load(open(STATE)))
    except Exception:
        pass                       # ไฟล์ยังไม่มี = เริ่มจากค่าตั้งต้น ไม่ใช่ข้อผิดพลาด

def save_state():
    try:
        json.dump(S, open(STATE, "w"))
    except Exception as e:
        log("[state] %s" % e)

Another equally important detail is that on the first run after a restart, the system must read the current values to initialize its state and stay quiet. It must not send retrospective messages. Otherwise, every restart while the unit is running will send recipients another startup message, even though the unit has been on since morning. I explicitly log that startup is silent here, so when I read the logs later, I know why no message was sent at that point.

Automatically Suppress Alerts When the Unit Is Off, but Skip One Interval, Not the Whole Series

A problem that follows from configuring repeat alerts is that if the unit is turned off before the series finishes, the system will keep reporting how many minutes it has been running, even though the cabinet is already off. The fix is to check the actual current before every send. If no current is flowing, do not send.

But I got this fix wrong once. At first, I canceled the entire series whenever the current was zero. As a result, when the compressor cycled off normally under thermostat control, leaving the current at zero for two or three minutes, the whole report series disappeared. Recipients then complained about missing reports. I changed it to skip this interval and wait for the next one instead.

running = (stat.get("comp", 0) > COMP_THR) or (stat.get("heat", 0) > 0) \
          or (stat.get("line", 0) > 0)

if not running:
    # คอมตัดชั่วคราวตามเทอร์โมสตัท = ข้ามรอบนี้พอ ห้ามยกเลิกทั้งชุด
    # ถ้าเครื่องดับจริง ตัวตรวจจับ OFF จะเคลียร์ pd_end ให้เองอยู่แล้ว
    S["pd_next"] = now + PD_INTERVAL
    save_state()
    log("[pd] ไม่มีกระแสไหล ข้ามรอบนี้ รอรอบหน้า")
else:
    ...ยิงข้อความตามปกติ...

The principle is to let the state detector decide whether the event has ended. The message sender should only decide whether to send at this interval. Mixing those two responsibilities leads to the strange behavior I ran into.

Do Not Add Keepalive or Frequent Polling to a Chat App Session

This is the most important point in the article, and the one people most often try to fix the wrong way. The symptom is that recipients complain about messages arriving about a minute late. Most people respond by checking connection status more often or adding keepalive because they think the connection is going to sleep.

When I actually traced that one-minute delay, it all came from our side and could be fixed without touching the connection.

# ก่อนแก้: หน่วงรวมประมาณ 61 วินาที
POLL       = 10    # อ่านฐานข้อมูลทุก 10 วินาที
ON_CONFIRM = 30    # ต้องเห็นค่ากระแสต่อเนื่อง 30 วินาที ถึงจะฟันว่าเปิดเครื่อง

# หลังแก้: หน่วงรวมเหลือประมาณ 3 ถึง 4 วินาที
POLL       = 1     # ฐานข้อมูลอยู่บนเครื่องเดียวกัน อ่านทุกวินาทีไม่กระทบใคร
ON_CONFIRM = 3     # กระแสเดินคือเครื่องเปิดแล้ว ไม่ต้องหน่วงยืนยัน

I could reduce the confirmation setting from 30 to 3 because startup is inherently a sustained event. It is not a flickering signal that needs filtering. Once current starts flowing, the unit really is on. Reducing the read interval from 10 to 1 second affects nobody else because it reads a database on the same machine, without calling an external service. The principle I settled on is this: to reduce latency, lower our own confirmation delay and read interval, rather than increase how often we bother the downstream service.

Check Health Before Every Send, and Log Every Skip

The last point is about preventing silent message loss. If the messaging component responds in a fire-and-forget manner, reporting success immediately before it has actually sent anything, there is no way to know whether the message arrived. My approach is to call the health endpoint before each send, then skip and log it if the check fails. That is better than unknowingly sending messages into a void.

<?php
// เรียก /health ก่อน /send ทุกครั้ง แล้วบันทึกเหตุผลถ้าข้าม
$ctx  = stream_context_create(['http' => ['timeout' => 8, 'ignore_errors' => true]]);
$raw  = @file_get_contents('http://127.0.0.1:8080/health', false, $ctx);

$ok = false; $alive = false; $err = '';
if ($raw !== false && ($h = json_decode($raw, true)) && is_array($h)) {
    $ok    = !empty($h['ok']);
    $alive = !empty($h['session_alive']);
    $err   = $h['session_last_error'] ?? '';
}

if (!$ok || !$alive) {
    $reason = ($raw === false) ? 'ติดต่อบริการไม่ได้'
              : (!$alive ? "session ตาย ({$err})" : 'health ไม่ผ่าน');
    file_put_contents($logFile,
        sprintf("[%s] SKIP — %s\n", date('Y-m-d H:i:s'), $reason), FILE_APPEND);
    exit(1);
}

// ผ่านแล้วค่อยยิงจริง
foreach ($recipients as $name => $target) { send_message($target, $text); }

1. Count first Pull one day's logs, group them by event type, and see which type accounts for more than half.
2. Separate recipient tiers Give each person only the information they can act on.
3. Cap the repeat count for every repeat-alert series, and have the last message say it is the last.
4. Set the grace period to merge closely spaced cycles using historical cycle durations, not a gut feeling.
5. Save state to a file and always start silently after a restart.
6. Reduce latency on our side Do not increase how often the downstream service is contacted.
7. Check health before sending and log every decision to skip.

Personally, I think the quality of an alerting system is measured by how many messages recipients are still willing to open and read, not by detection accuracy. A system with one hundred percent accurate detection that sends fifty messages a day is worth as much as a system that is switched off, because people eventually mute its notifications anyway. Feel free to adapt these settings to your own work. If I have left anything out or made a typo, I will come back and fix it later.