PLC Reports a Healthy Link for 3 Seconds Without a Heartbeat

I was working on moving I/O across PLCs for a factory. I had already delivered the first version, 12 files, after every one passed the simulator test suite. The next day, I sat down to review my code again and found that the link watchdog I had written reported a healthy link throughout the first 3 seconds after power-up, even though it had never received a single heartbeat from the other end. If a command bit was left set on the receiving side during that interval, the valve output could actuate. My test suite had never caught this because it only started checking at the 5-second mark. In this article, I explain where the hole was, why the tests missed it, how I fixed it, and what I had to do about the files already delivered.

The system: two PLCs exchanging bits over an existing TCP link

This machine has two PLCs involved, which I will call Main PLC and Station PLC. Both are S7-1500 units programmed in TIA Portal V15.1, and they already communicate using TSEND_C and TRCV_C over an existing TCP link. My job was to add bits to that link. The main side generates the Up and Down commands for the pneumatic cylinder and sends them across. The station side controls the actual valve outputs and sends back the workpiece detection sensor status.

Once output commands have to travel over a network, the first question to answer is what happens if the link drops. I therefore had both sides send each other a 1 Hz heartbeat: a bit that toggles every half second. I set two fail-safe rules. First, if the station side sees no heartbeat from the main side, force the cylinder to the UP position and inhibit DOWN. Second, if the main side sees no heartbeat from the station side, treat that position as occupied by a workpiece so the main side does not send the next piece into it.

These two rules were correct according to the agreement with the client. What was wrong was how I decided that “the link is still up.”

The first version and the 3-second window nobody watched

In the first version, I wrote the watchdog using a common pattern: use a TON timer to check whether the heartbeat has gone unchanged for more than 3 seconds. If it has not, consider the link healthy.

// รุ่นแรก (ผิด) - ฝั่ง Station PLC
D.HbTimer.TON(IN := (Rx.Hb = D.HbPrev), PT := T#3S);
D.HbPrev := Rx.Hb;
D.PeerOK := NOT D.HbTimer.Q;

Valve_Down := Rx.DownCmd AND D.PeerOK;
Valve_Up   := Rx.UpCmd OR NOT D.PeerOK;

At a glance, it looks correct. Once the heartbeat stays unchanged for 3 seconds, Q goes high, PeerOK goes low, and the valve is forced UP. But trace it from the first scan after power-up: the timer has only just started counting, and Q is still FALSE. That means PeerOK becomes TRUE immediately, even though this PLC has never received a heartbeat. The other end might not even be powered on. The cable might not be plugged in. This line of code still reports a healthy link for a full 3 seconds.

PeerOK incorrectly being TRUE does not, by itself, actuate the valve. A command bit on the receiving side must also be TRUE. There are two real ways this can happen. The first is a retentive receive DB: a Down bit left set before power was lost is still there. The second is a value that the partner has not yet overwritten with fresh data. If the Down bit on the receiving side is left TRUE, the line Valve_Down := Rx.DownCmd AND D.PeerOK can command the valve down during those 3 seconds without anyone actually issuing the command.

Why tests T1-T6 never saw it

I did not deliver the first version without testing it. I wrote a self-test as a separate OB in a test copy, with tests T1 through T6 running on PLCSIM for every PLC. They all passed before delivery. The problem was in the first step of the self-test itself.

// step แรกของ self-test รุ่นเดิม
CASE T.Step OF
    0:  T.Wait.TON(IN := TRUE, PT := T#5S);   // รอให้ระบบ "นิ่ง" ก่อน
        IF T.Wait.Q THEN
            T.T1 := NOT D.PeerOK;              // ตรวจ PeerOK ครั้งแรกตรงนี้
            T.Step := 10;
        END_IF;
END_CASE;

I deliberately waited 5 seconds to make sure the 3-second timer had expired before checking that PeerOK really went low without a heartbeat. The result was a passing test, but no test watched the interval from 0 to 3 seconds. The problem window fell inside the interval when the test suite deliberately closed its eyes.

My wrong assumption was that testing all the steady states was enough: the link drops and PeerOK goes low; the link returns and PeerOK goes high. Both cases passed, so I concluded that the fail-safe worked. Yet the moment when the fail-safe matters most is startup, when nothing has any information yet.

I wrote a short simulation to compare the two approaches at my desk without opening the PLC simulator. It assumes a 10 ms scan and no heartbeat arriving at all for 5 seconds.

SCAN_MS, PT_MS = 10, 3000

def run(use_seen, hb_series):
    et, hb_last, seen, ok_scans = 0, False, False, 0
    for hb in hb_series:
        if hb != hb_last:
            seen = True                      # เคยเห็น heartbeat เปลี่ยนค่าแล้ว
        et = min(et + SCAN_MS, PT_MS) if hb == hb_last else 0
        q = et >= PT_MS                      # TON.Q
        hb_last = hb
        link_ok = (seen and not q) if use_seen else (not q)
        ok_scans += link_ok
    return ok_scans

never = [False] * 500                        # 5 วินาที ไม่มี heartbeat
print('รุ่นแรก  PeerOK เป็น TRUE', run(False, never), 'scan')
print('รุ่นแก้  PeerOK เป็น TRUE', run(True, never), 'scan')

The first version produced 299 scans, almost a full 3 seconds, with PeerOK TRUE and no evidence to support it. The fixed version produced 0. This simulation is only a desk model, not a measurement from the machine, but it shows that this kind of bug can be found by tracing the first scan, without waiting for the real machine to reveal it.

Fixing it with HbRcvd: see it before trusting it

The fix is to separate the two states that the first version treated as one: “not timed out yet” and “a heartbeat has been seen and has not timed out yet.” I added an HbRcvd bit that latches TRUE when the heartbeat first changes, then ANDed it with the original condition.

// รุ่นแก้ - ฝั่ง Station PLC
IF Rx.Hb <> D.HbPrev THEN
    D.HbRcvd := TRUE;
END_IF;
D.HbTimer.TON(IN := (Rx.Hb = D.HbPrev), PT := T#3S);
D.HbPrev := Rx.Hb;
D.PeerOK := D.HbRcvd AND NOT D.HbTimer.Q;

Valve_Down := Rx.DownCmd AND D.PeerOK;
Valve_Up   := Rx.UpCmd OR NOT D.PeerOK;

// ฝั่ง Main PLC ใช้ PeerOK แบบเดียวกัน
Prox := Rx.Prox OR NOT D.PeerOK;      // ลิงก์หาย = ถือว่ามีชิ้นงาน

There is one more thing to check when applying this approach: HbRcvd must return to FALSE every time the CPU starts again. If the DB holding it is configured as retentive, this bit will remain TRUE across a power loss, and the same hole will return in another form. Open that DB's retain settings and check them directly to be sure.

Test T0 checks from the first scan

Fixing the code was not the end of it, because the original test suite was still closing its eyes during the same interval. Run the original suite against either the first version or the fixed version, and both pass. A test suite that cannot distinguish incorrect behavior from correct behavior effectively has no test for this issue. I therefore added T0, which works in the opposite direction to the other tests: it starts as passing and fails immediately if it sees something that should not happen.

// T0: ระหว่างที่ยังไม่มี heartbeat PeerOK ต้องไม่เคยเป็น TRUE แม้แต่ scan เดียว
IF NOT T.Init THEN
    T.T0   := TRUE;
    T.Step := 0;
    T.Init := TRUE;
END_IF;

IF T.Step = 0 AND D.PeerOK THEN
    T.T0 := FALSE;                 // latch ตก แล้วไม่กลับขึ้นอีก
END_IF;

// ... ข้ออื่นทำงานตาม step เดิม ...

T.AllPass := T.Done AND T.T0 AND T.T1 AND T.T2 AND T.T3
             AND T.T4 AND T.T5;

This test runs every scan while Step is still 0, which is the interval before the self-test has fed in any simulated heartbeat. If PeerOK becomes TRUE for even one scan, T0 fails and stays failed until the test ends. It therefore catches the fault from scan 1, without relying on a tester happening to look at the right moment.

After the fix, T0 through T5 all passed on the simulator for all 4 PLCs involved. As for the first version, tracing the logic of T0 shows that it would fail from scan 1 because PeerOK is TRUE immediately. A useful test must fail incorrect code, not merely pass correct code.

Already delivered: going back to fix the whole set

The least enjoyable part was that the first version had already been delivered. I chose the most direct approach: rebuild all 12 files as version two, move the first version into the _old folder, and tell the recipient plainly what was wrong with the first version, why it was wrong, and which set was correct. I did not sidestep it by calling it an “improvement.”

1. Find every occurrence of the faulty pattern, rather than fixing only the file where it was found. Search all generated source files.
2. Fix the single source of truth, the code generator, and regenerate the whole set. Do not patch individual files by hand.
3. Add a test that fails the old version before trusting that the new version passes.
4. Run the test suite again on every PLC, not just the ones thought to be affected.
5. Move the old version out of the delivery folder immediately. Do not leave the two versions mixed together.
6. Notify the recipient in writing, specifying which set of files must not be used.

The first step can be done by searching the generated source text for every line that assigns PeerOK without HbRcvd on the same line.

Get-ChildItem .\changesets -Recurse -Filter *.scl |
    Select-String -Pattern 'PeerOK\s*:=' |
    Where-Object { $_.Line -notmatch 'HbRcvd' } |
    ForEach-Object { '{0}:{1}  {2}' -f $_.Filename, $_.LineNumber, $_.Line.Trim() }

The desired result after the fix is no lines printed. As for separating out the old version, I did that before placing the new files in the folder, so there would be no point when the delivery folder contained both versions.

$old = '.\deliver\_old'
New-Item -ItemType Directory -Force -Path $old | Out-Null
Get-ChildItem .\deliver -File | Move-Item -Destination $old
# จากนั้นค่อยคัดลอกรุ่นสองที่ build ใหม่เข้ามา แล้วนับจำนวนไฟล์ให้ครบ 12
(Get-ChildItem .\deliver -File).Count

One more issue: a heartbeat that would not toggle on S7-1500

There was another issue I found while testing the same job: the heartbeat generator itself. In the first version, I used a pattern dating back to the S7-300 era, feeding the timer's Q back into its own IN so it would reset itself every cycle.

// สำนวนเดิม ไม่แกว่งบน S7-1500
T(IN := NOT T.Q, PT := T#500MS);
IF T.Q THEN
    HbTx := NOT HbTx;
END_IF;

On PLCSIM for S7-1500, the heartbeat from this code stayed constant and never toggled. The reason is that the IEC timer on S7-1500 updates Q when the instance is called or read, so the moment when Q would be TRUE for the IF line to see never occurs as the old pattern expects. The working approach is to run the timer with IN explicitly TRUE, then explicitly reset it when Q goes high.

// แบบที่ใช้ได้
HbOsc.TON(IN := TRUE, PT := T#500MS);
IF HbOsc.Q THEN
    HbTx := NOT HbTx;
    HbOsc.TON(IN := FALSE, PT := T#500MS);   // reset ตัวจับเวลา
END_IF;
Tx.Hb := HbTx;

This relates to the first issue because if the sender's heartbeat does not toggle, a receiver using the fixed PeerOK will never see a healthy link, which is the correct behavior. A receiver using the first version, however, will see a healthy link for 3 seconds before it goes low. One bug therefore partly masked the symptoms of the other. I wrote about monitoring stalled work while the process is still alive from another angle in the article on why restarting does not fix a hung daemon. The same principle applies: the watchdog must look for evidence that work is actually progressing, not merely the absence of a failure signal.

The rules I took away, and what remains untested

The conclusion is that every fail-safe I write from now on must have tests for at least three moments, not two.

The second rule is never to put a wait before the first test without anything watching that wait. If a wait is necessary, a test that latches failure, like T0, must always cover that interval.

What I took away from this job beyond the code is that passing tests only tell me there were no problems during the intervals they watched. They say nothing about the intervals they did not watch. And those unwatched intervals are often the ones I deliberately skipped to make the tests easier to write. If you have a watchdog in a machine that looks like the first version, try tracing the first scan once. It takes only a few minutes. If I have explained any of the timer behavior incorrectly, I will come back and correct it later.