The NVR Kept Beeping, Then Fell Silent When the Disks Failed

I heard about an NVR that beeped almost all day. The alarms came in bursts until the people sitting nearby could not stand it. The technician who had checked it earlier concluded that it was videoloss because the cameras kept disconnecting, then went into the menu and disabled audible alarms for the whole unit. That was the end of it for two months, until someone requested recorded footage and found that both hard disks were showing Exception status, with no one knowing how long they had been that way, because the unit had never made a sound. I went through the whole setup, starting with what was actually causing the beeping, then fixed the notification policy instead of pressing mute.

The IP addresses in this article are examples, but the commands and XML structures are real, and all the work was done on equipment I maintain, using the administrator account provided for the job.

The beeping and the event behind it are different things

An NVR uses the same alarm sound regardless of the cause. Someone standing in front of it cannot tell whether a camera has disconnected, a disk has failed, or IP addresses are conflicting. Everyone ends up guessing from what they see on the screen, and since a channel happened to be flickering at the time, they concluded that it was videoloss.

To find out what actually happened, I had to read the event log on the unit, not guess from the sound. Looking back through it, the most frequent entry was ipconflict, not videoloss. There really were videoloss events, but they were a consequence: when IP addresses conflicted, the NVR could not connect to that camera anyway.

Read all the triggers first, before touching the interface

The NVR menu offers only one way to silence alarms: mute the whole unit. ISAPI provides much finer control because it treats everything as a separate trigger for each event and each channel. Each trigger has its own list of notification methods: sound the buzzer, record, send to the center, or send email.

export NVR_HOST=10.10.10.20
export NVR_USER=admin
export NVR_PASS='<PASSWORD>'

curl -s --digest -u "$NVR_USER:$NVR_PASS" \
  "http://$NVR_HOST/ISAPI/Event/triggers" | head -40

The response was very long because it covered every event multiplied by every channel. I saved it to a file and filtered out just the trigger names first to see what was available.

curl -s --digest -u "$NVR_USER:$NVR_PASS" \
  "http://$NVR_HOST/ISAPI/Event/triggers" \
  | grep -o '<id>[^<]*</id>' | sed 's/<[^>]*>//g' | sort | uniq

The names follow the format <eventType>-<channel>, such as videoloss-1 ipconflict-1 diskerror-1 diskfull-1 recordingfailure-1 illaccess-1. Only a few mattered for this job, but I needed to see them all to know what could be causing the beeping.

curl -s --digest -u "$NVR_USER:$NVR_PASS" \
  "http://$NVR_HOST/ISAPI/Event/triggers/ipconflict-1"

<EventTrigger version="2.0" xmlns="http://www.hikvision.com/ver20/XMLSchema">
  <id>ipconflict-1</id>
  <eventType>ipconflict</eventType>
  <eventDescription>ipconflict Event trigger</eventDescription>
  <EventTriggerNotificationList>
    <EventTriggerNotification>
      <id>beep</id>
      <notificationMethod>beep</notificationMethod>
      <notificationRecurrence>beginning</notificationRecurrence>
    </EventTriggerNotification>
    <EventTriggerNotification>
      <id>center</id>
      <notificationMethod>center</notificationMethod>
      <notificationRecurrence>beginning</notificationRecurrence>
    </EventTriggerNotification>
  </EventTriggerNotificationList>
</EventTrigger>

Seeing beep in this list answers the question of where the sound comes from. center means sending to the central software. I kept that because it does not bother anyone, and it is something I want available for later review.

Fix the policy: remove beep per event instead of muting the whole unit

The fix is to PUT the original XML back with only the beep block removed, leaving every other notification method in place. I do this one trigger at a time, for those I have decided do not warrant an audible alarm.

curl -s --digest -u "$NVR_USER:$NVR_PASS" -X PUT \
  "http://$NVR_HOST/ISAPI/Event/triggers/ipconflict-1" \
  -H 'Content-Type: application/xml' \
  --data-binary @ipconflict-1-nobeep.xml

Doing each one manually is manageable for a single unit, but with several units, it is worth writing a script. I use this approach: read the existing configuration, remove the block containing beep by cutting at the matching tags, then send it back.

import os, re, requests
from requests.auth import HTTPDigestAuth

B = "http://%s/ISAPI" % os.environ["NVR_HOST"]
A = HTTPDigestAuth(os.environ["NVR_USER"], os.environ["NVR_PASS"])
BEEP = re.compile(r"\s*<EventTriggerNotification>(?:(?!</EventTriggerNotification>).)*"
                  r"<notificationMethod>beep</notificationMethod>.*?"
                  r"</EventTriggerNotification>", re.S)

def set_beep(trigger_id, want_beep):
    xml = requests.get("%s/Event/triggers/%s" % (B, trigger_id), auth=A, timeout=8).text
    has = "<notificationMethod>beep</notificationMethod>" in xml
    if has == want_beep:
        return "unchanged"
    if has and not want_beep:
        new = BEEP.sub("", xml)
    else:
        blk = ("<EventTriggerNotification><id>beep</id>"
               "<notificationMethod>beep</notificationMethod>"
               "<notificationRecurrence>beginning</notificationRecurrence>"
               "</EventTriggerNotification>")
        new = xml.replace("<EventTriggerNotificationList>",
                          "<EventTriggerNotificationList>" + blk, 1)
    r = requests.put("%s/Event/triggers/%s" % (B, trigger_id), auth=A,
                     data=new.encode("utf-8"),
                     headers={"Content-Type": "application/xml"}, timeout=8)
    return "ok" if "<statusCode>1</statusCode>" in r.text else r.text[:120]

The policy I set afterward was to sound the alarm only for events that mean "footage is being lost." That left two: diskerror and recordingfailure. Everything else is logged and sent to the center, but makes no sound.

SILENT = ["ipconflict-1", "videoloss-1", "videoloss-2", "illaccess-1", "nettbroken-1"]
LOUD   = ["diskerror-1", "diskfull-1", "recordingfailure-1"]

for t in SILENT:
    print(t, set_beep(t, False))
for t in LOUD:
    print(t, set_beep(t, True))

The real cause was cameras using DHCP with conflicting IP addresses

Muting only masks the symptoms. The cause still needs fixing. All the cameras in this system were set to DHCP during installation, and the router's allocation range overlapped with IP addresses someone later assigned statically to other devices. When the router rebooted or several leases expired at once, it could assign an address already in use.

My fix was not to switch the cameras to static addresses, because configuring them one by one on a ladder is something no one wants to repeat. Instead, I reserved IP addresses by MAC on the router. The cameras still use DHCP; they just get the same address every time. I then shrank the dynamic allocation range until it no longer overlapped with the range used for static addresses.

The cost of silencing everything

During the two months the unit was muted, both hard disks entered Exception status within a short time of each other. As a result, an entire period of footage that should have been available was missing. Nobody knew until the day they needed to watch it.

The two disks failing close together was not much of a coincidence. Disks in recorders are often bought together, from the same batch, and both write around the clock, so they reach the end of their service lives around the same time. Anyone who thinks two disks are safer than one needs to reconsider if both came from the same box.

Monitor disk status through ISAPI instead of waiting for a beep

After this incident, I stopped relying on the unit's buzzer as the main line of warning and wrote a disk monitor to query it every hour instead. The endpoint is /ISAPI/ContentMgmt/Storage, which returns the status of every disk along with its capacity and free space.

curl -s --digest -u "$NVR_USER:$NVR_PASS" \
  "http://$NVR_HOST/ISAPI/ContentMgmt/Storage" \
  | grep -E '<(id|status|capacity|freeSpace|hddName)>'

import re, requests

def hdd_status():
    xml = requests.get(B + "/ContentMgmt/Storage", auth=A, timeout=8).text
    out = []
    for blk in re.findall(r"<hdd>(.*?)</hdd>", xml, re.S):
        g = lambda t: (re.search("<%s>([^<]*)</%s>" % (t, t), blk) or [None, ""])[1]
        out.append(dict(id=g("id"), name=g("hddName"), status=g("status"),
                        cap=int(g("capacity") or 0), free=int(g("freeSpace") or 0)))
    return out

for h in hdd_status():
    # ok / unformatted / error / mismatch / sleeping — อะไรที่ไม่ใช่ ok คือต้องแจ้งคน
    if h["status"] != "ok":
        print("ALERT hdd", h["id"], h["name"], h["status"])

I run this on a small machine on the same network and send alerts through a channel people actually read. The advantage is that it operates independently of the NVR. If the NVR itself stops responding one day, I will know immediately too, which is something the buzzer on the unit could never tell me anyway.

The H.265 trap that throws off retention estimates

The last thing I found on the same job concerned calculating how many days of footage the available disks could hold. Everyone read H.265 on the NVR interface and estimated storage use accordingly. The numbers did not match. When I queried the cameras directly, I found that several cameras connected through ONVIF were actually still sending H.264. The value shown on the NVR interface was what the NVR had requested when the channel was added, not what was flowing through it.

# ถามที่ตัวกล้องเอง ไม่ใช่ดูจากหน้า NVR
curl -s --digest -u "admin:$CAM_PASS" \
  "http://10.10.10.31/ISAPI/Streaming/channels/101" \
  | grep -E '<(videoCodecType|videoResolutionWidth|constantBitRate|maxFrameRate)>'

Trusting the wrong value threw the retention estimate off by almost a factor of two, because H.264 uses roughly twice the space at the same resolution and quality. My approach is to avoid estimating from the codec altogether and multiply the actual bitrate directly.

The simple calculation is to divide the bitrate in megabits per second by eight to get megabytes per second, multiply by eighty-six thousand four hundred seconds to get the daily amount per channel, then multiply by the number of channels. Four megabits per second is about forty-two gigabytes per day per channel. Sixteen channels is six hundred seventy-five gigabytes per day. Twelve terabytes of usable space therefore holds about eighteen days, not the thirty-six days previously quoted to the customer.

The alarm and monitoring policy I use

1. Before touching anything, retrieve the XML for every trigger and save it to files. This provides both a starting point for rollback and documentation of the system.
2. Review the event history before deciding what caused the beeping. Do not guess from what happens to be on the screen.
3. Remove beep only from triggers I have decided do not warrant it. Always keep logging and sending to the center enabled.
4. Keep audible alarms only for events that mean footage is being lost: disk problems and recording failures.
5. Fix the cause by reserving IP addresses by MAC and shrinking the DHCP dynamic allocation range, without setting static addresses one device at a time.
6. Run a disk monitor separately from the NVR, query it every hour, and have it alert when the NVR itself does not respond too.
7. Calculate retention from the actual bitrate read from the cameras, not from the codec shown on the NVR interface.

Personally, I think most camera system problems are not about bad equipment. They are about systems making so much noise that people stop listening, and once they stop listening, I am left with a system no one is watching at all. The right fix is to reduce the noise to things that actually require someone to get up and act, not silence everything and hope nothing happens. Anyone adapting this policy to their own site can change the SILENT and LOUD lists as needed, but do not leave LOUD empty.

Still on my plan is storing the XML for every trigger in git so I can see who changed the policy and when, along with an automated check to compare the settings on the unit against the agreed policy. After a technician goes in to do other work, these settings can always revert to their previous values. If I have left anything out or made a typo, I will come back and fix it later.