Registrar and Billing NS Mismatch: Audit All Three Sides First

I needed to move DNS for a group of domains to a new DNS provider. The final step was to change the nameservers in the billing system (the one connected to the registrar through a module). Normally, it takes ten seconds: enter ns1 and ns2, click Save, done. This time, though, clicking it returned Wrong response from the server. Unknown error, along with a red banner saying [] operator not supported for strings. I kept digging until I found that the problem was not at the registrar at all. It was leftover data in the billing system's own database. Once I opened it up and looked closely, I found that the values shown in the billing panel had differed from what the registrar actually held for years, without anyone knowing.

I have replaced all values in this article with fictional names and example IPs. The symptoms and investigation, however, are from work I did myself.

Symptoms: Only Some Domains Failed, Others Worked Fine

What confused me at first was that it was not a system-wide failure. Newly registered domains could change NS normally, while older domains that had been through several migrations threw an error every time. If the whole system had failed, I would have blamed the module. But since only some domains failed, the variable was the data for that domain, not the code.

I opened the billing database in read-only mode and pulled the two nameserver-related columns to compare failing domains with working ones.

-- อ่านอย่างเดียว: ดูค่า NS ที่ระบบบิลลิ่งเก็บไว้ กับ glue IP ที่ตกค้าง
SELECT name, status, nameservers, nsips
FROM   billing_domains
WHERE  status = 'Active'
ORDER  BY (nsips <> '') DESC, name;

The result was clear. Domains that saved successfully had nameservers set to |||| (four separators represent five empty slots) and an empty nsips. The failing domains had only a single | left in nameservers, while nsips contained two IPs separated by a pipe, such as 203.0.113.20|203.0.113.21. These were glue IPs for private nameservers from the old host several years earlier.

The error [] operator not supported for strings comes from an old PHP idiom that writes $var[] = ... to a variable that happens to be a string instead of an array. PHP 5 let it pass; PHP 7 and later throw a fatal error immediately. Simply put, the module's NS update function did not account for the glue IP field containing a value.

Why I Could Not Fix the Code

Normally, with a bug like this, I would just open the file, add a line with is_array(), and be done. But this registrar module came from the vendor encoded with ionCube. I could not inspect, edit, or patch it. There was no telling when the vendor would release a new version, and, crucially, the module's other functions (domain renewal, registration, and status retrieval) still worked normally. Only the NS update function was broken.

The Workaround: Call the Registrar API Directly

The registrar already provided an HTTP API, and updating NS was just one command. I wrote a small script to call it directly, bypassing the broken module. Two commands were enough: getns to read the current values, and setns to set new ones, followed by another read to verify them.

#!/usr/bin/env python3
"""ns_tool.py - อ่าน/ตั้ง nameserver ผ่าน API ของ registrar ตรงๆ
   credential อ่านจาก environment เท่านั้น ไม่ฝังในไฟล์"""
import os, sys, urllib.parse, urllib.request
import xml.etree.ElementTree as ET

API = "https://api.registrar.example.com/xml.response"
NS  = {"n": "http://api.registrar.example.com/xml.response"}

def call(command, **extra):
    params = {
        "ApiUser":  os.environ["REG_API_USER"],
        "UserName": os.environ["REG_API_USER"],
        "ApiKey":   os.environ["REG_API_KEY"],     # มาจาก env เท่านั้น
        "ClientIp": os.environ["REG_CLIENT_IP"],   # ต้องเป็น IP ที่อยู่ใน allowlist
        "Command":  command,
    }
    params.update(extra)
    url = API + "?" + urllib.parse.urlencode(params)
    with urllib.request.urlopen(url, timeout=30) as r:
        return ET.fromstring(r.read())

def split_domain(domain):
    sld, _, tld = domain.partition(".")
    return sld, tld

def getns(domain):
    sld, tld = split_domain(domain)
    root = call("registrar.domains.dns.getList", SLD=sld, TLD=tld)
    return [n.text for n in root.iter("{*}Nameserver")]

def setns(domain, servers):
    sld, tld = split_domain(domain)
    call("registrar.domains.dns.setCustom", SLD=sld, TLD=tld,
         Nameservers=",".join(servers))
    return getns(domain)          # อ่านซ้ำเพื่อยืนยันว่าเปลี่ยนจริง

if __name__ == "__main__":
    op, domain, *rest = sys.argv[1:]
    print(getns(domain) if op == "getns" else setns(domain, rest))

The important point is that authentication for this kind of registrar does not check only the key. It checks "key + source IP on the allowlist." The allowed IP was the IP of the server running the billing system, not my home machine (my home connection has a dynamic IP that changes daily, so adding it to the allowlist was not practical). That meant this script had to run on the server. I sent it through stdin to run without leaving a file on the machine.

# ส่งสคริปต์เข้าไปรันบนเซิร์ฟเวอร์ผ่าน stdin ไม่เหลือไฟล์ค้าง
cat ns_tool.py | ssh [email protected] \
  "REG_API_USER=\$REG_API_USER REG_API_KEY=\$REG_API_KEY \
   REG_CLIENT_IP=\$REG_CLIENT_IP python3 - getns example.com"

Audit All Three Sides Before Changing Anything

Once I had a tool to read values from the registrar, I did not rush to fix the failing domains. The bigger question was, "What about the others? Do they actually match?" There are three sides here that should agree, but nothing forces them to.

The left side is the legal truth. The middle is the truth the team sees on screen. The right side is the truth end users actually encounter when opening a website. All three can drift independently. I wrote an audit tool to compare them one domain at a time.

import time, csv
LIMIT_PER_MIN = 20                    # registrar จำกัดไว้ 20 คำสั่งต่อนาที
SLEEP = 60.0 / LIMIT_PER_MIN + 0.3    # หน่วง 3.3 วิ/ครั้ง เผื่อ jitter

rows = []
for d in active_domains_from_billing():      # ดึงจากฐานบิลลิ่ง (อ่านอย่างเดียว)
    try:
        real = getns(d["name"])              # ฝั่ง registrar
    except Exception as e:
        rows.append({"domain": d["name"], "verdict": "NOT-IN-REGISTRAR",
                     "detail": str(e)[:80]})
        time.sleep(SLEEP)
        continue

    billing = [x for x in (d["nameservers"] or "").split("|") if x]
    if not billing:
        verdict = "BILLING-EMPTY"            # ฐานบิลลิ่งว่าง แต่ registrar มีค่า
    elif sorted(billing) != sorted(real):
        verdict = "MISMATCH"
    else:
        verdict = "OK"

    rows.append({"domain": d["name"], "verdict": verdict,
                 "registrar": ",".join(real), "billing": ",".join(billing),
                 "glue_left": d["nsips"] or ""})
    time.sleep(SLEEP)

with open("ns_audit.tsv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0]), delimiter="\t")
    w.writeheader(); w.writerows(rows)

The 3.3-second delay per call is not there for decoration. This registrar limits requests to 20 commands per minute. Send them faster, and it starts returning errors without explaining why. I could then mistake a domain for being broken when I was merely being throttled. For a sweep of around a hundred domains, I can let it run for ten minutes and make coffee while I wait.

What the Audit Found Was Unsettling

The report file looked roughly like this. I opened it in a spreadsheet viewer and sorted by verdict, which separated the problems into clear groups.

$ column -t -s $'\t' ns_audit.tsv | sort -k2
domain           verdict            registrar                    billing
alpha.example    OK                 ns1.dnshost.example,ns2...   ns1.dnshost.example,ns2...
bravo.example    BILLING-EMPTY      ns1.dnshost.example,ns2...
delta.example    BILLING-EMPTY      ns3.otherdns.example,ns4...
echo.example     MISMATCH           ns3.otherdns.example,ns4...  ns1.oldhost.example,ns2...
foxtrot.example  NOT-IN-REGISTRAR   -                            ns1.oldhost.example,ns2...

From most to least serious, the first finding was that the nameserver field in the billing system could not be trusted at all. For 20 of the 28 domains still marked Active, it was empty or ||||, even though the registrar held a complete set of actual NS values. That meant anyone who had looked at the billing page and concluded, "This domain has no NS configured yet," had been wrong all along. The only way to get the truth was to ask the registrar.

The second finding was that the script I wrote to set NS through the API did not update the billing database. Domains I had already fixed showed the correct new values at the registrar, while billing still showed the old ones. I was creating a fresh mismatch myself unless I followed up and cleaned it up.

The third finding was that nine domains were still Active in our system, but their NS pointed entirely to other DNS providers. The customers had moved their DNS themselves without telling anyone, which was fine. But if I did not know they had moved, the day I changed a zone on our side and their website did not change, I would spend three hours investigating the wrong place.

Clear the Leftovers and Sync Back, but Back Up First

Only after I had the full picture did I start making changes. My sequence always began with a backup, followed by overwriting the values with the actual ones from the registrar and clearing the leftover glue IPs at the same time.

1. Dump the original rows as SQL first, so I can roll back with one command.
2. Write the actual NS values from the registrar into the billing database, adding separators to fill all five slots the module expects.
3. Clear nsips for domains that no longer use private nameservers.
4. Return to the billing panel and try clicking Save on one domain. It must no longer throw an error.

-- 1) สำรองก่อนแตะอะไรทั้งนั้น
SELECT CONCAT('UPDATE billing_domains SET nameservers=', QUOTE(nameservers),
              ', nsips=', QUOTE(nsips), ' WHERE id=', id, ';')
FROM   billing_domains
WHERE  status = 'Active';

-- 2) เขียนค่าจริงกลับ (ทำทีละแถวจากไฟล์ออดิต ไม่ใช่ UPDATE รวดเดียว)
UPDATE billing_domains
SET    nameservers = 'ns1.dnshost.example|ns2.dnshost.example|||',
       nsips       = ''
WHERE  name = 'example.com'
  AND  status = 'Active';

I want to emphasize step 2: do not write a single UPDATE that overwrites every row, because each domain can have different NS values (many DNS providers assign a pair of randomly named NS to each zone). I have to work through the audit file row by row and touch only rows where I can confirm that the registrar has actual values. Rows for which the registrar has no data (expired, transferred-out, or closed domains) should be left alone, because there is no source of truth to compare them against.

After the run, there were zero Active rows with leftover glue IPs, and the six domains that had previously thrown errors on Save could save normally in the billing panel again. The data that caused the ionCube-encoded code to blow up was gone.

Keep It from Coming Back

A one-time fix was not enough, because the real cause was that nobody was watching for mismatches. I scheduled this audit to run periodically and send a report only when it found discrepancies. If everything matched, it stayed quiet, without sending annoying weekly emails.

# รายสัปดาห์ เช้าวันจันทร์ ยิงเฉพาะตอนเจอ MISMATCH
15 7 * * 1 cd /opt/ns-audit && /usr/bin/python3 ns_audit.py \
  --out /var/log/ns-audit/$(date +\%F).tsv --alert-only >> /var/log/ns-audit/run.log 2>&1

Personally, I think the most expensive lesson from this job was not the old PHP bug in a module I could not patch. It was that I had trusted the screen in our own system all along without ever comparing it with the actual endpoint. If you manage domains for customers, try running a sweep. I think you will find at least one. If I have skipped over an explanation anywhere, or another registrar has an API that looks different from this, I will come back and add to the article later.