Egress Firewall Breaks Domain Checks — Tests as root Pass

I look after a domain ordering system, and someone reported that its availability check page had stopped working. Every name entered came back as unavailable. I tried making up long names that surely nobody would have registered, and they still came back as unavailable. That symptom immediately suggested a problem with the check not working at all, rather than with the data, and the code was treating a failed check as an unavailable name.

Before getting into it, I want to make clear that this is separate from the three firewall articles I wrote earlier. Those were all about inbound traffic: who can connect to us, blocking brute force attempts, unblocking the office, and controlling access with GeoIP. This article is about outbound traffic: who our machine can connect to, which involves an entirely different set of rules, configuration file, and testing method.

I have replaced all values in this article with examples, including usernames, hostnames, and destination addresses. The troubleshooting sequence is the actual one.

Symptoms: every name shows as unavailable

The first thing I did was distinguish a data problem from a connection problem. The quickest way was to find a name I knew was available and one I knew was unavailable, then see whether the system gave different answers.

It gave the same answer for both. That meant the code had never received a response from the destination at all. It was simply falling back to its default value, which in this system was "unavailable," because the designer thought it was safer to say unavailable and prevent an order than to say available and let a customer order a name that could not actually be registered.

Which port does the domain check use, and which user runs it?

These two questions need answers before touching anything, because getting either answer wrong makes every subsequent test misleading.

First, most ordering systems check domain availability through an old protocol that runs on TCP port 43, not 80 or 443. Most people assume it is an API over HTTPS because everything runs on 443 these days, so nobody checks that port.

Second, the code opening the socket runs as the php-fpm pool user, not root or the user we log in with over SSH.

# หาให้แน่ว่าใครเป็นเจ้าของ process ที่รันโค้ดนี้จริง
ps -o user:16,pid,cmd -C php-fpm8.2 | head
# หรือถามจาก pool config ตรงๆ
grep -E '^(user|group)' /etc/php/8.2/fpm/pool.d/app.conf

The trap: tests as root pass

This is the core of the article. If you read only one section, read this one.

A Linux firewall can apply rules based on the process owner through a module that matches the uid of the user opening the socket. Most packaged rulesets give root a shortcut in a separate subchain. The reasoning is understandable: system tools need to be able to fetch things at any time, or updates will fail.

The result is that when we SSH in and test as root, everything passes. We can connect to any port, which proves nothing about the process actually running the app.

# ผ่าน — และไม่มีความหมายอะไรทั้งสิ้น
exec 3<>/dev/tcp/203.0.113.43/43 && echo OK && exec 3<&-

A decisive test

My approach was to compare two connections made as the same user, to destinations differing only in port number. If one passed and the other failed, that meant the issue was port filtering rules, rather than network routing or DNS.

# ขาอ้างอิง — พอร์ตที่รู้ว่าอนุญาตอยู่แล้ว
sudo -u appuser bash -c 'exec 3<>/dev/tcp/203.0.113.43/443 && echo "[OK] 443"'

# ขาที่สงสัย — พอร์ตเดียวกับที่แอปใช้ ปลายทางเดียวกันเป๊ะ
sudo -u appuser bash -c 'exec 3<>/dev/tcp/203.0.113.43/43  && echo "[OK] 43"'
# bash: connect: Connection refused

That settled it. Same destination, same user, same time, with only the port number differing. The only remaining variable was the port filtering rules.

To confirm this at the application level and rule out the shell, make the connection directly from the same interpreter the app uses.

<?php
// รันด้วย: sudo -u appuser php probe.php
$h = @fsockopen('whois.registry.example', 43, $errno, $errstr, 5);
if (! $h) {
    fwrite(STDERR, "[ERR] 43 -> $errno $errstr\n");
    exit(1);
}
fwrite($h, "example-name-that-nobody-owns.test\r\n");
echo substr(stream_get_contents($h), 0, 200), "\n";
fclose($h);

Read the error to identify who rejected the connection

The message returned when a connection fails tells us more than we might expect. Here is my summary of the common ones.

The most misleading part is the difference between IPv4 and IPv6 on the same machine. If there are two separate outbound port lists for the address families and we add the port to only one, connections will sometimes pass and sometimes fail, depending on which address family the resolver returns first that time.

# บังคับทดสอบทีละตระกูลไปเลย จะได้ไม่ต้องเดา
sudo -u appuser bash -c 'exec 3<>/dev/tcp/203.0.113.43/43 && echo "[OK] v4"'
sudo -u appuser bash -c 'exec 3<>/dev/tcp/2001:db8::2b/43 && echo "[OK] v6"'
# v4: Connection refused   v6: Network is unreachable

These two messages have different causes. The first means a rule rejected the connection; the second means the machine has no outbound route for that address family at all. Skimming them and treating them as the same problem leads to fixing the wrong thing.

Fix the outbound port lists for both address families

Once I knew the outbound rules were responsible, all that remained was to add the port to the lists. The configuration file for the packaged ruleset I use has two relevant lines, one for each address family.

# /etc/csf/csf.conf — ลิสต์พอร์ตขาออกที่อนุญาต
TCP_OUT  = "20,21,22,25,43,53,80,110,113,143,443,465,587,853,993,995,2222"
TCP6_OUT = "20,21,22,25,43,53,80,110,113,143,443,465,587,853,993,995,2222"

Always make a backup before editing, because a mistake in this file can lock you out of the machine.

ts=$(date +%Y%m%d-%H%M%S)
sudo install -d -m 0700 "/root/csf-backup-$ts"
sudo cp -a /etc/csf/csf.conf "/root/csf-backup-$ts/csf.conf"

sudo sed -ri 's/^(TCP6?_OUT\s*=\s*")([^"]*)"/\1\2,43"/' /etc/csf/csf.conf
grep -E '^TCP6?_OUT' /etc/csf/csf.conf     # อ่านด้วยตาก่อน อย่าเพิ่งรีโหลด
sudo csf -r

# ย้อนกลับถ้าพลาด
# sudo cp -a "/root/csf-backup-$ts/csf.conf" /etc/csf/csf.conf && sudo csf -r

After reloading, repeat the same tests as the same user. Both connections must return OK, then test through the actual web page as the final step.

Why it disappeared during the migration

The next question was how something that had worked for years could go missing. The answer was during the operating system migration.

During a server migration, the person doing it focuses on things that visibly break: does the website load, does mail flow in and out, can the database be reached? The firewall configuration is often installed afresh using the new version's defaults, then the necessary inbound rules are added back, because missing inbound rules prompt immediate complaints.

Outbound port defaults, meanwhile, differ between versions, and unusual ports we added ourselves three years ago quietly disappear without anyone noticing, until a customer reports that a feature no longer works.

An outbound checklist after every server migration

I put together this checklist to run after moving a server, before declaring the migration complete.

1. List every feature in the system that needs an external connection, alongside the port it actually uses, not the port we think it uses — this requires reading the code or capturing actual traffic.
2. Compare that list with the outbound port lists in the configuration file, for both v4 and v6. Both must match.
3. Test every port on the list as the user that actually runs the app. Do not test as root or as the user we log in with over SSH.
4. Check that the application code distinguishes "could not connect" from "the destination says it does not exist." If it cannot, fix that first.
5. Set up a monitor that runs actual checks periodically and alerts when the results change, rather than just checking that the website returns 200.

Do not let "could not connect" mean "unavailable"

I want to close with something I think matters even more than the port itself, because even if the port is fixed today, an outage at the destination tomorrow will cause the same symptoms.

The root problem is that the code conflates a connection failure with a business result, which are entirely separate things. A domain check should have three possible results, not two.

enum NameStatus {
    case Available;     // ปลายทางตอบว่าไม่มีข้อมูล
    case Taken;         // ปลายทางตอบว่ามีข้อมูลแล้ว
    case Unknown;       // เช็คไม่สำเร็จ — ห้ามแปลเป็น Taken เด็ดขาด
}

// ฝั่งหน้าเว็บ
match ($status) {
    NameStatus::Available => 'ชื่อนี้ว่าง สั่งซื้อได้',
    NameStatus::Taken     => 'ชื่อนี้มีผู้ใช้แล้ว',
    NameStatus::Unknown   => 'ตรวจสอบไม่สำเร็จ กรุณาลองใหม่อีกครั้ง',
};

That third value is what makes the problem visible within the first minute instead of leaving it hidden for a day. It also helps the business, because telling a customer "The check failed, please try again" is tolerable, but telling them a name is unavailable when it is actually available simply drives them away from checkout, without us ever knowing how many customers we lost.

Personally, I still think blocking all outbound traffic and allowing individual ports is the right default. One incident like this should not make us switch to allowing everything. We just have to accept the cost: every time we add a feature that connects externally, or move a server, someone has to update the list. Choose whichever approach suits you, but if you choose to block by default, please use the checklist above as well, or this will certainly happen again. If I have left anything out or made a typo, I will come back and correct it later.