My Firewall Permanently Banned Googlebot and Hid Sites from Search

I was investigating why some sites on the server had an unusual drop in search traffic. I planned to look at robots.txt and the sitemap first, but before opening either, I tried checking a Googlebot IP against the server's firewall on a whim. It turned out to have been on the permanent ban list for three days. I put everything else aside and spent the day investigating. By the end, I had a system that runs daily to prevent it from happening again. I wanted to share it because I suspect plenty of other servers have the same problem without anyone knowing.

The domain names and server IPs in this article are fictional. The bot IP ranges are real because they are public information published by the providers themselves, and all the figures come from an actual server I manage.

First, check whether bots can actually reach the server

Here is a command I would like everyone to try right now. It takes ten seconds.

# ถามไฟร์วอลล์ตรง ๆ ว่ารู้จักไอพีนี้ไหม
csf -g 66.249.74.100

# ไล่หาช่วงไอพีของบอทในรายการแบนถาวร
grep -E '^(66\.249\.|40\.77\.167\.|52\.167\.144\.|157\.55\.39\.|207\.46\.13\.)' /etc/csf/csf.deny

# ดูรายการแบนชั่วคราวด้วย บางทีของจริงอยู่ตรงนี้
csf -t | grep -E '66\.249\.|40\.77\.'

This is the line I got back at the time.

66.249.74.100 # lfd: (BFM) mod_security1=25 - Sat Jun  9 17:35:02 2026

The value mod_security1=25 is at the heart of the whole issue. It means this IP had triggered 25 WAF alerts, so the brute-force protection system automatically banned it permanently. There was also an entry for a bot belonging to an SEO analysis service, banned six days earlier for the same reason.

The counter issuing the bans is not the one most people think

I spent quite a while working out "who issued the ban," because a DirectAdmin server has several counters that can ban IPs, and they are separate mechanisms. I went through every line in csf.deny and tallied its source.

# นับว่าแต่ละบรรทัดมาจากตัวนับไหน
awk -F'#' '{print $2}' /etc/csf/csf.deny | sed 's/ - .*//' | sort | uniq -c | sort -rn | head

# จำนวนบรรทัดทั้งหมด เทียบกับเพดานที่ตั้งไว้
grep -vc '^#' /etc/csf/csf.deny
grep '^DENY_IP_LIMIT' /etc/csf/csf.conf

Over four hundred of the five hundred lines came from a single counter: the one that tallies WAF alerts per IP. Once it reaches 25, it issues a permanent ban. The point is that it counts false positives and real attacks in the same bucket. The standard WAF rules are very strict. Even a slightly unusual-looking URL can count against a site.

The actual cause this time was a customer site whose tag-page URLs matched a pattern that one WAF rule treated as SQL injection. Every time Googlebot crawled a tag page, the counter went up by one. After a few days of crawling, it reached 25.

What makes this hard to spot is that the sites still work normally for everyone else. Nobody reports a problem. The search engine's tools only say the site is unreachable, and days can pass before anyone notices.

Lift the ban first, then check both places

Lifting the ban is straightforward, but a common mistake is to remove it from the file and assume the job is done, even though the actual blocking happens in the in-memory ipset.

# ปลดแบนถาวร (คำสั่งนี้จัดการทั้งไฟล์และ ipset ให้)
csf -dr 66.249.74.100

# ปลดแบนชั่วคราว ถ้ามี
csf -tr 66.249.74.100

# ยืนยันของจริงว่าไม่เหลือใน ipset แล้ว อย่าเชื่อแค่ไฟล์
ipset list chain_DENY 2>/dev/null | grep 66.249.74.100 && echo "[WARN] ยังอยู่" || echo "[OK] หลุดแล้ว"
csf -g 66.249.74.100

Once I had lifted the ban, I went back to fix the root cause in the WAF by adding an exception for that specific rule and that specific URL pattern on that site. I did not disable the entire rule across the server. This is the part that needs the most care, because frustration with false positives often leads people to disable a whole rule, which lets real attacks through too.

But that alone is not enough. If another rule produces a false positive next time, Googlebot will just get banned again. I needed to make sure bots I know are genuine can never be banned in the first place.

The permanent fix: fetch official bot IP ranges into the exemption lists daily

At first, I entered the IP ranges by hand, but I quickly realised that would not work in the long run. Providers keep changing their IP ranges, and they already publish official lists as downloadable files. I removed the manual entries and wrote a script to fetch them automatically.

1. Fetch each provider's official IP range list from its published URL: Google, Bing, Apple, DuckDuckGo, and the CDN ranges used by sites on the server.
2. Validate every line with a proper IP address library, not regex. Discard any line that cannot be parsed as a network.
3. Write to the exemption files inside a block marked with my own start and end markers. Never touch lines outside the block that other people have written.
4. Restart the daemon only when the content actually changes, rather than restarting it every day for no reason.
5. Scan both permanent and temporary ban lists. If an IP falls within a bot range, lift its ban automatically and record it in the log.

These three parts are the core of the script. I have included only the important sections.

SOURCES = {
    "googlebot":   {"kind": "json", "urls": ["https://developers.google.com/static/search/apis/ipranges/googlebot.json"], "min": 5},
    "bingbot":     {"kind": "json", "urls": ["https://www.bing.com/toolbox/bingbot.json"], "min": 3},
    "applebot":    {"kind": "json", "urls": ["https://search.developer.apple.com/applebot.json"], "min": 3},
    "duckduckbot": {"kind": "json", "urls": ["https://duckduckgo.com/duckduckbot.json"], "min": 1},
    "cloudflare":  {"kind": "text", "urls": ["https://www.cloudflare.com/ips-v4",
                                             "https://www.cloudflare.com/ips-v6"], "min": 10},
}

def valid_nets(cands):
    """แปลงทุกบรรทัดเป็น network object จริง อันไหนแปลงไม่ได้ทิ้งทันที"""
    nets = []
    for c in cands:
        c = c.strip()
        if not c:
            continue
        try:
            nets.append(str(ipaddress.ip_network(c, strict=False)))
        except ValueError:
            pass
    return sorted(set(nets), key=lambda n: (":" in n, n))

def get_vendor(vendor, cfg):
    """ดึงของใหม่ ถ้าล้มเหลวหรือได้น้อยผิดปกติ ให้ใช้สำเนาที่แคชไว้แทน"""
    cands = []
    try:
        for u in cfg["urls"]:
            raw = fetch(u)
            cands += parse_json_prefixes(raw) if cfg["kind"] == "json" else raw.splitlines()
        nets = valid_nets(cands)
    except Exception as e:
        log(f"WARN {vendor}: fetch failed ({e}) -> using cache")
        nets = []
    cache = os.path.join(CACHE_DIR, vendor + ".list")
    if len(nets) >= cfg["min"]:
        open(cache, "w").write("\n".join(nets) + "\n")
        log(f"OK   {vendor}: {len(nets)} ranges (fresh)")
        return nets
    if os.path.exists(cache):
        nets = [l.strip() for l in open(cache) if l.strip()]
        log(f"WARN {vendor}: fresh list too small -> cache ({len(nets)} ranges)")
        return nets
    log(f"FAIL {vendor}: no fresh data and no cache - vendor skipped this run")
    return []

For writing to the files, I use block markers and overwrite only what is inside them.

MARK_BEGIN = "# === BOT-WHITELIST-AUTO BEGIN (managed by script - DO NOT EDIT) ==="
MARK_END   = "# === BOT-WHITELIST-AUTO END ==="

def replace_block(path, block_lines):
    """แทนที่เฉพาะในบล็อก คืนค่า True ถ้าไฟล์เปลี่ยนจริง"""
    txt = open(path).read() if os.path.exists(path) else ""
    lines = txt.split("\n")
    block = [MARK_BEGIN] + block_lines + [MARK_END]
    if MARK_BEGIN in lines and MARK_END in lines:
        i, j = lines.index(MARK_BEGIN), lines.index(MARK_END)
        new = lines[:i] + block + lines[j + 1:]
    else:
        new = lines + ([""] if lines and lines[-1] else []) + block
    out = "\n".join(new)
    if not out.endswith("\n"):
        out += "\n"
    if out != txt:                       # เขียนเมื่อเนื้อหาต่างจริงเท่านั้น
        tmp = path + ".bwl-tmp"          # เขียนไฟล์ชั่วคราวแล้วสลับ กัน daemon อ่านไฟล์ครึ่ง ๆ กลาง ๆ
        with open(tmp, "w") as f:
            f.write(out)
        os.chmod(tmp, 0o600 if "csf" in path else 0o644)
        os.replace(tmp, path)
        return True
    return False

The final part automatically lifts existing bans. This matters more than it might seem, because IPs banned earlier will not be released automatically just because I add exemptions later.

for ip, kind in banned_ips():                       # ทั้ง csf.deny และ csf -t
    addr = ipaddress.ip_address(ip)
    hit = next((v for net, v in nets_typed if addr in net), None)
    if hit:
        flag = "-dr" if kind == "perm" else "-tr"
        subprocess.run(["csf", flag, ip], capture_output=True, text=True)
        log(f"UNBAN {ip} ({kind}) - matched {hit} range")

Results from the first run

The difference between the two lists is that I add CDN ranges only to the firewall's list, not the control panel's. CDN edge servers should have no reason to try guessing passwords for any service on the server. They need firewall exemptions because, if a site behind a CDN is attacked, the IP that gets banned belongs to the CDN edge. That would take down every site using that CDN on the server at once.

Set up cron, and do not forget the logs

# /etc/cron.d/bot-whitelist
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
10 5 * * * root /usr/local/sbin/bot-whitelist-update.py >> /var/log/bot-whitelist-update.log 2>&1

# /etc/logrotate.d/bot-whitelist
/var/log/bot-whitelist-update.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
    create 0640 root root
}

I scheduled it for ten past five in the morning. The reason was to avoid the top of the hour, when scheduled jobs across the server tend to pile up, and to run before the morning traffic arrives.

The grep trap that made the script fool itself

This is the part I want to stress most in this article, because the bug is very quiet and hard to find.

Besides the IP exemption lists, there is another file for reverse-hostname exemptions, used for bots whose domain names I can verify. I wrote the installation script to add these two lines.

# ที่อยากได้ในไฟล์ csf.rignore
.googlebot.com
.search.msn.com

My original code looked like this, which seemed perfectly fine.

# ผิด — เช็คว่ามีอยู่แล้วหรือยัง
grep -q '.googlebot.com' /etc/csf/csf.rignore || echo '.googlebot.com' >> /etc/csf/csf.rignore

I ran it, and nothing happened. The script said the entries already existed, but when I opened the file, neither line was there. The reason was that the template file installed with the software already contained .googlebot.com in an example comment at the top.

head -20 /etc/csf/csf.rignore
# Ignore file for reverse DNS lookups, e.g:
#
# .googlebot.com
# .search.msn.com
#

grep found it and reported that the entry existed, even though the matching line was a comment with no effect at all. To make matters worse, the dots in the search pattern also matched any character.

# ถูก — ยึดหัวบรรทัด และหนีอักขระจุดให้เรียบร้อย
grep -qE '^\.googlebot\.com[[:space:]]*$' /etc/csf/csf.rignore \
  || echo '.googlebot.com' >> /etc/csf/csf.rignore

grep -qE '^\.search\.msn\.com[[:space:]]*$' /etc/csf/csf.rignore \
  || echo '.search.msn.com' >> /etc/csf/csf.rignore

# ตรวจของจริง นับเฉพาะบรรทัดที่ไม่ใช่คอมเมนต์
grep -vE '^\s*(#|$)' /etc/csf/csf.rignore

The rule I have followed since that day is to always check the actual final result when writing a script that adds a line if it is missing, rather than trusting the value returned by grep. This problem does not occur with files managed through blocks, since I overwrite the entire block.

I also found the other side of the same coin during this inspection: grep failed to find something that was already there. When checking whether the server could read users' real IPs behind the CDN, I searched the main config file and found nothing, so I concluded it had not been configured. In fact, the control panel had put that configuration in a separate file that was already included. If I had added another copy, the web server would have reported duplicate settings, immediately breaking the configuration across the server. The lesson is to check the effective settings the program reads after combining all the files, rather than checking any one file.

Side effects nobody considers when enabling country blocking

One last issue surfaced during this inspection. I checked whether each search engine's bots could actually reach the server and found that one provider could not get through from a single IP, even though it did not appear in any ban list.

# ไอพีนี้ไม่ได้อยู่ใน csf.deny แต่เข้าไม่ได้ ทำไม
csf -g 5.255.253.10

# เพราะมันอยู่ในชุดไอพีของการบล็อกรายประเทศ
ipset list cc_ru | grep -cE '^5\.255\.|^87\.250\.224\.|^213\.180\.203\.|^141\.8\.'

# ดูว่าเปิดบล็อกประเทศไหนไว้บ้าง
grep -E '^CC_DENY|^CC_ALLOW' /etc/csf/csf.conf

The cause was inbound country blocking configured during installation. The reason at the time was to reduce password-guessing attempts, which it did. But the side effect nobody mentioned was that every bot belonging to a search engine based in that country was also blocked. Every one of its main IP ranges was in the blocked sets. That meant no site on the server could ever appear in that provider's search results.

This time, I removed that country from the list and reloaded the rules, then confirmed that the rule table really contained no remaining references to that set.

# แก้ค่าในไฟล์ แล้วโหลด allow/deny list ใหม่
sed -i 's/^CC_DENY_PORTS = "RU,KP,IR"/CC_DENY_PORTS = "KP,IR"/' /etc/csf/csf.conf
csf -ra

# ต้องได้ 0 คือไม่มีกฎไหนอ้างชุดนั้นอีกแล้ว
iptables-save | grep -c 'cc_ru'

# ชุดของประเทศที่ยังบล็อกอยู่ ต้องยังอยู่ครบ
ipset list -n | grep '^cc_'

# ยิงทดสอบว่าบอทเข้ามาได้แล้ว
csf -g 5.255.253.10

Fake bots exist too: do not let them through based on their claimed names

To finish, there is something to watch out for in the other direction. While going through the logs, I found plenty of traffic claiming to be Googlebot in its user-agent but coming from IP ranges that did not belong to Google at all.

This is why I grant exemptions only by official IP ranges published by the owners themselves and reverse hostnames. I never exempt by user-agent, because anyone can fake a user-agent in three seconds, whereas they cannot fake an IP if they need to receive the response.

To recap everything I did in this case: I lifted existing bot bans, added a targeted WAF exception to address the root cause, installed a script to fetch official IP ranges into the exemption lists daily with safeguards against my own mistakes, set up cron and logrotate, and reviewed country blocking to see who I was losing.

What I want to leave readers with is this: a server with several layers of automatic protection will always have more than one mechanism that can issue bans. When something cannot get in, do not stop at checking one file. Trace every mechanism to find out which one issued the ban. For anyone adapting the script, I recommend starting with a dry run for two or three days, checking the logs to see what it would touch, then enabling actual file changes. Add or remove providers as needed for your setup. If I have left anything unexplained or made a typo, I will come back and fix it later. </content>