AdSense Says ads.txt Is Missing: Tracing the Logs Layer by Layer
'"Google cannot see my file" is a case where guessing leads you astray — the only way to get a definite answer is to follow the same path as the crawler, then open the access logs to see whether it visited and what it received.'
The AdSense Sites page was showing the ads.txt status as "Not found", even though I had checked over ssh and the file was clearly there, with the correct contents. My first thought was, "Could my firewall be blocking Google's crawler?" — the server I manage has CSF, ModSecurity, and another layer of bot filtering. Plenty of suspects.
Instead of guessing, I checked each layer until I found the answer (spoiler: nothing was blocking it) — that process is what I want to share. It applies to any case of "Google cannot see my file". The IP in this article is a placeholder (203.0.113.10).
Layer 1: Does the file exist, and is it served correctly?
cat /home/user/domains/example.org/public_html/ads.txt
# google.com, pub-XXXXXXXXXXXXXXXX, DIRECT, f08c47fec0942fa0
stat -c '%y' .../ads.txt # ดูวันที่สร้างไฟล์ไว้ด้วย — จะสำคัญตอนท้าย
Layer 2: Send test requests with Google's UA
Most bot filters check the User-Agent, so I need to test with the UAs Google actually uses:
IP=203.0.113.10 # IP เซิร์ฟเวอร์เรา
for ua in 'curl/8.0' 'Mediapartners-Google' \
'Mozilla/5.0 (compatible; Google-adstxt)' \
'Mozilla/5.0 (compatible; Googlebot/2.1)'; do
code=$(curl -sk --resolve example.org:443:$IP -A "$ua" \
-o /dev/null -w '%{http_code}' https://example.org/ads.txt)
echo "[$code] $ua"
done
Every UA returned 200, showing that the origin was not filtering them — but that was not enough to draw a conclusion, because the site sits behind Cloudflare. Google goes through CF rather than connecting directly to my server.
Layer 3: Test through the actual Cloudflare edge
This is a classic pitfall: when testing from the server itself, DNS may resolve directly to localhost, bypassing CF. I need to force resolution to a CF edge IP to follow the same path Google takes:
dig +short example.org A # ได้ IP CF เช่น 104.21.x.x
curl -sk --resolve example.org:443:104.21.x.x \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1)' \
-D - https://example.org/ads.txt | head -20
# HTTP/2 200 / content-type: text/plain / cf-cache-status: DYNAMIC + เนื้อไฟล์ถูกต้อง
Remember to test redirects too, because the crawler starts at http:
curl -sk -A 'Mediapartners-Google' -o /dev/null \
-w '%{http_code} -> %{redirect_url}\n' http://example.org/ads.txt
# 301 -> https://example.org/ads.txt (hop เดียว โดเมนเดิม = โอเคตาม spec)
And www / non-www must both return the same file.
Layer 4: The decisive evidence — access logs
Everything above establishes that "it is accessible now", but the real question is "Has Google visited, and what did it receive?" — the logs can answer that:
zgrep -h 'ads.txt' /var/log/nginx/domains/example.org.log* | grep -i google
# 66.249.79.1 "GET /ads.txt HTTP/1.1" 301 162 Googlebot <- เริ่มที่ http
# 66.249.79.6 "GET /ads.txt HTTP/2.0" 200 85 Googlebot <- ตาม redirect มา ได้ไฟล์
# หา 4xx/5xx:
zgrep -h 'ads.txt' ...log* | grep -i google | awk '$9!~/200|301/'
# (ว่าง = Google ไม่เคยโดนปฏิเสธเลย)
The response to Googlebot was 85 bytes, even though my file was 59 bytes — do not jump to the conclusion that the wrong file was served. That is the 59 bytes file after gzip compression (the file is so small that compression overhead makes it larger). The crawler sent Accept-Encoding: gzip, so the log recorded the compressed size.
On a machine running nginx in front of apache, the same request appeared in both logs. nginx reported sending 59 bytes (the correct file), while apache reported 4738 bytes (the app's HTML). I traced it and found that nginx had served the static file to the client itself, while the response to the copy of the request proxied to apache was discarded and never reached the client — I proved this by sending 1 request with a unique UA (-A 'TRACE-1'), then using grep on both logs to compare them. Without proving this, I would have spent a lot longer chasing ghosts.
Case closed: Nothing was blocking it — the status was stale
The evidence fit together:
- The ads.txt file had only just been created during the site rebuild (shown by
stat). - The "Not found" status in AdSense was timestamped shortly before that — it was the result of a check when the file really did not exist yet.
- The status on the AdSense Sites page is not realtime — refreshes can be days to several weeks apart, especially for new sites.
- The logs confirmed that Google's latest fetch had already received 200.
After waiting a few days, the status turned green on its own. I did not need to change anything on the server.
Lessons
- "Google cannot see it" does not mean I should immediately blame the firewall — follow the logs, not fear
- Tests must follow the same path as the crawler: go through the CF edge, start at http, use the actual UA, and test both www/non-www
- A log size that differs from the file size does not mean the wrong file was served — consider gzip first
- Google's dashboard status always has a lag — give my own access logs more weight than its status label
I follow the same rule in every forensic investigation: do not change anything until there is evidence that it is actually broken. In this case, rushing to tear apart the firewall out of fear would have damaged a working setup. If I missed anything in the explanation or made a typo, I will come back and correct it later.
Want us to set this up?
Get help with server setup, configuration, migration, or troubleshooting. Save time for the work that matters.
Hit something like this? Stuck somewhere?
Come trade notes on the LinuxThai board
Comments 0
No comments yet — be the first