AI Crawler Overloads ?add-to-cart= — Block It Without Hurting SEO

One morning, I received an email from DirectAdmin warning that average load had exceeded the threshold. It identified one store's PHP-FPM pool and mysql as the CPU consumers. At first I thought it was an attack, but when I went through that vhost's access log, it turned out to be nothing of the sort. It was a single AI crawler clicking the ?add-to-cart= links for every product in the store. I wanted to share this case as a practical guide, because every WooCommerce store has this same opening.

All domain names, usernames and IP addresses in this article are fictional (shop.example, shopuser, 203.0.113.x). The config structure, figures and test results are real, from a machine I manage.

Check the logs to see who is sending requests before guessing

The first commands I use whenever load spikes count IP addresses and user-agents directly in that vhost's log file. No extra tools needed.

LOG=/var/log/nginx/domains/shop.example.log

# ใครยิงเยอะสุด 10 อันดับแรกของวันนี้
awk '{print $1}' $LOG | sort | uniq -c | sort -rn | head -10

# IP นั้นเป็นใคร ใช้ UA อะไร
grep '^203.0.113.45 ' $LOG | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -3

# มันไปกดอะไรบ้าง
grep '^203.0.113.45 ' $LOG | awk '{print $7}' | cut -d'?' -f2 | cut -d'&' -f1 \
  | sort | uniq -c | sort -rn | head -10

# สถานะที่ตอบกลับไป
grep '^203.0.113.45 ' $LOG | awk '{print $9}' | sort | uniq -c | sort -rn

The results were clear enough to need no interpretation.

The user-agent belonged to a crawler from an AI company. It was not a spoofed UA, and it was not even sending requests at flood rates. It was simply persistent, following the links the store's theme had laid out for it.

The machine itself was nowhere near collapse. With 16 cores, load reached 10, roughly 63%, with another 41 GB of RAM free. The day's 5xx rate was 0.3%, which meant part of the problem was an alert threshold set too low (I later raised it from 6 to 14). But the real issue to fix was why those seventy thousand requests were so expensive.

Why ?add-to-cart= costs more than any other page on the site

If product pages or the store's homepage have a page cache, a bot can request them as often as it likes without much impact. It receives ready-made HTML without touching PHP or MySQL. But ?add-to-cart= is an entirely different matter.

A rough calculation makes the scale clear. The store has thousands of products, so one complete crawl tells the machine to boot WooCommerce tens of thousands of times without a single buyer. That user's pool had pm.max_children = 10. Once cart operations occupied all 10 slots, real customers had to wait in line. This explained the "site gets slow intermittently" report the store owner had just sent me.

Layer 1: robots.txt for bots that listen

I always start with the cheapest layer. This store did not have an SEO plugin controlling robots.txt, so I could write it directly as a physical file. If you use an SEO plugin, you need to edit it through the plugin's interface instead, or the physical file will not be used.

# /home/shopuser/domains/shop.example/public_html/robots.txt
User-agent: *
Disallow: /*add-to-cart=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?wc-ajax=
Disallow: /*?orderby=
Disallow: /*?add_to_wishlist=
Disallow: /*?yith_wcwl_

Sitemap: https://shop.example/sitemap_index.xml

The point I want to emphasize is that not a single line in this list blocks product or category pages. Everything blocked here is a page specific to an individual user, which should not appear in search results anyway. Even if Google crawled it, it would just be an empty cart page with no use to anyone.

# ตั้ง owner ให้ถูก ไม่งั้น DA/ปลั๊กอินเขียนทับไม่ได้ทีหลัง
chown shopuser:shopuser /home/shopuser/domains/shop.example/public_html/robots.txt
chmod 644 /home/shopuser/domains/shop.example/public_html/robots.txt

# เช็คว่าเสิร์ฟออกจริง ไม่ได้ถูกปลั๊กอินดักไว้
curl -s https://shop.example/robots.txt | head -20

The AI bot responsible this time respected robots.txt. After the file went live, its traffic dropped noticeably on the next crawl. But I did not stop there, because robots.txt is just a sign asking for cooperation, not a locked door.

Layer 2: Add rel=nofollow directly to the links

The actual source of the problem was the store's theme placing "Add to cart" links on every product card in the listing. Those links were ordinary <a href> elements that every bot read as "I can follow this." So I wrote a short mu-plugin to add rel="nofollow" to every link in the product loop, without touching the theme's code or losing the change when the theme updates.

<?php
// wp-content/mu-plugins/shop-bot-nofollow.php
// ใส่ rel=nofollow ให้ปุ่ม add-to-cart ในหน้ารายการสินค้า (ไม่แตะหน้าสินค้าเดี่ยว)

add_filter('woocommerce_loop_add_to_cart_link', function ($html) {
    if (strpos($html, 'rel=') === false) {
        $html = str_replace('<a ', '<a rel="nofollow" ', $html);
    }
    return $html;
}, 20);

// ธีมบางตัวสร้างปุ่มผ่าน args แทน string ตรง ๆ เลยต้องดักทั้งสองทาง
add_filter('woocommerce_loop_add_to_cart_args', function ($args) {
    $args['attributes']['rel'] = 'nofollow';
    return $args;
}, 20);

Why a mu-plugin instead of a regular plugin? Because files in mu-plugins/ load automatically, cannot be disabled from the admin page, and cannot be accidentally deactivated while someone is debugging something else. I like to keep anything that "must stay in place" here.

Check the result by counting links on the actual page, not by eye.

curl -s https://shop.example/shop/ \
  | grep -o 'add_to_cart_button[^>]*' | grep -c 'rel="nofollow"'
# 12   <- ครบทั้ง 12 ลิงก์ในหน้ารายการ

Layer 3: nginx map + limit_req for just one route

The first two layers ask for cooperation. This one enforces it. But I did not want to rate-limit the whole machine, because it hosts more than eighty other unrelated domains. The technique is to use a map that makes the zone key empty for every case outside the target. nginx skips limit_req whenever the key is empty, making it a no-op for every other site on the machine.

# /etc/nginx/nginx-includes.conf  (DirectAdmin include ไฟล์นี้ไว้ใน http{} ให้แล้ว
# และไม่เขียนทับตอน rewrite config — เป็นที่วางของ http-context ที่ปลอดภัยที่สุดบน DA)

map $host$request_uri $addtocart_key {
    default "";
    "~*shop\.example.*add-to-cart="  $binary_remote_addr;
}

limit_req_zone $addtocart_key zone=addtocart:10m rate=2r/s;

Then enable it in the nginx file for that user's domain only. On DirectAdmin, this is the .cust_nginx file, which survives a DA rewrite.

# /usr/local/directadmin/data/users/shopuser/domains/shop.example.cust_nginx
# |CUSTOM| = DA แทรกให้ที่หัว server block ทั้งขา :80 และ :443
limit_req zone=addtocart burst=5 nodelay;

Finish with a careful reload. Always test the syntax first, because breaking this file breaks every domain on the machine at once.

nginx -t && systemctl reload nginx

# สั่งให้ DA เขียน config ของ user นั้นใหม่ให้รับไฟล์ cust_nginx
echo 'action=rewrite&value=nginx&user=shopuser' >> /usr/local/directadmin/data/task.queue
/usr/local/directadmin/dataskq d

Layer 4: Return 403 only for private pages, keeping all products indexable

A few days later, I saw another bot requesting ?wc-ajax= even though it was already disallowed in robots.txt. So I added a final layer that returns 403 directly. Both conditions must be true: "it is a bot" and "it is accessing a private page."

# ยังอยู่ใน /etc/nginx/nginx-includes.conf
map $http_user_agent $is_crawler {
    default 0;
    "~*(googlebot|bingbot|baiduspider|yandex|applebot|petalbot|gptbot|claudebot|bytespider|crawler|spider)" 1;
}

map "$is_crawler:$request_uri" $bot_private_block {
    default 0;
    "~^1:/(cart|checkout|my-account)(/|\?|$)"                 1;
    "~^1:/wp-login\.php"                                       1;
    "~^1:.*[?&](add-to-cart|wc-ajax|add_to_wishlist|yith_wcwl_)=" 1;
}

# shop.example.cust_nginx — วางไว้ระดับ server ไม่ใช่ใน location
if ($bot_private_block) { return 403; }
limit_req zone=addtocart burst=5 nodelay;

This has to go at server level because if and return run in the rewrite phase, before location selection and before proxy_pass. If placed inside the location that proxies to Apache, it will not run in time.

After all four layers are in place, the expected results fall into two clear groups.

The testing trap that wasted several rounds of my time

This is the part I most wanted to share in this article, because it cost me several rounds of work before I caught it.

After adding the config, I tested it the way I normally do: using curl --resolve to point the domain to 127.0.0.1 so I would not need to go out over the internet. Every request returned 200, even though nginx -T showed all the directives in place. I added an add_header for debugging, but it did not appear. I added a test location, but got WordPress's 404 page instead. By then, I had almost concluded that .cust_nginx did not work.

The reason is that the server block DirectAdmin generates for each domain has listen on the machine's public IP, not loopback. Requests to 127.0.0.1 therefore landed in another server block that had none of my domain's directives. It returned 200 without anything being wrong. It just was not the block I had edited.

This is the correct test sequence I use, sending 20 requests in parallel to check whether the rate limit actually works.

IP=203.0.113.10          # ไอพีสาธารณะของเซิร์ฟเวอร์ ไม่ใช่ 127.0.0.1
D=shop.example

# 1) add-to-cart 20 อันขนานกัน -> ต้องมีตัวที่โดน 429
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" --resolve $D:443:$IP \
    "https://$D/?add-to-cart=1234" &
done; wait | sort | uniq -c

# 2) หน้าแรก 20 อันขนานกัน -> ต้อง 200 ทั้งหมด ห้ามโดนหางเลข
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" --resolve $D:443:$IP "https://$D/" &
done; wait | sort | uniq -c

# 3) เทสต์ฝั่งบอท -> ต้อง 403 เฉพาะหน้าส่วนตัว
for p in /cart/ /checkout/ /my-account/ "/?wc-ajax=get_refreshed_fragments"; do
  printf '%s -> ' "$p"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Googlebot/2.1" \
    --resolve $D:443:$IP "https://$D$p"
done

# 4) เทสต์ว่าหน้าที่ต้อง index ยังเข้าได้ -> ต้อง 200 ทั้งหมด
for p in / /shop/ /sitemap.xml /feed/; do
  printf '%s -> ' "$p"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Googlebot/2.1" \
    --resolve $D:443:$IP "https://$D$p"
done

The actual results from the run with 20 parallel requests:

Those 429 responses matter more than they might seem, because nginx returns them itself without waking PHP-FPM at all. The cost per request drops to almost zero, compared with booting the full WooCommerce stack for every request before the fix.

Finishing with PHP-FPM and a lesson from my own mistake

Another common mistake is to see a full pool and immediately raise pm.max_children. On a shared host, I see the child limit as protection for neighboring sites, not a problem. Raising it only risks taking the entire machine down with it. So I kept it at 10 and just added a timeout to stop stuck requests.

; /usr/local/directadmin/data/users/shopuser/php/php-fpm84.conf
[shopuser]
pm = ondemand
pm.max_children = 10
pm.max_requests = 500
request_terminate_timeout = 60s

systemctl reload php-fpm84

This is the order I use for cases like this, from cheapest to most expensive. I apply one layer at a time and always measure the result before moving to the next.

1. Read the logs to identify the actual source first: which bot, which paths, and how many requests. Do not guess from the system's alert email, because it only identifies the symptom: PHP and MySQL consuming resources.
2. Use robots.txt to block only private pages. Be careful not to block product or category pages, then wait for the next crawl.
3. Add rel=nofollow to cart links on product listing pages through a mu-plugin, addressing how bots discover these links in the first place.
4. Use nginx limit_req with an empty map key so it applies only to the intended domain and paths, without affecting other sites on the same machine.
5. Return 403 only when both conditions hold: bot + private page. Always test both sides: requests that must be blocked and requests that must stay accessible.
6. Test through the machine's public IP, not loopback, and keep the test commands for reuse when someone rewrites the config later.

What I want to leave you with is this: do not assume a load spike means an attack. In this case, blocking the entire IP at the start would have ended things quickly too, but it would also have lost the opportunity for the store's product content to be collected and used to answer people's questions in AI tools. So I chose to close only the doors nobody should be entering, while leaving the storefront open to everyone as before.

If you use this setup, adjust the rate and UA list to suit your environment. Stores using different cache plugins may need to block slightly different paths. Keep the same principle: "pages tied to individual sessions = no need to index." If I have left anything out or made a typo, I will come back and correct it later.