Passing the Sandbox Doesn't Mean It Works — 7 False Signals

I was writing an automated ordering system against a provider's API. They offered a sandbox with a large balance of fake money in the account. I tested every command until they all passed, from checking availability to placing orders and retrieving items. I thought I was done. Only when I moved to production did I realize that, after two days of testing, there were at least seven places where the sandbox told a different story from production. In several cases, I spent a long time trying to fix my own code because I thought I had written it wrong. I have collected them all in this article, along with ways to write code that does not fall for these signals.

Signal 1 — So slow I thought the code had hung

The create command in the sandbox took 60 to 120 seconds to respond, while production responded in under 5 seconds. At first, I thought I had misconfigured my HTTP client because the library's default timeout was 30 seconds. Every request timed out.

Before changing anything, I recorded the actual elapsed time, because arguing with myself based on a feeling that "it's slow" was pointless.

# จับเวลาจริงหลายรอบ แล้วดูค่าที่แย่ที่สุด ไม่ใช่ค่าเฉลี่ย
for i in $(seq 1 5); do
  curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' \
    --max-time 200 -X POST "$ENDPOINT/create" -d @payload.json
done
# 200 78.412
# 200 112.905
# 200 64.330

The fix was not to keep lowering the timeout, but to configure it separately for each environment and set the sandbox timeout to the slowest response I had seen plus another half as a margin.

// config/provider.php — แยกค่าตามสภาพแวดล้อม อย่าใช้ค่าเดียวกันทั้งสองฝั่ง
return [
    'endpoint' => env('PROVIDER_ENDPOINT'),
    'timeout'  => (int) env('PROVIDER_TIMEOUT', 180),  // sandbox ช้ากว่ามาก
    'retries'  => (int) env('PROVIDER_RETRIES', 0),    // คำสั่งที่ตัดเงิน ห้าม retry อัตโนมัติ
];

Signal 2 — It says unavailable when it is available

The sandbox availability check returned unavailable for a freshly generated random name. Checking the same name again sometimes returned available. This was neither a bug in my code nor someone actually reserving the name. It was simply sandbox instability.

What I did was make a clear distinction in the code between "unavailable" and "inconclusive response," then log both the request and the raw response so I would not have to guess when reviewing them later.

$r = $client->check($name);
match (true) {
    $r->available === true  => $this->proceed($name),
    $r->available === false => Log::info('unavailable', ['name' => $name, 'raw' => $r->raw]),
    default                 => Log::warning('ambiguous availability', ['raw' => $r->raw]),
};

Signal 3 — Names containing test are rejected

This one was funny. The sandbox rejected requests where the item name contained test, demo, or sandbox, which are the first words anyone testing a system types. When it rejected them with the same code as "invalid name," I spent almost an hour checking my own format validator.

The approach that resolved it was to make the test name generator always produce names that looked real, but were still identifiable by a prefix and date.

function testName(string $prefix): string
{
    // หน้าตาเหมือนของจริง แต่ไล่ย้อนได้ว่าเกิดจากการทดสอบวันไหน
    return sprintf('%s-%s-%s', $prefix, date('Ymd'), bin2hex(random_bytes(3)));
}
// ตัวอย่างผลลัพธ์: acme-20261121-9f31c0

Signal 4 — Created successfully, charged, but nowhere to be found

This was the most misleading one. The sequence was that the create command reported success, the test account balance was actually deducted, and the order summary email actually arrived, but fetching all items returned zero items.

I spent half a day checking whether I had sent the wrong pagination parameters. The eventual answer was that the sandbox did not always save items to the user account, even when every step of the create command succeeded.

The practical lesson is do not use an item lookup to confirm successful creation. Confirm it using evidence the provider issues for that item, such as a reference number included in the response or order summary email, and save that number in my own DB from the very start.

-- เก็บหลักฐานตั้งแต่ก่อนยิง แล้วค่อยเติมผลทีหลัง
INSERT INTO provider_orders (local_ref, payload_hash, status, created_at)
VALUES (?, ?, 'sending', NOW());

UPDATE provider_orders
   SET status = 'confirmed', remote_ref = ?, confirmed_at = NOW()
 WHERE local_ref = ? AND status = 'sending';

Signal 5 — The amount charged is not the real price

An item that cost about ten to fifteen units in production was charged at over forty units in the sandbox because they bundled extra services into test mode without saying so. Using that number to calculate margins or display prices would break things immediately on the day I switched to production.

The rule I set for myself afterward was never design screens or write decision logic based on numbers from the sandbox. If the real price list is not available yet, have the screen show that pricing data is not yet available. That is better than displaying the wrong number.

Signal 6 — The catalog data structures differ

Following on from the previous point, when I fetched the product list with prices to prepare the screen, the aggregator reported that it could not find a price for a single item, even though there were thousands of items. The reason was that the sandbox and production used different names for the price field, so code that read the field directly got an empty value.

The safeguard is to write a result converter that accepts multiple field names, then fail loudly when none is found, instead of silently returning zero.

function priceOf(array $row): ?float
{
    foreach (['price', 'retailPrice', 'yourPrice', 'amount'] as $k) {
        if (isset($row[$k]) && is_numeric($row[$k])) {
            return (float) $row[$k];
        }
    }
    Log::error('price field not found', ['keys' => array_keys($row)]);
    return null;   // คืน null ให้ชั้นบนตัดสินใจ ห้ามคืน 0
}

Returning zero here is the most dangerous bug in this article, because zero looks like a reasonable number, and it will flow through the entire system without anyone noticing.

Signal 7 — I cannot test from my own machine at all

The provider required requests to come from a registered address, which was the server's address, not the machine where I was writing the code. That meant I could not run even a single test from my own machine.

I did not solve this by allowing another address, because my home address changes regularly. Instead, I ran commands on the authorized machine while watching the logs live.

# รันบนเครื่องที่อยู่ในรายการอนุญาต แล้วดู log สดอีกหน้าต่าง
ssh [email protected] \
  'cd /srv/app && php artisan provider:demo-order --confirm-sandbox-mutation --name=acme-20261121-9f31c0'

ssh [email protected] 'tail -f /srv/app/storage/logs/laravel-$(date +%F).log'

Notice the mandatory --confirm-sandbox-mutation flag. I added that myself as a rule: any command that changes state on the provider's side requires me to type this flag every time, with no default, to stop myself from accidentally pressing enter again on a command from history.

Checklist before switching to production

1. Keep separate sets of endpoint, timeout, and credential settings for the two environments, reading them entirely from the environment
2. For every command with external effects, reserve a state in my own DB before sending it, then fill in the result afterward in a way that is safe to write again
3. Never automatically retry commands that charge money or create items; leave them pending review instead
4. Confirm success using the reference number issued by the provider, not an item lookup
5. Treat every price and quantity from the sandbox as unreal until it has been seen in production
6. The result converter must fail loudly when a field cannot be found; never silently return zero or an empty value
7. For the first production test, choose the cheapest item that can be canceled, and watch the logs live throughout the run

I encountered this set of issues while building an automated ordering system, but when I later integrated a payment API, I ran into the same patterns again, as described in Connecting to a Bank Open API to Generate Payment QR Codes. That means this is not about any particular provider, but the nature of a test environment that is not connected to the real thing. Feel free to make your own checklist longer than this. If I have missed anything in the explanation or encounter more false signals, I will come back and update it later.