Receipts Vanished When Failed Syncs Were Marked as Sent

I happened to go back through the sync layer of a point-of-sale app I had built. It is an Android app that saves receipts to a local database first, then pushes them to an API later, because the shops it serves are in markets where connection drops are routine. When I reached the error-handling branch, I found code I had written myself: if the server returned 4xx, it marked the row as sent and moved on. That receipt disappeared from the pending counter. The cashier saw that everything had reached the server, even though the server had never received that receipt. So I reworked the entire layer, and I am documenting the reasoning and code here.

I have rewritten the class names, column names, and sample data in this article to make them neutral, but the policy numbers, response-code table, and test counts are real.

The old behavior: receipts could disappear in two ways, both silently

The app is built with Kotlin, Room, WorkManager, and Retrofit. The local receipt table has two columns that also serve as a queue: syncedAt (empty = not yet sent) and syncAttempts (the number of attempts so far). The worker picks up rows with an empty syncedAt and pushes them one receipt at a time. The problem was in two branches of the error handling.

The first path: for temporary errors, the worker kept returning retry to WorkManager without a limit. Rows that would never succeed stayed in the queue forever. The pending counter never reached zero, and nothing told the cashier which receipt was causing the problem.

The second path was much worse. If the server returned 4xx, the old code called markSynced followed by markSyncFailure. It saved the error message but also set syncedAt, so the row dropped out of the queue immediately.

My original reasoning sounded good but missed the point

I did not write it that way out of carelessness. I had a clear reason at the time: I was afraid one broken row would block the entire queue. If validation rejected a receipt, it would reject it again on every attempt. If I left it at the front of the queue, the receipts behind it could not move forward. So I chose the simplest way I had to remove that row from the queue: mark it as sent.

My mistake was solving two separate problems with one action. The first was "the worker should stop trying this row." The second was "this row is done." Those are not the same thing. A row rejected by the server is one the worker should stop trying, but it is not done. It needs someone to look at it.

Extract the policy into a pure object with no Android dependencies

Previously, these rules were scattered through the worker, mixed with API calls and DAO code. Writing tests meant setting up WorkManager and Room first. So I extracted all the rules into a single object that takes numbers and returns answers, with no knowledge of Context or the database.

object SyncPolicy {
    const val MAX_ATTEMPTS = 5

    // ถาวร = ลองอีกกี่ครั้งก็ได้คำตอบเดิม
    fun isPermanentError(code: Int): Boolean =
        code in 400..499 && code != 408 && code != 429

    fun shouldGiveUp(attempts: Int): Boolean = attempts >= MAX_ATTEMPTS

    // เส้น backoff เป็นวินาที ส่งต่อให้ตัวจัดตารางงานอีกที
    fun nextBackoffSeconds(attempts: Int): Long = when {
        attempts <= 1 -> 1L
        attempts == 2 -> 5L
        attempts == 3 -> 30L
        attempts == 4 -> 120L
        else -> 300L
    }

    sealed class State {
        data object Pending : State()
        data object Stuck : State()
        data object Synced : State()
    }

    fun classify(syncedAt: Long?, attempts: Int): State = when {
        syncedAt != null -> State.Synced
        shouldGiveUp(attempts) -> State.Stuck
        else -> State.Pending
    }
}

Response-code classification is the core of this object. The codes actually encountered in this system fall into the following groups.

The backoff sequence is 1, 5, 30, 120 seconds, then caps at 300. In practice, a row is parked before reaching that value because it has already failed 5 times. These numbers are policy values passed to the scheduler. When the worker actually wakes up also depends on WorkManager and the state of the device.

My favorite side benefit is that this file does not contain a single import android line, so its tests run on a plain JVM without a device or emulator.

Rows with permanent errors must be parked, not cleared

The worker's error handling is down to three paths that are easy to understand on the first read.

val resp = api.pushSale(bearer, body)
when {
    resp.isSuccessful ->
        dao.markSynced(sale.localId, System.currentTimeMillis())

    SyncPolicy.isPermanentError(resp.code()) ->
        // จอด ไม่เคลียร์ และเก็บข้อความที่คนอ่านรู้เรื่องไว้กับแถว
        dao.markSyncStuck(sale.localId, readableError(resp))

    else ->
        // ชั่วคราว: attempts + 1 แล้วให้รอบถัดไปลองใหม่
        dao.markSyncFailure(sale.localId, "HTTP ${resp.code()}")
}

I did not add a new queue table. The existing two columns were enough. I just gave syncAttempts one additional meaning: 99 is a sentinel value meaning "parked because of a permanent error." It already exceeds the limit of 5, so classify categorizes it as Stuck without a special condition. And syncedAt stays empty, so the row cannot be counted as sent.

-- แถวที่รอส่ง: worker หยิบชุดนี้แล้วกรองแถวที่ attempts >= 5 ออกอีกชั้น
SELECT * FROM sales WHERE syncedAt IS NULL ORDER BY id LIMIT 50;

-- จำนวนแถวที่ติด ใช้ขับ badge สีแดง (:max = 5)
SELECT COUNT(*) FROM sales WHERE syncedAt IS NULL AND syncAttempts >= :max;

-- จอดแถวถาวร: ตั้ง sentinel 99 เก็บ error ไม่แตะ syncedAt
UPDATE sales SET syncAttempts = 99, lastSyncError = :error WHERE localId = :localId;

-- ปุ่มลองอีก: คืนแถวเข้ารอบปกติ
UPDATE sales SET syncAttempts = 0, lastSyncError = NULL WHERE localId = :localId;

The server must handle duplicates, or retries create duplicate receipts

Every retry mechanism has the same prerequisite: the destination must accept repeated submissions without repeating their effects. A common case in shops with poor connections is that the request reaches the server and is saved, but the response is lost along the way. The register sees a timeout and sends it again. Without server-side protection, that produces two receipts.

The backend already handled this correctly, so I did not need to change anything. Both receipts and stock movements are deduplicated using the pair (branch_id, local_id), where local_id is an identifier generated by the register when it saves the record and remains unchanged no matter how many times it is sent.

$accepted = [];
$duplicates = [];

foreach ($request->input('movements') as $m) {
    $seen = StockMovement::where('branch_id', $branch->id)
        ->where('local_id', $m['local_id'])
        ->exists();

    if ($seen) {
        $duplicates[] = $m['local_id'];   // เคยรับแล้ว ไม่สร้างซ้ำ
        continue;
    }

    StockMovement::create($m + ['branch_id' => $branch->id]);
    $accepted[] = $m['local_id'];
}

return response()->json(['accepted' => $accepted, 'duplicates' => $duplicates]);

{
  "accepted":   ["loc-0007", "loc-0008"],
  "duplicates": ["loc-0006"]
}

The key point is that the server returns two separate groups, and the register marks both groups as synced, because a duplicate means the data already arrived on a previous attempt; it is not an error. If the server returned 4xx for a duplicate, it would enter the permanent-error branch and be parked even though all the data was already there. I have written separately about claiming work before executing it on the server in Jobs that call external systems must claim their status before sending. That article covers the server sending out; this one covers a client sending in. The principle is the same, applied from opposite sides.

Another contract confirmed by tests is that the server rejects stock entries with a reason of void through the register's push endpoint, because only the shop owner can void entries. If the register sends one, it receives 422 and the row appears as Stuck, which is the correct behavior.

The UI: failures need a visible place

The part that actually solves the problem is the UI, not the worker code, because a parked row that nobody can see is no different from a row that was cleared away.

The sync status badge on the sales screen previously took one value: the number waiting to be sent. It now takes two: pendingCount and stuckCount. When stuckCount is greater than zero, the badge turns red with text such as "N stuck". Tapping it opens a dialog listing each stuck row with its error message and a retry button. That button does just one thing: reset syncAttempts to 0 so the worker picks it up as usual on the next run.

A 401 is an easy example to picture. The branch's token expires, and all receipts pushed during that period are parked. The cashier logs in again, then taps retry, and the rows return to the queue. Previously, that whole batch of receipts would have been marked as sent without anyone knowing.

Tests: 14 on the JVM and 6 at the API

Once the logic was in a pure object, the tests were straightforward to write. The JVM suite grew from 29 to 43 tests. The 14 additions test only the policy, covering four groups: isPermanentError for each code, shouldGiveUp at the boundary between 4 and 5, every backoff step, and the classify state machine.

@Test fun permanentVsTransient() {
    listOf(400, 401, 403, 404, 409, 422).forEach { assertTrue(SyncPolicy.isPermanentError(it)) }
    listOf(408, 429, 500, 502, 503).forEach { assertFalse(SyncPolicy.isPermanentError(it)) }
}

@Test fun sentinelIsStuckNotSynced() {
    assertEquals(SyncPolicy.State.Stuck, SyncPolicy.classify(syncedAt = null, attempts = 99))
    assertEquals(SyncPolicy.State.Pending, SyncPolicy.classify(syncedAt = null, attempts = 4))
    assertEquals(SyncPolicy.State.Synced, SyncPolicy.classify(syncedAt = 1L, attempts = 99))
}

The API suite grew from 82 to 88 tests. The 6 additions verify the stock-entry push endpoint's contract: accepting new entries, preventing duplicates, batches mixing new and duplicate entries, validating allowed reason values, and authentication.

# ฝั่งแอป: test นโยบาย รันบน JVM ไม่ต้องมีเครื่อง
./gradlew testDebugUnitTest --tests '*SyncPolicyTest'

# ฝั่ง backend: สัญญาของเส้น push
php artisan test --filter=StockMovementsPush

I have not yet written automated tests for the stuck-row UI. I test it manually in this order.

1. Create one row with a permanent error. The easiest way is to use a nonexistent product_id in a stock entry. The row must be parked on the first attempt, without repeated retries.
2. Create a temporary error by shutting down the backend while the register is pushing, then wait for all 5 attempts.
3. Check the badge on the sales screen. It must turn red and show the number of stuck rows, and the pending counter must exclude those rows.
4. Tap the badge. Each receipt must appear with an understandable error message.
5. Fix the cause (restart the backend or correct the data), then tap retry. The row must return to Pending and disappear from the stuck list once it is sent successfully.
6. Check the server to confirm that there is only one copy of the receipt or entry, even if it was sent multiple times.

What I deliberately left undone

There are three gaps I know about and chose not to address this time.

First, 409 is still treated as permanent and gets stuck like other 4xx responses. In many cases, 409 means the data already exists at the destination, so the correct response is to treat it as sent. But the dialog does not yet have a "This is a duplicate; treat it as sent" button, so these rows remain red until someone looks at them. I would rather leave them stuck than let the system decide to clear them on its own, because that is the very bug I just fixed.

Second, there is no local sync history log. Each row stores only the latest error. If I want to know what caused each of those five failures in order, I currently cannot see that.

Third, there is no separate queue table. That is deliberate, because the existing two columns already express everything the queue needs. Adding a table would add another place where two sets of data could fall out of sync.

The rule I will apply to every offline queue from now on is just this: a row can leave the queue in only two ways—confirmation from the destination that it was received, or a person's decision. The worker has no authority to decide that a row is done simply because it could not send it. Feel free to adjust the attempt limit and backoff sequence to suit your own work. If I have left anything out or made a typo, I will come back and correct it later.