nothing to send With Thousands Left: A Failed-Item Cache

One of my clients had a website with a lot of Thai content, covering several kinds of pages, including articles and service pages, all of which needed English versions. The system processed millions of Thai characters and consumed tokens on the order of a hundred million using Claude. I use these figures only to show the scale; I do not intend anyone to rely on the decimal places. Nobody could manage checking every page, so I chose to use "code" as the validation gate instead of having a second AI check everything. Along the way, I also ran into a queue-blocking bug I had not anticipated, so I wanted to tell the whole story.

One row per field, with a hash of the source

My data model is simple: one row per (type, id, field, locale), such as "this page, content field, English." Each row stores the translated value, a status, and source_hash, which is the sha1 of the Thai text at the time it was translated. There are four statuses: machine (translated by the model), reviewed (accepted by a person or reviewer), stale (the source has changed), and needs_human (an admin needs to look at it).

CREATE TABLE translations (
  type        VARCHAR(32)  NOT NULL,
  id          BIGINT       NOT NULL,
  field       VARCHAR(64)  NOT NULL,
  locale      CHAR(2)      NOT NULL,
  value       MEDIUMTEXT   NOT NULL,
  source_hash CHAR(40)     NOT NULL,
  status      ENUM('machine','reviewed','stale','needs_human') NOT NULL,
  PRIMARY KEY (type, id, field, locale)
);

The server has two API endpoints. The first, GET /jobs, returns only fields that are "missing" or "stale," along with the Thai text, hash, and context. The second, POST /translations, accepts translated results and returns one of these three responses for each item.

1. ok Saved.
2. stale_source The Thai text has changed since the job was fetched. The server recalculates and compares the hash itself rather than trusting the value returned by the runner, so it does not overwrite the row.
3. kept_reviewed A row accepted by a person or reviewer is not overwritten unless force is specified.

The result is that I can resume at any time, rerun without duplicating work, and have a row queued automatically whenever someone edits its Thai text and changes the hash, without having to remember what was edited. Personally, I think this is half the system already. The other half is deciding how much to trust the model's output.

Seven code checks, with no AI involved

I use code as the validation gate because broken translations tend to fail structurally, rather than through subtle errors in meaning. Actual examples include missing closing tags, a missing number, an answer in the wrong language, or an answer half the length of the source because the model omitted content on its own. Code catches these problems more cheaply and accurately than having another AI read everything again. I use seven checks.

1. Not empty The answer must contain content.
2. Tag, link, and image signatures These must match the source, including tag types and href and src values.
3. Every number In the source must appear in the result, after converting Thai digits to Arabic digits, removing thousands separators, and allowing minor formatting differences.
4. Item codes With a matching prefix pattern must all be present, but strings joined to a time, such as SKU10:30, do not count as codes.
5. Placeholders Must all be present.
6. No Thai characters left Except spans copied exactly from the source that are no longer than 40 characters, which are treated as proper names. Thai characters left in the English version are detected with the regex [฀-๿], except in fields containing lists of Latin names only.
7. Length ratio Between 0.5 and 4.0 relative to the Thai source when the source is at least 30 characters long, along with any defined length limit for SEO fields.

An abbreviated version of these checks looks roughly like this. I have left out details to make it easier to read, but this is the actual structure.

import re

THAI = r'[฀-๿]'
THAI_DIGITS = str.maketrans('๐๑๒๓๔๕๖๗๘๙', '0123456789')

def signature(html):
    names = sorted(re.findall(r'</?\s*([a-zA-Z0-9]+)', html))
    urls  = sorted(re.findall(r'(?:href|src)="([^"]*)"', html))
    return names, urls

def numbers(text):
    text = text.translate(THAI_DIGITS)
    return {n.replace(',', '') for n in re.findall(r'\d+(?:[.,]\d+)*', text)}

def leftover_thai(src, out):
    bad = []
    for run in re.findall(THAI + r'+(?:\s+' + THAI + r'+)*', out):
        if not (run in src and len(run) <= 40):   # คัดลอกตรงจากต้นฉบับและสั้น = ชื่อเฉพาะ
            bad.append(run[:20])
    return bad

def check(src, out, locale, max_len=None):
    if not out.strip():
        return ['empty']
    problems = []
    if signature(src) != signature(out):
        problems.append('tag/link/image signature differs')
    missing = numbers(src) - numbers(out)
    if missing:
        problems.append('missing numbers: %s' % sorted(missing))
    left = leftover_thai(src, out)
    if left:
        problems.append('thai left over: %s' % left[:3])
    if len(src) >= 30:
        if not 0.5 <= len(out) / len(src) <= 4.0:
            problems.append('length ratio out of range')
    if max_len and len(out) > max_len:
        problems.append('over seo length cap')
    return problems

The checks I spent the most time adjusting were three and six. Number matching needs to be flexible enough to treat "1,500" in the source and "1500" in the result as equivalent; otherwise, the gate rejects hundreds of valid results. Check six is tricky in both directions: too strict, and Thai proper names that should be preserved get rejected; too loose, and Thai sentences slip into the English version for readers to discover themselves. That led to the rule "copied exactly and no longer than 40 characters," a limit I chose based on what I encountered. Feel free to adjust it to your own data.

I have covered the same idea for document reading, with code doing the arithmetic, in Let AI read documents, but let code do the arithmetic, which you can read for more. The differences here are the translation-specific checks and per-field hashes.

Retry once after failure, then record it if it still fails

I do not discard a field immediately when it fails validation. I try once more, adding the problems found by the checks to the prompt, such as "the number 2,500 is missing." The model can often fix the problem on the second attempt. But if it fails again, I store it in a failed file, keyed by type:id:field, together with the source hash at the time of failure and the reason.

import json, pathlib

def translate_field(item, locale, ask, failed):
    key = '%s:%s:%s' % (item['type'], item['id'], item['field'])
    problems = []
    for attempt in (1, 2):
        answer = ask(build_prompt(item, locale, problems))   # รอบสองแนบ problems ไปด้วย
        problems = check(item['th'], answer, locale, item.get('max_len'))
        if not problems:
            return answer
    failed[key] = {'hash': item['hash'], 'why': problems}
    return None

def save_failed(locale, failed):
    pathlib.Path('cache').mkdir(exist_ok=True)
    p = pathlib.Path('cache/failed-%s.json' % locale)
    p.write_text(json.dumps(failed, ensure_ascii=False, indent=1), encoding='utf-8')

I need this record because, without it, a field that can never pass—for example, one whose source contains odd text with internally inconsistent numbers—would be sent to the model on every round, burning tokens for nothing. I store the hash as well as the field name because an edit to the Thai source changes the hash and should give the field another chance. This looked good on paper, but in practice it led me to the next bug.

The blocked-queue bug: nothing to send with thousands still left

The symptom was that the runner reported "nothing to send" and stopped, but when I counted the table, thousands of fields were still missing. My first guess was that the API was returning the wrong queue, but GET /jobs checked out. Next, I guessed the quota had run out and the runner was misinterpreting it, but that was not it either. Eventually, I had to go back and read comments I had left in my own code before realizing that the failed cache was the cause.

The runner fetched batches of N items from the head of the queue. At the head were items whose fields had all failed before with the same Thai text. These items were still fetched to fill the batch on every round. When it came time to send them for translation, every field was skipped because it was in the failed cache, leaving the batch empty. The runner concluded that nothing remained, even though thousands of fields that had never been translated were waiting behind the failed rows.

My fix was to filter out items where "every field has a failed-entry whose hash matches the current source" before filling the batch, except when --retry-failed is specified or a reviewer has left a note asking for another attempt. Once someone edits the item's Thai text, its hash changes and no longer matches the record. The item then passes the filter and gets retried automatically, without anyone having to clear the file.

def worth_sending(entry, failed, retry_failed=False):
    """False เมื่อทุกฟิลด์ของรายการนี้ล้มมาแล้วด้วยข้อความไทยตัวเดิม"""
    if retry_failed or entry.get('reviewer_note'):
        return True
    for f in entry['fields']:
        rec = failed.get(f['key'])
        if rec is None or rec['hash'] != f['hash']:
            return True          # มีอย่างน้อยหนึ่งฟิลด์ที่ยังไม่เคยล้ม หรือต้นฉบับเปลี่ยนแล้ว
    return False

def fill_batch(queue, failed, size, retry_failed=False):
    batch = []
    for entry in queue:
        if worth_sending(entry, failed, retry_failed):
            batch.append(entry)
            if len(batch) == size:
                break
    return batch

My lesson is that a failure log has two jobs. The first is to prevent repeated token spending. The second, which I had overlooked, is to avoid blocking work that has not been done yet. If you build a queue system that "skips failed items," test the case where failed items occupy an entire batch at the head of the queue. I wrote about a similar problem, where every layer reported success despite the result being garbage, in Every layer reports success at once. "Nothing left" is another kind of success message that needs to be checked against a count of the actual table.

Translation memory and splitting long blocks

Websites contain more repeated paragraphs than I expected: warnings, footers, button text. I therefore store a translation memory in a SQLite file, mapping the sha1 of each Thai block to its translation, so repeated paragraphs are free and consume no tokens. For long HTML, I split it into top-level blocks of no more than roughly 3,000 characters and translate one block at a time. The reason is that if an attempt fails, I only need to redo one block rather than the whole page. Once all blocks are translated, I reassemble them and run the same checks again on the assembled value, because block boundaries are places where tags can get lost.

CREATE TABLE IF NOT EXISTS tm (
  src_hash CHAR(40) NOT NULL,
  locale   CHAR(2)  NOT NULL,
  value    TEXT     NOT NULL,
  PRIMARY KEY (src_hash, locale)
);

I made the translation engine replaceable through a single interface: ask(text) returns (answer, usage). It can use Claude, Gemini, agy, or reserve API credits. The code checks and hashes know nothing about which model is behind it, so I can switch engines midway through the work without touching the rest of the system.

A per-page publishing gate and the review loop I left disabled

Another essential part, though not directly about translation, is that readers should not encounter half-translated pages. On an en page, if not every block has been translated, the website shows the Thai content with a banner and adds noindex to that page. Only "ready" pages enter the sitemap, and IndexNow discards en URLs that are not in the sitemap. This keeps users from seeing unfinished pages and search engines from seeing duplicate Thai content under English-version URLs. I cover the mechanism for submitting URLs to IndexNow separately in Notify Bing as soon as a page changes.

The system also supports a review loop: a second model gives a score from 1 to 5 with notes, using value_hash to guard against the row changing between the reviewer reading it and writing back the result. Scores below the threshold enter the correction queue. The correction model returns only the incorrect words, saved as machine plus reviewed and kept out of the queue for another pass. Exceeding the maximum number of rounds sets needs_human for an admin to inspect. But the project owner decided that a Flash model did not need a second review pass and left only the code checks enabled throughout. Personally, I agree with the cost savings as long as spot checks show nothing concerning. Each organization can make its own call.

What I would do the same way again

If I had to translate another large website, I would keep these five things. First, a source hash per field, so runs can resume and source edits automatically queue the work again. Second, have the server recalculate the hash when accepting results instead of trusting the value sent by the runner. Third, code checks with no AI to validate every field before saving. Fourth, record failures after two attempts, together with the hash, and prevent repeat submissions—but filter before filling the batch, not afterward. Fifth, a per-page publishing gate, so readers and search engines see only pages that are actually ready.

What I would not do is believe that code checks make translations correct. They only keep translations from being "broken." Correct meaning still depends on a good model, a good glossary, and human spot checks. Anyone adapting this approach to their own work can adjust values such as 0.35 or 40 characters to suit their data. I chose these values based only on what I encountered on one website. If I find anything I missed or cases where the checks reject valid results, I will come back and update them later.