Making 20,000+ API Calls That Resume Without Leaving Corrupt Files

I had a job to generate 26,850 audio files through a provider's API, from just over five thousand vocabulary words multiplied by five speaker voices. I was using a free-tier account, with a clearly stated quota of 20 calls per 60 seconds. My first version was straightforward: loop through the list and make the calls. Within a few minutes of running it, I was getting both 429 responses and messages saying I had exceeded the quota. While fixing it, I concluded that this kind of batch caller always needs three pieces, not one: a throttle, a resume mechanism, and a validator. Leave any one out and it hurts in a different way.

I wrote this article as a recipe with code I actually use, with the provider's name and connection settings removed. It can be used with any API that limits the number of calls.

The limit belongs to the whole account, not each thread

My first mistake was putting time.sleep() inside each worker, assuming that delaying every thread equally would keep me under the limit. That was wrong, because the provider counts calls per account, not per thread. With 10 threads, each waiting 3 seconds, the actual output is 10 calls every 3 seconds, or 200 calls per minute—ten times the limit.

A proper throttle has to be shared, with a single variable recording the time of the last call, and every thread queuing to pass through that same throttle.

import threading, time

_rate_lock = threading.Lock()
_last_call = 0.0
_min_interval = 3.0          # 20 ครั้ง/60 วิ = 1 ครั้งต่อ 3 วินาที

def set_rps(rps):
    global _min_interval
    _min_interval = 1.0 / max(rps, 0.01)

def throttle_global():
    """ทุกเส้นต้องผ่านประตูนี้ก่อนยิง ไม่ว่าจะเปิดกี่เส้น"""
    global _last_call
    while True:
        with _rate_lock:
            now = time.time()
            since = now - _last_call
            if since >= _min_interval:
                _last_call = now
                return
            wait = _min_interval - since
        time.sleep(wait)      # นอนนอก lock เสมอ

I chose 0.3 calls per second, which is 18 calls per minute. I did not set it to the full 20 because the timing on my side and theirs does not match exactly, and a sliding-window quota always counts the last call from the previous minute alongside this minute's calls. Leaving ten percent of headroom costs less than chasing 429 responses afterward.

More threads do not make it faster when the limit is shared

Once I had a shared throttle, the worker count took on a completely different role. It did not make anything faster, because the throttle already fixed the rate. The workers only hide the network wait for each call.

I started with 10 threads and got occasional max retries exceeded errors because threads waited in the queue long enough for their connections to time out before they could make a call. Reducing the count to 5 made the problem disappear. The overall rate stayed exactly the same.

Those 25 hours are exactly why the second piece is necessary. A job that runs overnight is bound to be interrupted: a network drop, a power flicker, or someone shutting down the machine.

Retrying and throttling are separate concerns—keep them separate

The throttle controls the outgoing call rate, while retries handle individual failures. The two need to be clearly separated, and the crucial point is that every retry must pass through the throttle again, rather than firing immediately.

def generate_one(session, endpoint, task, retries=5):
    key, payload, out_path = task
    last_error = "unknown"
    for attempt in range(retries):
        try:
            throttle_global()                       # เข้าคิวใหม่ทุกครั้ง
            r = session.post(endpoint, data=payload, timeout=20)
            if r.status_code == 200 and len(r.content) >= 100:
                out_path.parent.mkdir(parents=True, exist_ok=True)
                tmp = out_path.with_suffix(out_path.suffix + ".part")
                tmp.write_bytes(r.content)
                tmp.replace(out_path)               # เปลี่ยนชื่อทีเดียวจบ
                return True, len(r.content)
            if r.status_code == 429:
                time.sleep(2 ** attempt + 0.5)      # ถอยแบบทวีคูณ
                last_error = "rate limited"
                continue
            body = (r.text or "")[:150].replace("\n", " ")
            return False, f"HTTP {r.status_code}: {body}"   # ผิดถาวร ไม่ต้องลองซ้ำ
        except requests.RequestException as e:
            last_error = str(e)[:100]
            if attempt < retries - 1:
                time.sleep(1 + attempt)
                continue
            return False, last_error
    return False, f"max retries ({last_error})"

The two lines I added later were tmp and replace. Originally, I wrote directly to the destination file, which caused all the corrupt files I found at the end. If the network drops before the write finishes, what remains on disk is a file with the right extension but incomplete contents. Writing to a temporary name and then renaming it gives the destination file only two possible states: absent or complete.

Resume from what is actually on disk, not a status file

Early in the design, I considered keeping a status file to record how far the job had progressed. I eventually dropped that idea because a status file can fall out of sync with the actual files on disk at any time. If the program is killed midway, the status file retains its previous value, and if someone manually deletes files, the mismatch gets even worse.

The most robust approach is to rebuild the full task list on every run, then filter out what already exists. The disk is the single source of truth.

tasks, cached = [], 0
for item_id, text, level in items:
    for alias in chosen_voices:
        out = AUDIO_DIR / alias / f"lv{level}" / f"item_{item_id}.mp3"
        if out.exists() and out.stat().st_size > 100:
            cached += 1
            continue
        tasks.append((item_id, text, level, alias, out))

total_target = len(items) * len(chosen_voices)
pct = cached / total_target * 100 if total_target else 0
print(f"Target: {total_target} · cached: {cached} · to generate: {len(tasks)}")
print(f"Resuming from {cached}/{total_target} ({pct:.1f}% already done)")
if not tasks:
    print("Nothing to do · all cached.")

The st_size > 100 condition on that line is a rough check to catch zero-byte files left when the disk fills up or the program is killed just after opening a file. It only catches the most obvious cases. It has no way to recognize a file missing half its contents. That is the job of the third piece, which I will cover next.

The second run produced this output, making it immediately clear that existing work was not being repeated.

$ python tts_batch.py --workers 5 --rps 0.3
Rate limit: 0.3 req/sec (18 req/min)
Loaded 5370 items from DB
Target: 26850 files · cached(already exist): 13936 · to generate: 12914
Resuming from 13936/26850 (51.9% already done)
 52%|#####     | 13936/26850 [00:00<20:01:12, 0.18file/s, ok=0, fail=0]

The progress bar must start from existing work, not zero

This is a small detail, but it matters a lot to whoever is watching the job. If the bar starts at 0 out of 12,914, the person watching cannot tell how much of the entire batch remains. I therefore set the total to the actual target and start the bar at the number already completed.

with ThreadPoolExecutor(max_workers=args.workers) as ex:
    futures = {ex.submit(generate_one, session, endpoint, t): t for t in tasks}
    with tqdm(total=total_target, initial=cached, unit="file", smoothing=0.1) as pbar:
        for fut in as_completed(futures):
            ok_flag, info = fut.result()
            if ok_flag:
                ok += 1
            else:
                fail += 1
                if fail <= 20:
                    tqdm.write(f"FAIL {futures[fut][1]}: {info}")
            pbar.update(1)
            pbar.set_postfix({"ok": ok, "fail": fail})

The single line initial=cached changes the display from 0 out of 12,914 to 13,936 out of 26,850—a number I can give someone when they ask. The if fail <= 20 condition keeps logs from flooding the screen. If the entire remote service goes down, I do not need twelve thousand error lines. The first twenty are enough to see that they all have the same cause.

The validator must be a separate program and inspect the first bytes

The third piece was the last one I added, and it was what made the file collection reliable. It has to be a separate program because the caller's job is to complete the batch, while the validator's job is to distrust the caller. If the two are combined into one program, the validator will always trust its own results.

There are just two layers to the check: first, check the minimum size; second, check whether the first bytes really are the header for that file type.

def is_mp3_valid(path, min_size):
    try:
        size = path.stat().st_size
    except OSError as e:
        return False, f"stat error: {e}"
    if size < min_size:
        return False, f"size={size}<min({min_size})"
    with open(path, "rb") as f:
        head = f.read(10)
    if len(head) < 4:
        return False, "header<4 bytes"
    if head[0] == 0xFF and (head[1] & 0xE0) == 0xE0:   # เฟรม MPEG audio
        return True, ""
    if head[:3] == b"ID3":                              # มีแท็ก ID3v2 นำหน้า
        return True, ""
    return False, f"bad magic: {head[:4].hex()}"

Do not pick the minimum size arbitrarily. I calculated it from the actual output: 24 kilohertz at 96 kilobits per second equals 12 kilobytes per second. The shortest words last about half a second, so 2,000 bytes is a threshold that valid files should never fall below. Set it too high and it deletes good files; set it too low and it lets bad files through.

Let the validator delete files, then run the caller again

I do not let the validator regenerate files itself, because that would leave two programs capable of making API calls, without a shared throttle controlling both. The cleaner approach is to let the validator only delete files, then let the original caller see the gaps and fill them on the next run.

# 1) ตรวจอย่างเดียว ยังไม่แตะอะไร
python verify_files.py --verbose

# 2) พอใจแล้วค่อยลบตัวเสีย
python verify_files.py --fix

# 3) วนกลับไปยิงรอบใหม่ เดี๋ยวมันเห็นช่องว่างแล้วเติมเอง
python tts_batch.py --workers 5 --rps 0.3

The validator must always default to report mode; deletion must require an explicit flag. I have been burned by a script that deleted files by default. One wrong minimum-size setting wiped out thousands of good files, and I had to spend another night making the calls again.

I also break the report down by group, because bad files concentrated in one group indicate a problem with that group's input rather than the network.

ตรวจสอบ 26850 ไฟล์ · OK: 26761 · BAD: 89

Per group:
  group-a       5361     9
  group-b       5352    18
  group-c       5310    60
  group-d       5369     1
  group-e       5369     1

BAD breakdown:
  group-c/lv4: 47
  group-c/lv5: 13

The example above immediately shows that something about group c at lv4 differs from the rest. It is not a random network drop, because network drops do not choose groups.

The sequence I actually use, ready to apply to other jobs

1. Find the actual limit before writing code—how many calls per how many seconds, and whether it is counted per account or per thread. Then set the actual rate about ten percent below that.
2. Build one shared throttle that every thread must pass through, and always call sleep outside the lock.
3. Separate permanent and transient errors. Return immediately for the first group; use exponential backoff with no more than 5 attempts for the second.
4. Write to a temporary name, then rename. The destination file will have only two states: absent or complete.
5. Rebuild the task list on every run, then filter out what is already on disk. Do not rely on a status file.
6. Start the progress bar from existing work so whoever is watching can read the overall count.
7. Write the validator as a separate program. Check the minimum size and the file's first bytes, and default to report mode.
8. Let the validator only delete files, then return to the caller to fill the gaps. Do not let two programs call the API at the same time.

I never put connection settings or passwords in the script. I read them only from environment variables, so nothing gets carried along when I put the script on another machine or send it to someone to look at.

import os, sys, requests

def make_session():
    key = os.environ.get("TTS_API_KEY")
    if not key:
        sys.exit("ยังไม่ได้ตั้ง TTS_API_KEY")
    s = requests.Session()
    s.headers.update({"Authorization": f"Bearer {key}",
                      "Content-Type": "application/ssml+xml"})
    s.mount("https://", requests.adapters.HTTPAdapter(
        pool_connections=32, pool_maxsize=64, max_retries=2))
    return s

I think these three pieces apply to any batch of API calls, not just audio generation. Whether downloading images, converting documents, or calling any kind of model, the same recipe can be reused whenever the remote service has a limit and the job runs overnight. Feel free to adjust the numbers or rewrite it in another language. If I have missed anything in the explanation or made a typo, I will come back and fix it later.