Truncating Before Redacting Leaked 31 Token Characters into Logs

I happened to ask another agent to scrutinize the command port code after I had finished writing the log token redactor and had a passing test confirming that "the log contains no token." The reviewer pointed out that the order was wrong: text was truncated before it was redacted, and if a caller put its token in the wrong field, half the token would end up in the file, on the status endpoint, and on screen, while my test stayed green.

This system is a real-time monitoring application I wrote in C#, with an engine running as a child process. The tool names, caller names, and token values in this article are all fictional. I rewrote the code as shorter C# examples for readability, but the sequence of events and the numbers are real.

The system involved: command port, audit log, and latest caller label

The engine exposes a command port on loopback for AI tools to use. The controls at this port include everything the textbook calls for: a token, an allow-list of permitted tools, a rate limit, an idempotency key, and a jsonl audit log recording every request. Each line stores caller, the name supplied by the caller itself, along with the tool name, status, and time.

The latest caller does not stay only in the file. It is forwarded to two other places. The first is a status endpoint on a separate port that requires no token. The second is an on-screen label showing who called most recently.

The caller field is free text supplied by the caller, so I limited it to 32 characters, a normal precaution for a free-text field written to a log. I also had a redactor that searched the text for tokens and replaced them with asterisks. Both measures looked reasonable in isolation.

The bug: truncate first, redact afterward

The problematic code looked roughly like this.

// รุ่นที่มีบั๊ก
static string Clip(string s, int max) => s.Length <= max ? s : s.Substring(0, max);

static string Redact(string s, string token) => s.Replace(token, "****");   // ค้นทั้งก้อน แบบ exact

static AuditEntry Build(string caller, string tool, string status, string token) => new AuditEntry
{
    Caller = Redact(Clip(caller, 32), token),      // ตัดเหลือ 32 ก่อน แล้วค่อยส่งเข้า redact
    Tool = tool,
    Status = status,
    At = DateTimeOffset.UtcNow,
};

The system token is 64 hex characters long. If a caller uses the token as its caller name, which can happen when fields get swapped in the caller's config, Clip first cuts it down to the first 31-32 hex characters of the token. Then Redact searches for the full 64-character token in text only half that length, which can never match. Nothing gets replaced, so that half-token continues to every destination.

{"at":"2026-10-01T09:00:00Z","caller":"0123456789abcdef0123456789abcde","tool":"get_status","status":"ok"}

The line above shows what ended up in the file (with fictional values). Those 31 hex characters of the token were in the audit file, on the status endpoint that any process on the machine could GET without authentication, and on the screen label.

Why the original test stayed green

The test I was proud to already have looked like this.

// เทสต์เดิม: ผ่านเขียวมาตลอด และพิสูจน์ผิดสิ่ง
[Fact]
public void AuditLogDoesNotContainToken()
{
    string tok = SampleToken();
    string log = RunRequestAndReadAuditLog(caller: tok, tool: "get_status");
    Assert.DoesNotContain(tok, log);          // ถามว่า "มี token ทั้งก้อนไหม" คำตอบคือไม่มี จริง ๆ
}

The assertion was literally true. The log really did not contain the complete token; it contained only half of it. This test used the same condition as the redactor, looking for the whole token, so it had exactly the same blind spot. The bug that fooled the redactor fooled the test at the same time. A test that uses the same logic as the code it checks is not checking anything extra.

My first wrong assumption: changing the order would be enough

When I saw the bug, I thought a one-line fix would suffice: switch to Clip(Redact(caller, token), 32) and call it done. But when I traced every point where external text entered the log, I found at least two more ways it could leak.

First, the request body was also truncated at 64 KB before being passed to the redactor. It was the same ordering mistake in another place. If a token happened to straddle the cutoff, its leading portion would survive.

Second, the redactor used exact matching. If the token arrived as uppercase hex, it would not match the stored lowercase token, even though it was the same secret, bit for bit.

These two findings showed that the problem went beyond ordering. The redactor itself was too fragile. It recognized the secret in only one form: an exact character-for-character match with the stored value.

The three fixes I put in place

1. Always redact every string before truncating it This applies to both caller and body, without exception, so truncation only sees text that is already clean.
2. The redactor removes runs, not just complete tokens It finds contiguous hex runs of 16 characters or more. If any 16-character window in a run matches any part of the token, it removes the entire run, regardless of case.
3. Tests iterate over every 16-character window of the token They assert that none of those windows appear in the log, on the status endpoint, or in the text returned to the caller.

The new redactor does not take much code.

public sealed class Redactor
{
    private const int Win = 16;
    private readonly HashSet<string> _windows = new(StringComparer.Ordinal);

    // รับความลับได้หลายตัว: token กลาง, credential ของหน้าจอ ฯลฯ
    public Redactor(params string[] secrets)
    {
        foreach (string s in secrets)
        {
            string t = s.ToLowerInvariant();
            for (int i = 0; i + Win <= t.Length; i++) _windows.Add(t.Substring(i, Win));
        }
    }

    public string Redact(string s)
    {
        if (string.IsNullOrEmpty(s)) return s;
        var sb = new StringBuilder(s.Length);
        int pos = 0;
        while (pos < s.Length)
        {
            if (!Uri.IsHexDigit(s[pos])) { sb.Append(s[pos++]); continue; }
            int end = pos;
            while (end < s.Length && Uri.IsHexDigit(s[end])) end++;      // ช่วง hex ต่อเนื่อง [pos, end)
            string run = s.Substring(pos, end - pos);
            sb.Append(Hits(run) ? "****" : run);
            pos = end;
        }
        return sb.ToString();
    }

    private bool Hits(string run)
    {
        if (run.Length < Win) return false;
        string r = run.ToLowerInvariant();
        for (int i = 0; i + Win <= r.Length; i++)
            if (_windows.Contains(r.Substring(i, Win))) return true;
        return false;
    }
}

And the call site uses the one correct order.

static string Clip(string s, int max) => s.Length <= max ? s : s.Substring(0, max);

entry.Caller = Clip(redactor.Redact(caller), 32);            // ลำดับนี้เท่านั้น
entry.Body   = Clip(redactor.Redact(body), 64 * 1024);       // body ก็เหมือนกัน

Why 16 characters? This number is a tradeoff between two risks. If the window is too short, say 4 characters, ordinary text that happens to contain four hex characters matching part of the token will also get removed. The log would fill with asterisks until it became unreadable. If the window is too long, fragments shorter than the window can slip through. 16 hex characters represent 64 bits, making accidental matches in ordinary text extremely unlikely, and fragments shorter than 16 characters reveal less than a quarter of the token.

The new tests: every window, every destination

The new tests differ from the old ones in two ways. They look for fragments instead of the whole token, and they check every place caller reaches, rather than just the log file.

private static string SampleToken() => string.Concat(Enumerable.Repeat("0123456789abcdef", 4));   // 64 ตัว ของสมมุติ

public static IEnumerable<object[]> Shapes()
{
    string tok = SampleToken();
    yield return new object[] { tok };                               // ทั้งก้อน
    yield return new object[] { tok.ToUpperInvariant() };            // ตัวพิมพ์ใหญ่
    yield return new object[] { "agent-" + tok };                    // มีคำนำหน้า รอยตัดเลื่อน
    yield return new object[] { tok.Substring(20) + tok };           // ชิ้นส่วนต่อกับก้อนเต็ม
}

[Theory]
[MemberData(nameof(Shapes))]
public void NoWindowOfTheTokenSurvivesAnywhere(string caller)
{
    string tok = SampleToken();
    using var h = TestHost.Start(tok);

    string reply  = h.Call(caller: caller, tool: "get_status");
    string audit  = h.ReadAuditLog();
    string status = h.GetStatusEndpoint();                           // endpoint ที่ไม่ต้องใช้ token

    foreach (string sink in new[] { audit, status, reply })
        for (int i = 0; i + 16 <= tok.Length; i++)
            Assert.DoesNotContain(tok.Substring(i, 16), sink, StringComparison.OrdinalIgnoreCase);
}

A 64-character token has 49 windows of 16 characters, multiplied by three destinations, multiplied by the number of caller variants supplied. Against the old code, this test must fail, because the leaked 31 hex characters contain as many as 16 of the token's 16-character windows. That is what I should have seen on day one. Another benefit is that the test is no longer tied to how the redactor works. Even if I replace the redactor with an entirely different approach later, the test still asks the same question: do any fragments of the secret appear at any destination?

Three related findings from the same review

The reviewer found more than the ordering issue. Three other findings belonged to the same family: secrets and command-port controls that looked correct on paper.

A secret the tool writes to a git-tracked config file The tool generated a token and wrote it to the same config file held in the repo. Without a safeguard, the next blanket commit would carry the token into the repo too. I deliberately kept this value out of the commit, leaving null in the index. Verification must inspect the index, not the working tree, like this.

# ดูค่าที่อยู่ใน index (ของที่จะถูก commit) ไม่ใช่ไฟล์บนดิสก์
git show :configs/profile.json | grep -n '"token"'
#   12:  "token": null,

# กวาดทั้ง index หา hex ยาว 64 ตัวที่ไม่ควรอยู่ตรงนั้น
git grep --cached -nE '\b[0-9a-f]{64}\b' -- configs/ && echo "[BLOCK] มี hex 64 ตัวใน index" || echo "[OK]"

Per-IP lockout does not work on a loopback port The rule "5 failures in one minute locks that IP" is a good rule on a public port. On loopback, though, every caller is 127.0.0.1. One tool supplies the wrong token, and every caller on the machine gets locked out, including those doing everything correctly. Whenever I bring a mechanism over from another context, I need to ask whether the key it uses to distinguish callers can still distinguish them in the new context.

The shortcut for my own UI needs a separate credential The control UI needs to make some calls outside the AI tools' allow-list. The path that bypasses the allow-list for my own UI therefore needs its own credential, randomly generated every time the engine starts and written to a file the UI can read. It must not share the AI token, because then anyone holding the AI token could immediately bypass the allow-list. This value must also be passed to the redactor, or the debug log will become a new leak path.

// ตอนเอนจินสตาร์ต: credential ของหน้าจอ สุ่มใหม่ทุกรอบ อายุเท่ากับ process นี้
string uiKey = Convert.ToHexString(RandomNumberGenerator.GetBytes(32)).ToLowerInvariant();
File.WriteAllText(Path.Combine(stateDir, "ui.key"), uiKey);

// ตัว redact ต้องรู้จักความลับทุกตัวที่ process นี้ถืออยู่ ไม่ใช่แค่ตัวแรกที่นึกถึง
var redactor = new Redactor(apiSecret, uiKey);

Another way logs leak data is through loose file permissions followed by off-machine backups. I covered that separately in the article about 0644 logs uploaded every night. These two issues complement each other: that one is about logs ending up in the wrong place; this one is about data that should never have been in the logs.

Numbers and lessons

I should be precise about the scale of this issue. 31 hex characters represent 124 bits out of 256 bits. The remaining 132 bits are still infeasible to guess in practice. This was therefore not an incident that rendered the token unusable. It was evidence that my rule, "secrets never enter logs," was not true in the way my tests claimed. And if the token had been shorter, or the field's truncation limit longer, the outcome would have been very different.

The short lesson I now apply anywhere external text gets logged is to always redact before truncating. Every text transformation must happen after redaction, not before, and tests must look for fragments of secrets at every destination, not the whole secret at a single destination. Another lesson is that having someone else, or another agent, scrutinize security code is well worth it, because the author and the author's tests often share the same blind spot. Anyone using the redactor and tests above can adjust the window size to suit their own token length. As for cases where a token is transformed into another form before reaching the log, which this redactor still cannot catch, I will come back and address those later.