Cross-Model Audits of AI-Written Code — Process, Not Faith

I had a set of structural calculation spreadsheets that people had been using for years, with formulas spread across 35 sheets, totaling more than eight thousand cells. I wanted a website I could use on my phone on site, so I had an AI agent port the formulas into web code. The work moved very quickly, but once it was done, I did not dare use it, because I had no way to know whether the ported formulas actually matched the original or just "looked similar." So I tried something that sounded silly at first: having an agent from another provider, which had not written this code, audit it, then repeating that for three rounds.

The real problem is having nothing to compare against

When porting code from one language to another, I have the existing tests to catch mistakes. When porting from a spreadsheet, there is nothing like that. What I have are cells containing both formulas and their last calculated values. That is the starting point.

The first thing I asked for, before touching a single line of code, was to inventory the formulas in the entire workbook, so I would know what I was dealing with, instead of porting along and then guessing whether everything was covered.

import json, openpyxl

wb = openpyxl.load_workbook('workbook.xlsm', data_only=False)
inv = {}
for ws in wb.worksheets:
    cells = [(c.coordinate, c.value) for row in ws.iter_rows() for c in row
             if isinstance(c.value, str) and c.value.startswith('=')]
    if cells:
        inv[ws.title] = cells

print('sheets with formulas:', len(inv))
print('formula cells       :', sum(len(v) for v in inv.values()))
json.dump(inv, open('formula_inventory.json', 'w'), ensure_ascii=False)

Once I saw those numbers, I immediately dropped the idea of "porting every cell," because most cells were helpers, form controls, or hidden formulas that never appeared in reports. What needed to match was the set of formulas producing results on screen, which was much smaller, but I had to identify exactly which ones.

Rule one: the auditor reads the original, not the previous conclusions

I found in the first round that if I let the auditor read the previous audit report, it would start from "the last round concluded that it passed" and look for supporting reasons, instead of actually checking again. The result was a longer report with no new findings.

The rules I set and found effective covered four points.

I put these rules in a single file in the project and made reading it the first instruction for both agents in every round. I did not type the instructions again each time, because doing that gradually lets the rules drift.

AUDIT-RULES.md  (ทั้งสองฝั่งต้องอ่านก่อนเริ่มทุกรอบ)

1. ต้นฉบับคือไฟล์สเปรดชีตเท่านั้น รายงาน audit รอบก่อนไม่ใช่ต้นฉบับ
2. อ่านโค้ดจากสถานะปัจจุบันของไฟล์ ห้ามตัดสินจาก diff อย่างเดียว
3. ทุกข้อค้นพบต้องมี: อ้างอิงเซลล์ต้นฉบับ + รูปสูตรสองฝั่ง + คำสั่งรันซ้ำ
4. ผู้เขียนโค้ดแก้ได้เฉพาะซอร์สและเทส ห้ามแก้ไฟล์รายงาน
5. ผู้ตรวจเขียนได้เฉพาะรายงานของตัวเอง ห้ามแก้ซอร์ส
6. จบรอบต้องมีหัวข้อ "สิ่งที่ยังไม่ได้ตรวจ" เสมอ

The rule I think people overlook most is "read the current code, not the diff", because an auditor looking at a diff sees only what changed and misses bugs that have been sitting in the existing code from the start. I found two of those in my rounds.

Classify findings and require evidence

A useful report has to make it clear what to do first. So I required every finding to follow the same structure and classified them into three levels: P0 means incorrect numbers that must be fixed before use, P1 means the method differs from the original but the results are still acceptable, and P2 covers presentation or wording.

{
  "id": "P0-03",
  "severity": "P0",
  "area": "แรงเฉือนของบันได",
  "source_ref": "ชีต Stair1 เซลล์ AC7",
  "source_shape": "(w * L / 2) - (w * d)",
  "code_ref": "calc-rc.js :: rcStairCalc",
  "code_shape": "w * L / 2",
  "evidence_cmd": "node validate_excel.js --case stair1",
  "observed": "1359.607",
  "expected": "1271.026",
  "fix": "หักระยะ effective depth ออกจากแรงเฉือนที่ฐานรองรับ"
}

The source_shape and code_shape fields are the core of this, because they force the auditor to write out both formulas side by side, instead of making an unsupported statement that "the check found a mismatch." Often, just writing those two lines makes it obvious which side is wrong.

For evidence, I would not accept a finding without an attached command I could run again.

node validate.js          # PASS 108/108 | worst diff = 0.000%
node validate_excel.js    # PASS 55/55 fixtures
node --check calc-rc.js calc-foundation.js templates.js

Classes of errors AI kept making in every round

This is what I got from running three rounds and then sitting down to group all the findings. They turned out to fall into just three recurring patterns, regardless of which provider's agent I used.

One: porting a formula into an equivalent but differently written algebraic form. The numbers matched exactly in every tested case, but stopped matching under a different loading condition, because the original enclosed the entire denominator in parentheses while the code divided by each term in sequence. In normal cases they were equivalent; only when a zero or negative term appeared did things break.

// รูปที่ต้นฉบับเขียน: (w / (8 * L^2)) * (L + a)^2 * (L - a)^2
const m1 = (w / (8 * L ** 2)) * (L + a) ** 2 * (L - a) ** 2;

// รูปที่เอเจนต์พอร์ตมารอบแรก — เท่ากันเฉพาะบางช่วงค่า
const wrong = w / 8 / L ** 2 * ((L + a) * (L - a)) ** 2;

Two: using a table from the wrong edition of a standard. This was much more dangerous than the first case, because every line of code was correct, but the constants came from a different year's edition of the standard. The resulting numbers were wrong across the board, while every test passed because the tests used values from that same table. My fix was to prohibit embedding values from standards tables in code without identifying their source. Every value had to come from a data file with fields specifying the edition and year.

Three: copying table values one column off. This is the same kind of mistake people make: the original table had merged header cells, and reading it row by row led to picking values from the next column over. Visual inspection could not catch it. The only way to find it was to write a script that read values from the original and compared them, one by one, with those embedded in the code.

# เทียบค่าคงที่ในโค้ดกับค่าที่อ่านจากต้นฉบับ ทีละตัว ไม่เชื่อสายตา
import json
src  = json.load(open('table_from_source.json'))
code = json.load(open('table_in_code.json'))
for k in sorted(src):
    if abs(src[k] - code.get(k, float('nan'))) > 1e-9:
        print(f'MISMATCH {k}: source={src[k]} code={code.get(k)}')

Confidence levels to keep claims within the evidence

A problem surfaced in round two: the website and documentation started saying things like "matches the original in every case, zero percent difference." That was an overstatement, because only the tested set had passed, not all eight thousand cells. So I required every claim to carry one of these three labels.

The third label matters just as much as the first two, because in some places I deliberately differed from the original, for example by choosing a more conservative method. Without that label, the next auditor would report it as an error every round, wasting time on both sides.

There were two rules attached to the labels. First, a topic with no comparison cases saved in a file must never be labeled validated exact. Second, the wording on the website must draw from the same numbers printed by the validator, with no manual entry, or one day the tests will fail while the website still says they pass.

{
  "generated_by": "validate_excel.js",
  "generated_at": "2026-11-20T09:14:22+07:00",
  "claims": [
    { "area": "แผ่นพื้นสองทาง", "level": "validated exact",
      "fixtures": 12, "worst_diff": "0.000%" },
    { "area": "จันทันตะเข้", "level": "formula aligned",
      "fixtures": 0, "note": "สูตรตรงรูป แต่ยังไม่มีเคสเทียบเก็บไว้" },
    { "area": "แรงเฉือนฐานราก", "level": "standard extension",
      "fixtures": 4, "note": "ตั้งใจใช้วิธีอนุรักษ์กว่าต้นฉบับ" }
  ]
}

The website reads this file and renders it directly, so the wording cannot go beyond the evidence, because this file is generated only by running the validator. If it has not been run, the file is old and has a visible date.

What happens in the three-round loop

1. Round one: the auditor reads the original and the current code, then produces a report classifying findings as P0/P1/P2, with evidence for each. The code author has not touched anything yet.
2. The author fixes only P0 findings and adds comparison cases from the original for every fix. Editing the report is prohibited.
3. Round two: the auditor starts again from the original without reading the first report, then compares the two reports at the end.
4. Clean up wording that overstates the evidence and apply the three confidence labels to every topic.
5. Round three: check only formula form and presentation, such as the multiplication and division signs and parentheses shown on screen, to ensure they allow only one interpretation.
6. Close the round by recording what has not been checked, not just what passed.

That last point is what I found most useful in the entire process. A report ending with a list of "things not yet checked" lets the next round start where nobody has looked, instead of repeatedly checking the same things.

What this loop does not cover

To be direct, this process answers only one question: does the code calculate the same results as the original file for the tested set? It does not answer whether the original file is technically correct, whether it is suitable for the intended work, or replace review by a qualified professional.

The actual remaining gap in my work is coverage, rather than correctness in the parts I know about. Several sheets have their formulas inventoried but still lack comparison cases. Until those exist, I label them honestly as unchecked instead of saying they passed.

Another thing I did not do was compare which provider's agent was better, because what I found was that they made mistakes in different places, not that one was better than the other in every respect. The benefit of using another provider's agent to audit the work is that it does not make the same mistakes, which is the same reason I have someone else read my code in the first place. Anyone adapting this can run as many rounds as time allows. If I have left anything unclear or there are rules worth adding, I will come back and update this later.