One Stage Dies Silently: Forward Hook Turns NamedTuple into tuple
A job that dies and reports exit 0 is one of the most convincing lies a background system can tell, because it keeps me from looking for a traceback that was never written to a file. The bug itself was smaller than I expected: a single hook that rebuilt a result with the wrong type.
1-minute summary
- Symptom / problem
- A GPU training job dies within a minute, only at the stage with the discriminator, yet exits with 0.
- Who should read
- People training image models in PyTorch with forward hooks and detached jobs.
- Fix in short
- Make the forward hook return the original type. tuple(...) strips the NamedTuple field names. Add a test to prevent a repeat.
I was running an image model training job on a GPU using open-source PyTorch code with several stages. The early stages ran normally. At stage 3, which enabled adversarial training (setting the adversarial loss weight above zero, bringing in a discriminator), the job finished loading data samples, then quietly exited about 1 minute later. The captured log contained a single line, TRAIN_EXIT 0, with no traceback or any other message to read.
I am not identifying the code, project name, or machine used. I rewrote the code examples in this article as self-contained, runnable examples using only PyTorch, and tested them before publishing. This is a bug involving forward hooks and Python data types that could occur in any code, regardless of the workload.
Symptoms: the stage with a discriminator dies, while the others run normally
The strangest part was not that the job died, but that it died in only one stage. Stages without a discriminator ran normally. Once adversarial training was enabled, the job ended in under a minute, and all I could see from the system was this:
- The job was launched detached, independent of the SSH session. The captured log contained only the line
TRAIN_EXIT 0. - A watchdog automatically relaunched the job about 3 times per hour, then stopped. Anyone checking the process list periodically would keep seeing the job reappear, making it look as though it was "running."
- There was no traceback anywhere, because stdout and stderr were not captured when the job detached.
A wrapper that captures a child process's exit code but does not forward stderr can only say "the process ended," not why it ended. A watchdog that relaunches without capturing stderr hides this kind of bug even more effectively, because a new process keeps appearing.

Investigation: rerun in the foreground to see the actual messages
With a log this empty, there was nothing left to make an informed guess from. I went back to the simplest thing I should have done from the start: run the stage 3 .bat in the foreground over SSH, with stdout and stderr displayed on my screen.
# foreground + รวม stderr เข้า stdout ให้เห็นทุกอย่างที่จอ (ฝั่ง Windows ที่เป็นเครื่องเทรน)
ssh host-a 'cmd /c train_stage3.bat 2>&1'
This time, the message appeared right on the screen.
Traceback (most recent call last):
...
File "train_step.py", line 142, in train_step
d_real = pred_src_src_d.center
AttributeError: 'tuple' object has no attribute 'center'
That single line said almost everything: the training code expected the discriminator's result to be an object with a field named center, but received a plain tuple. I replaced the filename and line number in the example above with made-up values.
The cause: a forward hook that rebuilds the output with tuple(...)
The discriminator's result in that code was a NamedTuple with a center field (which I will call DiscOutput in this article). Meanwhile, the training code registered a forward hook on every network, including the discriminator, when training on CUDA. The hook's job was to cast the output's numeric type, and it used tuple(...) to rebuild the result. That was where the NamedTuple lost its field names.
This is ordinary Python behavior and has nothing to do with PyTorch. I can check it immediately in a REPL.
from typing import NamedTuple
class DiscOutput(NamedTuple):
center: float
scales: float
o = DiscOutput(center=1.0, scales=2.0)
print(o.center) # 1.0
t = tuple(v for v in o) # สร้างใหม่จาก iterable ของค่า
print(t) # (1.0, 2.0)
print(type(t).__name__) # tuple
print(hasattr(t, "center")) # False <- ชื่อฟิลด์หายไปแล้ว
print(DiscOutput._fields) # ('center', 'scales') <- ตัวบอกว่าเป็น NamedTuple
tuple(...) simply takes the values yielded by iteration and builds a new tuple, with no knowledge of the original class. A NamedTuple is a tuple with names attached, but those names belong to the class. Rebuilding it with tuple(...) produces an instance of the base class without those names.
Here is a complete example that runs in a single file. I wrote it to match the real situation as closely as possible: a module that returns DiscOutput and a hook that only touches it when a cast is needed.
# hook_bug.py - ทำซ้ำบั๊กด้วย PyTorch ล้วน ๆ
from typing import NamedTuple
import torch
import torch.nn as nn
class DiscOutput(NamedTuple):
center: torch.Tensor
scales: torch.Tensor
class Disc(nn.Module):
def __init__(self):
super().__init__()
self.conv = nn.Conv2d(3, 1, 3, padding=1)
def forward(self, x):
y = self.conv(x)
return DiscOutput(center=y, scales=y.mean(dim=(2, 3)))
def cast_outputs_hook_buggy(module, args, out):
if all(t.dtype == torch.float32 for t in out):
return None # เป็น fp32 อยู่แล้ว ไม่แตะ
return tuple(t.float() for t in out) # <-- สร้างใหม่ด้วย tuple: NamedTuple หาย
def run(hook, dtype):
net = Disc().to(dtype)
h = net.register_forward_hook(hook)
out = net(torch.randn(2, 3, 8, 8).to(dtype))
h.remove()
return out
# fp32: hook ข้าม ได้ DiscOutput เดิม
print(type(run(cast_outputs_hook_buggy, torch.float32)).__name__) # DiscOutput
# bf16 (เส้นทางที่ต้อง cast): hook ทำงาน ได้ tuple ธรรมดา
out = run(cast_outputs_hook_buggy, torch.bfloat16)
print(type(out).__name__, out[0].dtype) # tuple torch.float32
out.center # AttributeError
When run, the last line produces AttributeError: 'tuple' object has no attribute 'center', exactly the same message I saw in the training job.
Why only some stages fail
I am describing only what I actually observed. This hook touches the output only when a cast is needed. When the output is already fp32, it does not need to rebuild anything, so the NamedTuple passes through with its type and field names intact. In the path where the output is not fp32 (the autocast path), which occurs in the stage with a discriminator, the hook rebuilds the result with tuple(...). That is where the name .center disappears. The other stages have no discriminator, so they never encounter this path. I will not go into the exact rules the hook uses to decide when to cast, because I have not read the original author's reasoning.
The fix: have the hook return the original type
I changed only this part, having the hook check whether the output has _fields. If it does (meaning it is a NamedTuple), it rebuilds it using the original class. Otherwise, it returns a plain tuple as before.
def cast_outputs_hook_fixed(module, args, out):
if all(t.dtype == torch.float32 for t in out):
return None
vals = [t.float() for t in out]
if hasattr(out, "_fields"): # NamedTuple: คืนชนิดเดิมพร้อมชื่อฟิลด์
return type(out)(*vals)
return tuple(vals) # tuple ธรรมดา ทำเหมือนเดิม
out = run(cast_outputs_hook_fixed, torch.bfloat16)
print(type(out).__name__, out.center.dtype, out.center.shape)
# DiscOutput torch.float32 torch.Size([2, 1, 8, 8])
type(out)(*vals) works because a NamedTuple accepts positional values in field order, which exactly matches the iteration order of out. If the output is some other structure that is neither a tuple nor a NamedTuple (such as a dict or dataclass), it needs its own handling. This example does not cover that.
In the real code, I edited the dependency's file directly, added a comment marking it as a local patch with the date, and kept the original file alongside it as .orig_<วันที่>.
# ก่อนแก้ เก็บของเดิมไว้ข้างไฟล์ที่แก้
cp trainer/hooks.py trainer/hooks.py.orig_20270401
# ในไฟล์ที่แก้ ติดคอมเมนต์ให้คนถัดไปเห็น
# # LOCAL PATCH 2027-04-01: คืนชนิดเดิมเมื่อ out เป็น NamedTuple (มี _fields)
# # ถ้าอัปเดตหรือติดตั้ง dependency ใหม่ ต้อง re-apply จุดนี้
# เทียบหลังอัปเดตว่า patch ยังอยู่ไหม
diff -u trainer/hooks.py.orig_20270401 trainer/hooks.py
After the fix, stage 3 kept running instead of ending after 1 minute. GPU utilization was about 99% and VRAM usage around 8.6 GB, which I consider evidence that the job was actually working, beyond simply having "no error."

Preventing a recurrence: write tests for this hook
After fixing it, my only regret was that this hook was small enough for a twenty-line test to catch the bug on day one, without waiting for a training job to die. The tests need to cover two things that were missed: the output type must not change after passing through the hook, and both the fp32 path and the path that requires a cast must run.
# test_hook.py - รันด้วย pytest ก่อนปล่อยงานเทรนยาว ๆ
import pytest
import torch
from hook_bug import Disc, DiscOutput, cast_outputs_hook_fixed
@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
def test_hook_keeps_namedtuple(dtype):
net = Disc().to(dtype)
net.register_forward_hook(cast_outputs_hook_fixed)
out = net(torch.randn(2, 3, 8, 8).to(dtype))
assert isinstance(out, DiscOutput) # ชนิดต้องไม่เปลี่ยน
assert out.center.dtype == torch.float32 # และต้อง cast แล้วจริง
This test needs no GPU and runs on an ordinary machine in a few seconds, answering a question that a training job takes much longer to answer. Personally, whenever a hook touches model output and builds a new value, I always require a test that checks the type first, because Python does not complain when the type changes but still yields the same values during iteration. It only complains when someone accesses a field by name.
Operational lesson: require evidence that it is running, beyond an exit code
The bug was small, but the way I checked background jobs made it time-consuming. I wrote down these rules to use for every detached job.
- "Detached and showing exit 0" does not mean the job succeeded. There must be evidence that the process is still running, with all four conditions met at once: it appears in the process list, the GPU is being used, the output file's mtime is advancing, and there is no Traceback at the end of the log.
- If the log is empty or unusually short, run the same command in the foreground with
2>&1to see the traceback on screen before making any guesses. - A wrapper or watchdog that relaunches must capture stderr to a file on every relaunch. Without that, it is a machine for hiding bugs.
- A local patch in a dependency will be overwritten during an update or reinstall. Keep
.orig, note that the patch needs to be re-applied, and consider sending the fix upstream.
I turned the first rule into a short check script to run before trusting that a job is still making progress. Here is an example for Windows (PowerShell) with an NVIDIA GPU.
# check_training.ps1 - หลักฐาน 4 อย่างว่างานยังรันจริง
$out = "C:\train\output\model.ckpt"
$log = "C:\train\logs\stage3.log"
$proc = Get-Process python -ErrorAction SilentlyContinue
$gpu = (nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader)
$age = if (Test-Path $out) { [int]((Get-Date) - (Get-Item $out).LastWriteTime).TotalMinutes } else { -1 }
$tb = if (Test-Path $log) { Select-String -Path $log -Pattern 'Traceback' -Quiet } else { $false }
"process : {0}" -f $(if ($proc) { "รันอยู่ ($($proc.Count))" } else { "ไม่มี" })
"gpu : $gpu"
"output : แก้ไขล่าสุด $age นาทีก่อน"
"log : $(if ($tb) { 'พบ Traceback' } else { 'ไม่พบ Traceback' })"
if (-not $proc -or $tb) { Write-Host "[FAIL] งานไม่ได้รัน อย่าเชื่อ exit code"; exit 1 }
Write-Host "[OK] มีหลักฐานว่างานยังเดิน"
Each can be misleading on its own. A process with an idle GPU may be stuck waiting. A recently updated output file may belong to the previous relaunch. A log with no Traceback may never have been written to at all. Taken together, they provide enough confidence.
The numbers in this case
There are three things I still do not know and will not guess at: whether this bug still exists or has been fixed in the latest upstream version, why the original hook author chose tuple(...) in the first place, and whether this fix affects model accuracy. I only know that the job could continue after the fix. Anyone who wants to extend the hasattr(out, "_fields") condition to cover other structures or send it upstream is welcome to do so. I will come back and make further changes later.
Want us to set this up?
Get help with server setup, configuration, migration, or troubleshooting. Save time for the work that matters.
Hit something like this? Stuck somewhere?
Come trade notes on the LinuxThai board



Comments 0
No comments yet — be the first