4K Playback Fails When Image Subtitles Force a Full Transcode

I set up a home video library on a Proxmox machine that already ran VMs for other workloads, using Jellyfin in a small container. Ordinary files played normally until one day someone at home tried a 4K file and got a spinning indicator with no picture. I blamed Wi-Fi, the TV, and the LAN port in turn. In the end, the answer was in a single server log line: it was burning subtitles into 4K video with a 2 core CPU at 8 fps. I wrote down the troubleshooting sequence and the fixes here.

I have replaced all filenames, account names, and IP addresses in this article with example values. The measured figures and log messages are real.

A video library on LXC designed to interfere with the host as little as possible

This host's main job is running other VMs, so I set up Jellyfin in a separate unprivileged LXC running Debian and limited its resources from the start. The video folder on the host's HDD is mounted read-only inside the container. The container connects to the existing bridge without changing the host's network configuration.

# /etc/pve/lxc/200.conf (เลข container เป็นค่าตัวอย่าง)
arch: amd64
cores: 2
cpuunits: 50
memory: 1536
swap: 512
rootfs: local:200/vm-200-disk-0.raw,size=16G
mp0: /mnt/data/video,mp=/media/video,ro=1
startup: order=99,up=60
unprivileged: 1

cpuunits: 50 is half the value of a normal guest, so when workloads compete for CPU, this container gets a lower priority. The 16G rootfs sits on the OS disk, separate from the VM pool, and startup: order=99 makes it start near the end, after the main workloads are up.

Inside the container, I configured nftables to open port 8096 only to IPv4 addresses on the LAN and blocked all IPv6, because the container automatically received a public IPv6 address from the router. Without blocking it, I would be exposing the login page to the internet. I also disabled DLNA and auto-discovery (UDP 7359).

# /etc/nftables.conf ใน container
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    iif "lo" accept
    ct state established,related accept
    meta nfproto ipv6 drop
    ip saddr 192.168.10.0/24 tcp dport 8096 accept
  }
}

One firewall mistake I made was initially allowing access only from the TV's IP address. The next day, someone at home reported that “the server is down; it didn't auto start.” I checked, and the server had never gone down. What had changed was the TV's IP address: DHCP had reassigned it from 192.168.10.190 to 192.168.10.134, so the old rule blocked the TV. I changed the rule to allow the whole LAN and relied on Jellyfin's login instead.

First symptom: a 4K file would not start playing

The problem file was 4K HEVC 10-bit HDR with a bitrate of around 80 Mbps. Pressing play left the indicator spinning. My guesses, in order, were insufficient Wi-Fi bandwidth, a TV too slow to decode the video, and a LAN port running at only 100 Mbps. The last one was true, so I moved it to a 1 Gbps port, but the file still would not play. That ruled it out as the cause in this case.

Only when I opened Jellyfin's logs did I see that the server was not sending the original file. It had instructed ffmpeg to burn Thai PGS subtitles, which are image subtitles, from a track flagged Forced into the 4K video because that player could not receive and render the subtitle images separately. The ffmpeg progress line gave the figures directly.

# ดู log การแปลงล่าสุดใน container (ตัดให้เหลือเฉพาะค่าที่ใช้)
$ ls -t /var/log/jellyfin/FFmpeg.Transcode-*.log | head -1 | xargs tail -n 3
frame= ... fps=8.0 ... speed=0.33x

Converting a 24 fps video at 8 fps means producing 1 second of video for every 3 seconds of work. No amount of waiting for the buffer would let the player keep up. Meanwhile, the lingering ffmpeg process slowed the entire host for around 12 minutes, even though I had set cpuunits low.

How the server decides what to send

Jellyfin has three levels of video delivery, from lightest to heaviest. It tries the top level first, then gets pushed down one level at a time when the player says it cannot handle the format.

Direct play sends the original file as it is, with the server acting only as file storage. Remux, called direct stream in Jellyfin's interface, takes the video and audio streams out of their original container and wraps them in a container the player can open. It does not decode the video and uses very little CPU. Transcode decodes every video frame and re-encodes it, which is a job this 2 core container was not designed to do.

I encountered three groups of reasons that push the job down a level: a container the player cannot open, a subtitle codec the player cannot render, and a bitrate or resolution limit set in the app. Every time the server makes a decision, it writes the selected method and the reason to the log, ready to read. There is no need to guess.

Second symptom: an ordinary 1080p file froze partway through

A week later, an ordinary 1080p H.264 file froze partway through playback. My first guess was that the file was corrupt at that point, so I had ffmpeg decode just the section where it froze and discard the output. If the file was corrupt, it would produce errors.

# ถอดรหัสช่วงที่ค้าง 2 นาทีแล้วทิ้งผล ถ้าไม่มีอะไรพิมพ์ออกมา = ไฟล์ไม่เสีย
ffmpeg -v error -ss 00:41:00 -t 120 -i /media/video/sample.mkv -map 0:v:0 -f null -

# ดูว่า cgroup ของ container ถูกบีบหน่วยความจำกี่ครั้ง (รันบนโฮสต์)
grep -E '^(high|oom_kill) ' /sys/fs/cgroup/lxc/200/memory.events
high 10996
oom_kill 0

The file produced no errors at all, but the memory figures told a different story. During playback, the container's RAM usage reached 1467 out of 1536 MB. The high counter in memory.events reached 10,996, while oom_kill was 0. That meant the container had not been killed; the kernel was throttling it each time it hit the memory.high limit. The visible symptom was intermittent freezing, not a shutdown.

Someone at home discovered that the app's max resolution was set to 4K, causing the 1080p file to be upscaled and re-encoded, which did add load. But when I read that session's logs, the server had recorded a different reason.

# เหตุผลของ session ที่ค้าง (รูปแบบย่อจาก log ของเซิร์ฟเวอร์)
method = transcode, reasons = ContainerNotSupported, SubtitleCodecNotSupported

That app used the TV box's system player, which could not open .mkv files directly and did not support the file's subtitle codec, so it asked the server to transcode. What I had not expected was that even with the account policy set to prohibit video transcoding, the app could still call the server's transcode endpoint directly. Setting the policy alone was therefore not enough. I had to check the logs every time to confirm which method the actual session was using.

The GPU in this machine could not help

My next question was whether the GPU could handle the transcoding. This machine had an older desktop CPU with no iGPU. The installed graphics card was an older low-end model without NVENC, and the host used the nouveau driver. There was no /dev/dri available inside the container.

The misleading part was ffmpeg's encoder list: it included h264_nvenc, h264_qsv, and h264_vaapi. That meant ffmpeg had been compiled with support for those encoders, not that the machine had hardware they could use.

# รายชื่อนี้คือ compile-in ไม่ใช่ของที่ใช้ได้จริง
/usr/lib/jellyfin-ffmpeg/ffmpeg -hide_banner -encoders | grep -E 'nvenc|qsv|vaapi'

# ของจริงต้องมีอุปกรณ์ให้เห็นใน container
ls -l /dev/dri
ls: cannot access '/dev/dri': No such file or directory

The fix: disable video transcoding and stop enabling subtitles by default

The conclusion was that this machine had to work only at the top two levels. I disabled video transcoding for every account, leaving only audio transcoding and remux enabled. I had not recorded the field names in my original notes; I took the names below from Jellyfin's public API documentation. Before using them, check them against your installed version, or use the checkboxes on each account's Dashboard page for the same result.

import json, os, urllib.request

BASE = "http://192.168.10.20:8096"
HDR = {
    "Authorization": 'MediaBrowser Token="%s"' % os.environ["JF_API_KEY"],
    "Content-Type": "application/json",
}

def call(method, path, body=None):
    data = json.dumps(body).encode() if body is not None else None
    req = urllib.request.Request(BASE + path, data=data, headers=HDR, method=method)
    with urllib.request.urlopen(req, timeout=15) as r:
        raw = r.read()
    return json.loads(raw) if raw else None

for u in call("GET", "/Users"):
    pol = u["Policy"]
    pol["EnableVideoPlaybackTranscoding"] = False   # ห้ามเข้ารหัสภาพใหม่
    pol["EnableAudioPlaybackTranscoding"] = True    # แปลงเสียงได้ งานเบา
    pol["EnablePlaybackRemuxing"] = True            # ห่อ container ใหม่ได้
    call("POST", "/Users/%s/Policy" % u["Id"], pol)

for n, u in enumerate(call("GET", "/Users"), 1):
    p = u["Policy"]
    print("user %d  direct play %s / remux %s / transcoding %s" % (
        n, p["EnableMediaPlayback"], p["EnablePlaybackRemuxing"],
        p["EnableVideoPlaybackTranscoding"]))

The required check results for each account were direct play True, remux True, and transcoding False. I then changed three more default playback settings for every account.

This configuration had a side effect I had not anticipated. Some videos have sections of dialogue in another language that rely on a Forced subtitle track. With subtitles off by default and subtitle burn-in prohibited, those subtitles silently disappeared on players that could not render image subtitles themselves. There was no error and no warning; viewers simply could not understand what the characters were saying.

The solution was a one-time conversion from image subtitles to text subtitles. Almost every player can render text subtitles itself, so the server does not need to touch the video. I extracted the PGS track, passed it through a subtitle OCR program (I had not recorded which one in my original notes, so I will not name one; use whichever you are comfortable with), and saved the result as an external subtitle file beside the video. Not a single byte of the video file changed. I ended up with two tracks: Forced subtitles with 230 lines and full subtitles with 540 lines.

# ดูว่าแทร็กซับไหนเป็น PGS และแทร็กไหนติดธง forced
ffprobe -v error -select_streams s \
  -show_entries stream=index,codec_name:stream_tags=language:stream_disposition=forced \
  -of csv=p=0 /media/video/sample.mkv

# ดึงแทร็กซับลำดับที่ 2 ออกมาเป็น .sup โดยไม่เข้ารหัสใหม่
ffmpeg -i /media/video/sample.mkv -map 0:s:1 -c copy sample.th.forced.sup

# หลัง OCR แล้ว วางชื่อไฟล์ให้เซิร์ฟเวอร์จับคู่ได้เอง
#   sample.th.forced.srt   sample.th.srt

Fixing floating vowel marks from OCR

As soon as I put the subtitles in place, someone at home reported that “all the vowel marks are floating.” Opening the file showed that OCR had inserted spaces before above-base vowel marks, below-base vowel marks, and tone marks. Every character was correct, but the vowel marks floated over spaces instead of attaching to consonants. I counted 876 occurrences in the Forced file and 2,229 in the full file, plus 1 incorrectly composed sara am vowel that I fixed separately.

These Thai characters can never follow a space, so the rule was straightforward: remove spaces before characters in U+0E31, U+0E34 through U+0E3A, and U+0E47 through U+0E4E. I previously wrote about OCR vowel errors in a data import context in my article on migrating Thai data. This case involved subtitles, with the added constraint that line counts and timestamps must not change.

import re, shutil, sys

# สระบน สระล่าง วรรณยุกต์ ที่ต้องเกาะพยัญชนะตัวหน้าเสมอ
MARKS = "\u0e31\u0e34-\u0e3a\u0e47-\u0e4e"
GAP = re.compile("[ \u00a0]+(?=[" + MARKS + "])")
STAMP = re.compile(r"\d\d:\d\d:\d\d,\d{3}")

def fix(path, apply=False):
    with open(path, encoding="utf-8-sig", newline="") as f:
        text = f.read()
    n = len(GAP.findall(text))
    print("%s: ช่องว่างหน้าสระ/วรรณยุกต์ %d จุด" % (path, n))
    if not apply or n == 0:
        return n
    fixed = GAP.sub("", text)
    # ซับต้องยังตรงเวลาเดิม: จำนวนบรรทัดและ timestamp ห้ามเปลี่ยน
    assert fixed.count("\n") == text.count("\n")
    assert STAMP.findall(fixed) == STAMP.findall(text)
    shutil.copy2(path, path + ".orig")
    with open(path, "w", encoding="utf-8", newline="") as f:
        f.write(fixed)
    return len(GAP.findall(fixed))

if __name__ == "__main__":
    do_apply = "--apply" in sys.argv
    for p in sys.argv[1:]:
        if p != "--apply":
            print("  เหลือ", fix(p, do_apply))

I ran it without --apply first to check the counts, then ran it with the flag to apply the changes. Both files ended up with 0 occurrences remaining, with unchanged line counts and timestamps, and the originals saved as .srt.orig in case I needed to revert.

The troubleshooting sequence I use now

1. If playback fails to start or freezes, open the server logs first. Check whether the session uses direct play, remux, or transcode, and why.
2. If it is transcoding, check ffmpeg's fps and speed. Below 1.00x, smooth playback is impossible; there is no need to investigate the network.
3. Check the cgroup's memory.events. If high reaches tens of thousands while oom_kill is 0, it is being throttled, not killed.
4. Check for actual hardware acceleration through /dev/dri, not ffmpeg's encoder list.
5. If there is no hardware acceleration, disable video transcoding for every account, set subtitles to off by default, and check the logs again to confirm that the actual session is not transcoding.
6. For image subtitle tracks that are needed, use OCR to convert them to SRT, save them as external files, and check for spaces before vowel marks before handing them to viewers.

Another option that requires no OCR is to choose a player that can render image subtitles itself. Such a player can immediately direct play the original file with PGS subtitles. If your device supports installing one, that is an option. I chose OCR because the household uses several kinds of devices, and text subtitles work on all of them.

The last question someone asked was whether opening the file through SMB would be faster. The answer is that it makes no difference: direct play already retrieves the original file directly. Transcoding is what makes it slow, not the protocol used to transfer the file. All these figures come from my one machine; a machine with working hardware acceleration would give a different answer. If I have left anything out or a field name changes in a newer version, I will come back and update the article.