News Stuck for Three Days Because the Feed Wasn't Sorted by Date

I look after a niche news site that tracks a little over twenty brands in the same industry and puts their stories on the homepage every morning. One day, someone asked me, "Don't you post news every day?" I opened the site and found that the featured stories had been stuck on the same date for three days. What confused me was that when I checked the logs for every step, the scheduled jobs had run every day, the translation queue had finished, and the publisher had printed RESULT OK every time, all with exit 0. There wasn't a single error line anywhere in the pipeline. It turned out to be three overlapping problems, so I wrote them up here.

The site names, brand names, search terms, and all keys in this article are examples I made up to replace the real ones. The structure, measured numbers, and code are from the actual work.

What dragged this out was that every layer in that pipeline really was doing exactly what it had been told to do. The aggregator ran every query, got results, and wrote the file successfully. The translation step read the file, found no new items, and wrote it back unchanged. The publisher read the same items, successfully sent them to the API, and returned exit 0.

Put simply, every layer checked whether it had finished its work; none checked whether the output had changed since yesterday. I've run into this pattern several times in automated systems, and every time it has been just as silent.

Trap one: RSS sorted by relevance, not date

I wrote the first version of the aggregator in Node. It fetched RSS from a news aggregation service and took the first 5 items, which seemed perfectly reasonable if I assumed the feed put the newest items first.

// ก่อนแก้ — ตัดจากบนสุดตรงๆ โดยเดาว่าฟีดเรียงตามวันที่ให้แล้ว
function parseItems(xml, limit) {
  return [...xml.matchAll(/<item>([\s\S]*?)<\/item>/g)]
    .slice(0, limit)                       // <-- ต้นเหตุอยู่บรรทัดนี้
    .map(m => readOne(m[1]));
}

The problem was that this feed sorted by relevance, not date. One day, I printed out the actual order and saw 2 Sep., then 6 Sep., then 3 Sep.—the dates jumped back and forth. Today's stories really were in that same feed, but they were farther down the list. Taking only the first five gave me the same set every day, and the next step, which deduplicated by url, skipped all of them because every link it received was one it had already seen. So the site's feed never moved.

The fix was to separate those concerns clearly: read the whole feed first, then sort it myself, then take the slice.

// หลังแก้ — อ่านทุก item, เรียงตาม pubDate ที่แปลงเป็นเวลาจริง, แล้วค่อยตัด
function parseItems(xml, limit, must) {
  const items = [...xml.matchAll(/<item>([\s\S]*?)<\/item>/g)].map(m => {
    const b = m[1];
    const get = re => { const x = b.match(re); return x ? decode(x[1]) : ''; };
    const rawTitle = get(/<title>([\s\S]*?)<\/title>/);
    const source   = get(/<source[^>]*>([\s\S]*?)<\/source>/);
    // พาดหัวของบริการนี้ลงท้ายด้วย " - <สำนักข่าว>" เสมอ ตัดทิ้งให้เหลือหัวข้อสะอาด
    const title = source && rawTitle.endsWith(' - ' + source)
      ? rawTitle.slice(0, -(source.length + 3)) : rawTitle;
    return { title, source, link: get(/<link>([\s\S]*?)<\/link>/),
             pubDate: get(/<pubDate>([\s\S]*?)<\/pubDate>/) };
  });
  // แปลงเป็น epoch ก่อนเทียบ ถ้าแปลงไม่ได้ให้เป็น 0 (ตกไปท้ายแถว ไม่ใช่ขึ้นหัวแถว)
  const ts = it => { const t = Date.parse(it.pubDate); return Number.isFinite(t) ? t : 0; };
  return items.filter(hit(must)).sort((a, b) => ts(b) - ts(a)).slice(0, limit);
}

I also adjusted the time window in the query. It had been looking back 45 days, which, combined with relevance sorting, was a recipe for getting the same old stories repeatedly. I reduced it to 10 days, and the pool of candidates became noticeably fresher.

Trap two: Sorting dates as strings means sorting by weekday name

The next component in the pipeline was written in Python, and it also sorted dates, but it sorted the raw RFC-822 pubDate values directly.

# รูปแบบที่ฟีดส่งมา — ขึ้นต้นด้วยชื่อวันในสัปดาห์เสมอ
"Tue, 08 Sep 2026 10:00:00 GMT"
"Fri, 04 Sep 2026 07:30:00 GMT"
"Mon, 07 Sep 2026 22:15:00 GMT"

# เรียงสตริงตรงๆ ได้ลำดับนี้ — เรียงตามตัวอักษรของ "ชื่อวัน" ไม่ใช่วันที่
# Fri < Mon < Sat < Sun < Thu < Tue < Wed
sorted(dates)   # -> Fri 04 ก.ย. มาก่อน Mon 07 ก.ย. มาก่อน Tue 08 ก.ย.

That was why the newest stories could keep losing to week-old ones without any visible error. So I wrote a function to extract a numeric value to sort by instead.

MON = {v: i for i, v in enumerate(
    ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"], 1)}

def pubkey(it):
    """คีย์เรียงลำดับจาก pubDate แบบ RFC-822 ("Tue, 14 Jul 2026 10:00:00 GMT")
    การเรียงสตริงดิบคือการเรียงตามชื่อวันในสัปดาห์ (Fri < Mon < Sat < Sun < Thu < Tue < Wed)
    ซึ่งคือสาเหตุที่ข่าวใหม่สุดแพ้ข่าวเก่ามาตลอด"""
    m = re.search(r"(\d{1,2})\s+(\w{3})\s+(\d{4})(?:\s+(\d{2}):(\d{2}))?",
                  it.get("pubDate", "") or "")
    if not m:
        return ""                      # ไม่มีวันที่ -> คีย์ว่าง -> ตกไปท้ายแถว
    return "%s-%02d-%02d %s:%s" % (m.group(3), MON.get(m.group(2), 1), int(m.group(1)),
                                   m.group(4) or "00", m.group(5) or "00")

Trap three: Request too few, and the filter rejects everything before the rest gets a look

When I switched to another news provider's API, I thought that would be the end of it, because this provider really did put the newest stories first. But the news still got stuck. This time, the cause was different.

The code requested 4 stories per brand and passed them through a relevance filter requiring the brand name to appear in either the headline or the news outlet's name. It turned out that the first four items for each brand were regularly rejected. The items that could actually pass the filter were sitting at positions 5 through 10, cut off before they were even considered.

I measured this on the same day, making the same number of requests in both runs.

The point I want to highlight is that these services count quota by requests, not by the number of items returned. So requesting fewer than the allowed maximum doesn't save a single baht; it just loses items that would have passed the filter. I set the default to the provider's maximum and added a comment explicitly saying not to lower it again.

# ขอมาให้เต็มเพดานที่ผู้ให้บริการยอมให้ต่อหนึ่งคำค้น
# ห้ามลดค่านี้ลง: โควตาคิดเป็น "จำนวนครั้งที่ยิง" ไม่ใช่ "จำนวนข่าวที่ได้"
# ลดลงเมื่อไรของที่ผ่านด่านกรอง (ซึ่งมักอยู่ลำดับ 5-10) จะถูกตัดทิ้งตั้งแต่ยังไม่ได้ดู
PER = int(sys.argv[sys.argv.index("--per") + 1]) if "--per" in sys.argv else 10

for key, q, must in BRANDS:
    url = ("https://api.example.com/v4/search?q=%s&lang=en&max=%d&sortby=publishedAt&apikey=%s"
           % (urllib.parse.quote(q), PER, KEY))
    d, err = call(url)
    used += 1
    if err:
        print("  %-20s ERR %s" % (key, err))
        if "429" in err or "403" in err:
            quota_hit = True; break          # ชนโควตาแล้ว หยุดเลย อย่ายิงต่อให้โดนแบน
        time.sleep(1.1); continue
    ...
    time.sleep(1.1)                          # เพดาน 1 ครั้งต่อวินาที เว้นจังหวะเอง

The filter: Necessary, but choose the terms carefully

When I broadened the queries to find more stories, car reviews, provincial agricultural news, and press releases from completely unrelated companies immediately appeared on the niche news site that same day. Thai search matches very loosely, so a query combining a brand name with a common industry term can match almost anything.

My filter therefore required that the headline or news outlet's name actually contain the brand name, and there were two more traps inside this filter.

def norm(s):
    # พาดหัวไทยเว้นวรรคชื่อแบรนด์ไม่เหมือนกัน ("สตาร์ไลน์" กับ "สตาร์ ไลน์")
    # ถ้าไม่ตัดช่องว่างก่อนเทียบ ข่าวจริงของแบรนด์จะโดนตีตกทิ้งว่าไม่เกี่ยว
    return "".join((s or "").lower().split())


def relevant(it, must):
    if not must:
        return True                       # ไม่ได้ตั้งคำไว้ -> เก็บ อย่าเงียบๆ ทิ้งประวัติ
    hay = norm((it.get("title") or "") + " " + (it.get("source") or ""))
    return any(norm(t) in hay for t in must)

The first was whitespace. When I was still comparing without removing whitespace, real stories about one brand were regularly rejected because their headlines inserted a space in the middle of its name. Once I removed whitespace before comparing, the number collected went from 30 to 35 in a single run.

The second was the choice of terms. Never use country names, city names, or common words as filter terms. I once used a country name and ended up with lots of stories mourning a member of that country's royal family, along with stories about oil prices in another country. Another brand's name coincided with the name of a type of fabric, and three-letter abbreviations seemed to match everything on earth. Use the full brand name, not a word within the brand name.

The file keeps growing, so it needs a cleanup pass over old items

An easy point to overlook is that my output file keeps accumulating items and never shrinks. The code starts with items = list(prev) and appends new items, and the publisher sends the entire list to replace the whole set. That means anything that got through before the filter existed would stay on the site forever, because this news pipeline has no step involving human judgment.

So I wrote a separate cleanup tool that uses the same filter-term table to scan the items already stored. It defaults to a preview-only mode and only overwrites the file when explicitly given --apply.

def main():
    j = json.load(io.open(ENP, encoding="utf-8"))
    items = j.get("items", [])
    keep = [i for i in items if relevant(i, MUST.get(i.get("key", "")))]
    drop = [i for i in items if i not in keep]
    print("ไฟล์สะสม: %d รายการ | เก็บ %d | ทิ้ง %d" % (len(items), len(keep), len(drop)))
    for d in drop[:20]:
        print("  DROP [%s] %s | %s" % (d.get("key"), d.get("date"), (d.get("title") or "")[:64]))
    if "--apply" not in sys.argv:
        print("\n(โหมดดูเฉยๆ — ใส่ --apply ถึงจะเขียน แล้วค่อยสั่งเผยแพร่ซ้ำ)")
        return
    j["items"] = keep
    json.dump(j, io.open(ENP, "w", encoding="utf-8"), ensure_ascii=False, indent=1)
    print("\nเขียนแล้ว %d รายการ ต่อไปสั่งเผยแพร่เฉพาะส่วนข่าวเพื่อ sync ของจริง" % len(keep))

That time, I used it to clean up items that had slipped in before the filter existed: 11 unrelated stories, plus another 3 that had matched incorrectly because a filter term was a country name.

The freshness check: Make the job fail loudly instead of silently

The three fixes above addressed the causes, but this one mattered more to me. I'd run into "silently stuck" output several times, and each time I had to wait for someone to point it out. So I added a final check to the publisher to examine the date of the newest story after it finished its work. If that story was older than the configured threshold, it would print the reason and return a nonzero value.

# เกินกี่วันถือว่าฟีดค้าง — ปรับผ่านตัวแปรสภาพแวดล้อมได้โดยไม่ต้องแก้โค้ด
STALE_DAYS = int(os.environ.get("FEED_STALE_DAYS", "2"))


def freshness_gate(items):
    """ตรวจ 'ผลลัพธ์' ไม่ใช่ 'ว่ารันจบ' — คืนค่าที่ไม่ใช่ศูนย์เพื่อให้ตัวตั้งเวลาขึ้นแดง"""
    if not items:
        print("[gate] ไม่มีรายการที่เผยแพร่ได้เลยในหน้าต่างเวลา -> ถือว่าล้ม")
        return 1
    newest = max((i.get("date") or "") for i in items)
    cut = (datetime.date.today() - datetime.timedelta(days=STALE_DAYS)).isoformat()
    if newest < cut:
        print("[gate] ข่าวใหม่สุดคือ %s เก่ากว่าเส้น %s (FEED_STALE_DAYS=%d)"
              % (newest, cut, STALE_DAYS))
        print("[gate] แปลว่าต้นทางหยุดส่งของใหม่ หรือด่านกรองตีตกหมด -> ไปไล่ที่ต้นทาง")
        return 1
    print("[gate] ข่าวใหม่สุด %s ผ่าน" % newest)
    return 0


sys.exit(freshness_gate(out) or report("feed"))

I chose the default of 2 days based on the actual rhythm of the job. It runs every day, so if two days pass without anything new coming in, someone needs to take a look. If your source publishes more frequently, you can lower it as needed, but don't make it so lenient that the check never triggers.

Add a second source, and the same problem comes right back

After all that, there were still six brands for which I couldn't find any English-language news. So I added a Thai-language RSS feed as a second source. It was free, needed no key, had no quota, and returned up to a hundred items per query. I inserted it between the aggregator and the publisher, adding items to the same file in the same format, so nothing downstream needed changing.

# ต่อเข้าลำดับงานประจำวัน คั่นระหว่างตัวรวมข่าวกับตัวเผยแพร่
python feed_api.py --per 10
python feed_rss.py --all --days 10 --per 6
python feed_publish.py --days 21 --max 24

But I need to stress that this second source was the same source with the first trap: it sorted by relevance. I checked the actual results again and confirmed the order was 2 Sep., then 6 Sep., then 3 Sep. So this new component also had to parse pubDate into a real datetime and sort the items itself before taking a slice, or it would get the same old stories every day.

On its first day, the second source added 19 stories, which combined with the existing 5 to make a feed of 24. The site clearly showed stories from today and yesterday. The downside was that RSS had no story images, so the publisher had to fall back to a default image for each brand. The first run got actual images for 26 out of 40, with the other 14 using fallback images, which I could accept.

The order I follow when investigating these symptoms

1. Compare the output with yesterday's before looking at logs. If the item count is exactly the same, that means nothing new entered the system, not that the system broke.
2. Work backward from the end toward the source, asking at each layer, "How many new items did this layer receive?" rather than "Did this layer run successfully?"
3. Once you find the layer that received zero, print all the raw data it received and inspect it visually, especially the order and dates.
4. Distinguish between "the source has nothing," "the filter rejected it," and "it was cut off before reaching the filter." These are different symptoms with different fixes.
5. After fixing it, always add an output check at the very end so that next time it turns red on its own, without waiting for someone to point it out.

Summary of the changes

There wasn't a single bug here that looked like a bug. Every piece of code did exactly what it was written to do, but the assumptions underneath were wrong: "the feed is already sorted by date," "date strings can be sorted," and "requesting fewer items saves money." All three sounded reasonable when I wrote the code, and none would make a sound when it was wrong.

Personally, if I could choose just one thing from all of this, I would choose the freshness check from the earlier section. The other three fixes address problems I'd already found, but that check will tell me when I've hit a new problem I don't yet know about. As for the rest, prioritize them however works for you.

P.S. If your source really does publish irregularly—for example, some weeks it goes quiet for the entire week—don't make the freshness check fail the job yet. Start with alerts only for two or three weeks, learn the source's actual rhythm, and then set the threshold. Otherwise, you'll get so many false alerts that people stop reading them. I'll come back and add more on that later.