Ingesting Mail Without Disrupting Someone Else's Mailbox

I had a task where an app needed to fetch mail from the team's shared mailbox and turn it into work items in the system. It sounded small: connect over IMAP, fetch new mail, save it to the database, done. But once I started designing it, the first constraint turned out to be harder than I expected: people actually use that mailbox every day, reading mail on their phones and in webmail, organizing folders themselves, and deleting junk themselves. That meant my program must not change even a single status in that mailbox, and it must not miss any mail either. So I wrote up both the traps I nearly fell into and the approach I use now.

The domain names, account names, and work context in this article are all fictional. The logic and table structures are what I actually use.

The first trap: don't use UNSEEN to identify what's new

Almost every example I found on the internet teaches the same approach: search for unread mail and fetch it for processing.

// วิธีที่เจอบ่อยที่สุด และเป็นวิธีที่พังกับกล่องที่มีคนใช้จริง
$messages = $folder->query()->unseen()->get();

Worse still, if the program fetches mail and marks it as read to prevent fetching it again, I'm also disrupting the mailbox owner's workflow. They use unread status as their own to-do list. Suddenly, it all disappears every morning because my program got there first. I've seen this happen in another system, and I heard plenty of complaints about it.

The right cursor is UID, but it must come with UIDVALIDITY

IMAP has a message identifier within a folder called UID, which keeps increasing and is never reused. I store the latest number that was processed successfully, then search only for numbers greater than that on the next run.

// ค้นด้วยช่วง UID ไม่ยุ่งกับสถานะอ่าน/ยังไม่อ่านเลย
$from = ((int) $account->last_uid) + 1;
$messages = $client->getFolder($account->folder)
    ->query()
    ->setFetchBody(false)
    ->getByUidSequence("{$from}:*", limit: 25);   // ทีละ 25 ฉบับ กัน memory บวม

But UID is reliable only as long as the folder's UIDVALIDITY stays the same. This value is the folder's version number. If the server recreates the folder, moves mail, or restores from a backup, this value changes, and all the UIDs I've stored will point to different messages than I expect.

// ต้องตรวจทุกรอบ ไม่ใช่ตรวจครั้งเดียวตอนตั้งค่า
$uidvalidity = $folder->getStatus()['uidvalidity'] ?? null;

if ($account->uidvalidity !== null && (int) $account->uidvalidity !== (int) $uidvalidity) {
    Log::warning('uidvalidity changed — reset last_uid', ['account' => $account->id]);
    $account->last_uid    = 0;          // เริ่มนับใหม่ทั้งโฟลเดอร์
    $account->uidvalidity = $uidvalidity;
    $account->save();
}

BODY.PEEK reads without leaving a trace

A regular FETCH command requesting BODY automatically sets the Seen flag. The one to use is BODY.PEEK, which works exactly the same way except that it leaves the flag alone. A good library provides a setting for this, but I have to find it because many libraries don't use it by default.

// เปิดโหมด peek ให้ชัด และปิดทุกทางที่จะเขียนกลับ mailbox
$client = Client::make([
    'host'          => 'mail.example.com',
    'port'          => 993,
    'encryption'    => 'ssl',
    'validate_cert' => true,
    'username'      => env('MAIL_INGEST_USER'),
    'password'      => env('MAIL_INGEST_PASS'),
    'protocol'      => 'imap',
    'options'       => [
        'fetch'          => \Webklex\PHPIMAP\IMAP::FT_PEEK,  // ไม่ตั้งธง Seen
        'soft_fail'      => true,
        'fetch_flags'    => true,
        'message_key'    => 'uid',
    ],
]);

To enforce that my code can never write back, I don't just read through it and decide it looks correct. I make it part of the review process: grep for every command that changes state. The result must be empty.

# ด่านตรวจก่อน deploy: ผลต้องไม่มีสักบรรทัด
grep -rn --include='*.php' \
  -e 'setFlag' -e 'addFlag' -e '\\Seen' -e '->delete(' -e 'expunge' -e 'moveTo' \
  app/Services/MailIngest app/Console/Commands | grep -v 'test'

The IMAP extension is no longer in the core distribution

This is what forced me to change the original plan. PHP's imap extension was removed from the core distribution starting with 8.4, becoming something I have to install separately from PECL, and installation isn't easy on newer distros either. My choice was a library that implements IMAP entirely in PHP, with no extension dependency at all.

# ตรวจก่อนว่าเครื่องเป้าหมายมีอะไรบ้าง จะได้ไม่ออกแบบบนสมมติฐานที่ผิด
php -m | grep -i -E '^(imap|openssl|mbstring|iconv)$'
php -r 'echo function_exists("imap_open") ? "ext-imap ok\n" : "ext-imap ไม่มี\n";'

Deduplication must happen on our side, and Message-ID alone isn't enough

The Message-ID header should be present in every message according to the standard, but in practice, some messages really don't have one. Most come from systems with simple homegrown mail senders, and they often happen to be exactly the kind of mail I want to ingest.

// dedupe key: ใช้ Message-ID ถ้ามี ไม่มีก็สร้างจากสิ่งที่คงที่พอ
$msgId = trim((string) $message->getMessageId());

$dedupeKey = $msgId !== ''
    ? mb_substr($msgId, 0, 190)
    : hash('sha256', implode('|', [
        (string) $message->getFrom()[0]->mail,
        $message->getDate()->format('c'),
        (string) $message->getSubject(),
      ]));

There's another database trap here: if I allow this column to be NULL when there's no Message-ID and rely on a UNIQUE index to prevent duplicates, it won't.

-- UNIQUE ปล่อย NULL ซ้ำได้กี่แถวก็ได้ — กับดักคลาสสิก
INSERT INTO mail_messages (mail_account_id, dedupe_key) VALUES (1, NULL);
INSERT INTO mail_messages (mail_account_id, dedupe_key) VALUES (1, NULL);
INSERT INTO mail_messages (mail_account_id, dedupe_key) VALUES (1, NULL);
-- ผ่านทั้งสามแถว ไม่มี error สักตัว

The fix is to make this column NOT NULL and always fill in a fallback value I calculate myself. There must be no case where it's left empty.

CREATE TABLE mail_messages (
  id              BIGINT AUTO_INCREMENT PRIMARY KEY,
  mail_account_id BIGINT       NOT NULL,
  dedupe_key      VARCHAR(190) NOT NULL,              -- ห้าม NULL เด็ดขาด
  imap_uid        BIGINT       NOT NULL,
  uidvalidity     BIGINT       NOT NULL,
  from_addr       VARCHAR(255) NOT NULL,
  subject         VARCHAR(255) NULL,
  sent_at         DATETIME     NULL,
  status          VARCHAR(16)  NOT NULL DEFAULT 'new',
  created_at      DATETIME     NOT NULL,
  UNIQUE KEY mm_acct_dedupe (mail_account_id, dedupe_key),
  KEY mm_acct_uid (mail_account_id, imap_uid)
) ENGINE=InnoDB;

Storing imap_uid and uidvalidity alongside it isn't for deduplication. It's for investigating which run a message came from. When UIDVALIDITY changes and the whole folder is fetched again, this pair helps me trace what was fetched again.

What to watch when saving: the order of last_uid updates

// อัปเดต last_uid หลังจากที่แถวถูกบันทึกสำเร็จแล้วเท่านั้น
DB::transaction(function () use ($batch, $account) {
    $maxUid = (int) $account->last_uid;
    foreach ($batch as $m) {
        DB::table('mail_messages')->insertOrIgnore($this->toRow($m, $account));
        $maxUid = max($maxUid, (int) $m->getUid());
    }
    DB::table('mail_accounts')->where('id', $account->id)->update([
        'last_uid'        => $maxUid,
        'last_fetched_at' => now(),
    ]);
});

Attachments are the most dangerous part of the entire system

Incoming mail can carry any attachment, and the sender doesn't have to be someone I trust. I use four rules, and all four must be met.

const ALLOW_EXT  = ['pdf', 'jpg', 'jpeg', 'png', 'docx', 'xlsx'];
const ALLOW_MIME = ['application/pdf', 'image/jpeg', 'image/png',
                    'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
                    'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'];

public function accept(string $name, string $mime, int $bytes): bool
{
    $parts = explode('.', strtolower($name));
    array_shift($parts);                          // ทิ้งชื่อฐาน เหลือแต่นามสกุลทุกชั้น
    if (count($parts) > 1) return false;          // นามสกุลซ้อน เช่น .pdf.exe ตัดทิ้งทันที
    if (!in_array($parts[0] ?? '', self::ALLOW_EXT, true)) return false;
    if (!in_array($mime, self::ALLOW_MIME, true)) return false;
    return $bytes <= 10 * 1024 * 1024;
}

The most important rule is where files are stored. They must be outside any directory the web server can serve, and must always go through a controller that checks permissions before serving them. Files on disk get random names, not the names supplied in the email.

// เก็บนอก webroot ชื่อสุ่ม เก็บ sha256 ไว้เทียบ ส่วนชื่อเดิมเก็บไว้แค่ในฐานเพื่อแสดงผล
$disk = Storage::build(['driver' => 'local', 'root' => '/srv/app/var/mail-att']);
$path = $account->id . '/' . now()->format('Y/m') . '/' . Str::random(32);
$disk->put($path, $attachment->getContent(), ['visibility' => 'private']);

Tests that must pass before going live

I wrote this set as a checklist and actually run it against a test mailbox every time I change this part of the code, not just once when the implementation is finished.

1. Send two test messages to the mailbox, one with an attachment.
2. Read one message in webmail first, then run fetch — both messages must be retrieved. This is where UNSEEN would fail.
3. Go back and check the read or unread status of both messages in webmail. Neither must have changed at all.
4. Run fetch again immediately. It must produce zero new rows.
5. Change uidvalidity in the table to a value different from the actual one, then fetch again. It must fetch the whole folder and still produce zero new rows.
6. Send a message without Message-ID. It must be saved and deduplicated using the fallback key.
7. Send attachments with double extensions and files over the size limit. They must be rejected, with the reason for rejection recorded.
8. grep for state-changing commands throughout the module's code. The result must be empty.

One last point about Thai characters. Some older systems still send Thai email subjects using legacy character encodings. Saving them directly to the database produces garbled characters. Always convert them first, and remember that the subject and body can use different character encodings within the same message.

// แปลงชุดอักขระของหัวข้อก่อนเก็บ และตัดความยาวด้วยฟังก์ชันที่รู้จักหลายไบต์
$subject = $message->getSubject()->toString();
if (!mb_check_encoding($subject, 'UTF-8')) {
    $subject = mb_convert_encoding($subject, 'UTF-8', 'TIS-620,ISO-8859-11,UTF-8');
}
$subject = mb_substr($subject, 0, 250);

# cron: ให้ scheduler เป็นคนคุมจังหวะ อย่าเขียนหลายบรรทัดใน crontab
* * * * * appuser cd /srv/app && php artisan schedule:run >> /dev/null 2>&1

A brief recap for anyone about to build this: there are three main points. First, never use a signal controlled by a human as program state. Second, a position cursor must always be paired with the folder's version indicator. Third, deduplication must be robust enough to handle refetching the whole folder without trouble. As for attachments, think of it as accepting files from strangers, because that's exactly what it is. If you've built a system like this and run into traps I haven't encountered yet, I'd like to hear about them. If I've left gaps in the explanation or made typos, I'll come back and fix them later.