EST. 2026 — Notes from real server work SYSADMIN · SECURITY · AI · DIRECTADMIN Issue 01
THE SYSADMIN ALMANAC
Home / Tags / Troubleshooting
Troubleshooting

#Troubleshooting

91 articles
Workflow
Passing the Sandbox Doesn't Mean It Works — 7 False Signals
Linux

Passing the Sandbox Doesn't Mean It Works — 7 False Signals

The automated ordering system I wrote passed every command in a provider's sandbox, but almost everything the sandbox told me differed from reality, from response times and charges to accepted names and items that were created successfully but could not be found. Here are 7 false signals and how to write code that does not fall for them.

#api#rest-api#testing#debugging
9 min read·26 Oct
Labels Printing Off-Center? Calibrate the Print Offset Once
Hardware

Labels Printing Off-Center? Calibrate the Print Offset Once

After getting the label printer to fire its printhead, the next problem was that the text missed the sticker. I stopped guessing and printed rulers and corner markers to measure it, then clearly separated paper size, printable area, and offset into three distinct values.

#printer#kotlin#android#hardware
9 min read·25 Oct
Silent Overwrites on Upload: Three File Copying Traps
Linux

Silent Overwrites on Upload: Three File Copying Traps

Upload two files with the same name together, and the second silently overwrites the first. No error, no failing exit code, and every page still returns 200. The English page just becomes a Thai page. These are three traps that let file copying cause damage without any warning.

#deployment#pscp#plink#putty
8 min read·24 Oct
Let Clients Pick Photos on a Web Page Without a Login
Web

Let Clients Pick Photos on a Web Page Without a Login

When the person choosing photos for a website is not comfortable with computers, adding registration and login means the work stops at the login screen. This article covers a photo picker with no authentication at all, but with a server endpoint that checks every value against an allowlist strict enough to leave nothing from arbitrary input, plus limits at every layer and a backup before every write.

#php#javascript#web-security#permissions
8 min read·23 Oct
Build APM on Windows, Part 7: Deploy, Run, and Apply Security Patches
Web

Build APM on Windows, Part 7: Deploy, Run, and Apply Security Patches

The final part of the series on building Apache + PHP + MariaDB on Windows. Install into C:\APM, start and test it for the first time, open it to the LAN safely, walk through six real bugs (PidFile, MariaDB plugins, the DB port, the ASLR 500, paths with spaces and brackets), and cover the procedure for the next security release.

#devops#windows#apache#php
20 min read·22 Oct
Build APM on Windows Part 6: PHP 8.4 with redis and imagick
Web

Build APM on Windows Part 6: PHP 8.4 with redis and imagick

Part 6 of the series on building Apache + PHP + MariaDB on Windows from source walks through 30-php.ps1 step by step, from php-sdk and the full configure command to copying DLLs and testing that JIT, redis and imagick actually work.

#php#windows#powershell#tutorial
14 min read·22 Oct
Build APM on Windows, Part 5: Compile ImageMagick 7 for PHP imagick
Web

Build APM on Windows, Part 5: Compile ImageMagick 7 for PHP imagick

Part 5 of the Build APM on Windows series compiles ImageMagick 7.1.2-34 and around 40 delegate libraries from source with the official toolchain, including the channel mask patch from 64 to 32 that lets imagick build, and why HEIC cannot be written

#windows#php#devops#powershell
14 min read·22 Oct
Build APM on Windows, Part 4: Compile MariaDB 11.8 LTS
Web

Build APM on Windows, Part 4: Compile MariaDB 11.8 LTS

Part 4 of the Build APM on Windows series walks through compiling MariaDB 11.8.9 from source with CMake and Ninja, from preparing the tools and running the script to testing on port 3399 and explaining why collation must be pinned to utf8mb4_unicode_ci

#mariadb#windows#devops#powershell
14 min read·22 Oct
Build APM on Windows, Part 3: Apache httpd 2.4.69 with CMake
Web

Build APM on Windows, Part 3: Apache httpd 2.4.69 with CMake

Build APR, APR-util, httpd 2.4.69 and mod_fcgid with CMake and Ninja on Windows. Explain why each option is set, use dumpbin to find the DLLs needed beside httpd.exe, prevent links to debug DLLs, and test the config with httpd -t.

#apache#windows#web-server#cpp
14 min read·22 Oct
Scripting Government PDF Forms and Making Safer ID Copies
AI

Scripting Government PDF Forms and Making Safer ID Copies

Government PDFs distributed as fillable forms often cannot actually handle Thai text. Subdivided fields let digits run over the borders, and tone marks collide with vowels above the line. This article covers listing field names first, filling the fields and flattening them into page content, drawing Thai text with a font that includes all the vowel marks, and making actual-size ID copies with purpose-specific watermarks.

#pdf#python#document#printing
10 min read·22 Oct
Building a Graph API Page App with Three Required Endpoints
Web

Building a Graph API Page App with Three Required Endpoints

When a backend needs to read and reply to Page messages on its own, the only proper route is to build an app and pass the platform's review. This article covers three endpoints the platform requires but people forget until they fail review, the correct way to verify incoming request signatures, and the steps to exchange a short-lived token for a Page token stored outside the web root.

#api#rest-api#php#web-security
10 min read·22 Oct
VM Sees the Dongle but Has No Internet Without a Windows ECM Driver
Hardware

VM Sees the Dongle but Has No Internet Without a Windows ECM Driver

After passing a USB LTE dongle through to a Windows VM, the system sees the device but no network adapter appears. The cause is neither the SIM nor port passthrough, but the dongle exposing a CDC ECM interface that Windows has no built-in driver for, along with storage-mode pitfalls and dhclient overwriting the host's resolv.conf.

#proxmox#hardware#windows#linux
9 min read·21 Oct
A Report Form for a Static Site with No Backend
Web

A Report Form for a Static Site with No Backend

A calculator site made entirely of static files has no server to handle forms, but users still need a way to report mistakes in the formulas. This article covers the one approach I chose—a single worker file that accepts POST requests on one route, verifies Turnstile on the server using the real client IP, accepts images pasted from the clipboard, and sends email from the network edge—along with three traps that cost me an entire night.

#cloudflare#static-site#javascript#web-security
9 min read·21 Oct
Reverse Engineering Spreadsheet Constants into Formulas
AI

Reverse Engineering Spreadsheet Constants into Formulas

A sheet has a criteria table sitting beside the calculations, but no VLOOKUP connects them. People read the table and type values into cells themselves, and mistakes go unnoticed. This article covers separating formula cells from manually entered cells, then reconstructing a single formula that matches all 16 values in a multiplier column.

#spreadsheet#python#reverse-engineering#audit
10 min read·20 Oct
Speed Up Screen Recordings While Keeping the Original Clock
AI

Speed Up Screen Recordings While Keeping the Original Clock

How to make a timelapse from a screen recording by speeding up idle sections more than active sections, using split, trim, and setpts, then joining them with concat. The main trap is that drawtext must come before setpts for the clock to follow the original timeline instead of the accelerated one.

#ffmpeg#video-editing#python#automation
9 min read·20 Oct
A 26-Page Bilingual Site: Why .php Broke Every Link
Web

A 26-Page Bilingual Site: Why .php Broke Every Link

The production page displayed PHP tags as raw text because the web server served .html as static files. Changing all 26 pages to .php broke links across the site. Then, after adding clean URLs, subpages bounced back to .php because the rewrite and 301 setup was only half done. This article includes the full .htaccess, eight cases for $base, and scripts that handle the whole site in one pass.

#htaccess#apache#nginx#php
9 min read·19 Oct
Making 20,000+ API Calls That Resume Without Leaving Corrupt Files
Linux

Making 20,000+ API Calls That Resume Without Leaving Corrupt Files

Generating 26,850 files through an API limited to 20 calls per minute requires three pieces—a system-wide throttle, resuming from what is already on disk, and a separate file validator—because files interrupted by network failures still have the right extension and a nonzero size.

#python#api#rate-limiting#automation
9 min read·19 Oct
A Blank Page with No Errors, All Because I Named a Function top
Web

A Blank Page with No Errors, All Because I Named a Function top

After deployment, the page was blank, the logs had no errors, and node --check found nothing wrong. But the browser stopped the entire script before the first line ran because a function named top collided with window.top. Here are the names to avoid, the script loading order trap, and the Playwright check every project should have.

#javascript#debugging#troubleshooting#playwright
8 min read·18 Oct
The Monitor That Hammered Devices Until the Whole Farm Went Down
Linux

The Monitor That Hammered Devices Until the Whole Farm Went Down

A monitoring loop that runs every 30 seconds and tries to recover every device that fails a ping sounds useful, until one device hangs and gets hammered repeatedly until the whole bus goes down. This article walks through ruling out hardware, tracking what was accessing the devices, and the settings that fixed it.

#monitoring#linux#python#systemd
10 min read·17 Oct
Images Fail on Mobile in Pages Hidden with display:none
Web

Images Fail on Mobile in Pages Hidden with display:none

A site switches pages using display:none, but opening another page on mobile leaves blank black images, even though the files return 200 and both fetch and new Image succeed. The culprit is the IntersectionObserver behind loading=lazy not firing when the page changes from none to block, and simply setting loading=eager is still not enough.

#javascript#css#mobile-dev#web-performance
9 min read·17 Oct
Reading ID Cards from the Web with a Localhost Agent
Hardware

Reading ID Cards from the Web with a Localhost Agent

Browsers cannot access smart card readers directly. A workable approach is to write a small agent that opens a port on localhost and let the web page call it. This article covers secure contexts, CORS preflight, port conflicts with other agents, PC/SC in SHARED mode, decoding TIS-620 manually, and the day antivirus flagged the program as a backdoor.

#hardware#cpp#windows#gui
11 min read·16 Oct
The NVR Kept Beeping, Then Fell Silent When the Disks Failed
Network

The NVR Kept Beeping, Then Fell Silent When the Disks Failed

The NVR's all-day beeping was not videoloss as everyone thought. It was ipconflict from cameras using DHCP with conflicting IP addresses. Someone got annoyed and muted the whole unit, so when both disks failed together, there was not a single beep. This article fixes the policy one event at a time through ISAPI.

#cctv#nvr#hikvision#isapi
9 min read·16 Oct
Cannot Delete Faces from a Terminal: API Returns notSupport
Hardware

Cannot Delete Faces from a Terminal: API Returns notSupport

The door's face recognition terminal advertises a command to delete faces from FDLib, but every variation I tried returned notSupport. People who had left could still scan their faces and get in. This article covers the dead ends, the approach that actually worked using UserInfo/Modify with the smallest possible body, and the two-way user registry sync that runs every night.

#hikvision#isapi#hardware#python
10 min read·15 Oct
Reading an Unknown Modbus Device: Verify Before Interpreting
Hardware

Reading an Unknown Modbus Device: Verify Before Interpreting

My notes on tracking down an undocumented device on an RS485 bus, from scanning baud rates and addresses one combination at a time to proving which registers hold live values and which are dead, and the lesson that guessing a device's type from number patterns alone can lead to months of misinterpretation.

#modbus#rs485#serial#iot
11 min read·14 Oct
Writing to DirectAdmin Through Panel Forms When the API Only Reads
DirectAdmin

Writing to DirectAdmin Through Panel Forms When the API Only Reads

Notes on letting a back-office app create and edit hosting packages in DirectAdmin when the official CMD_API only provides read operations, requiring POST requests directly to control panel forms, with verification rules that prohibit regex and two layers of kill switches disabled by default.

#directadmin#shared-host#php#laravel
9 min read·14 Oct
Registrar and Billing NS Mismatch: Audit All Three Sides First
Network

Registrar and Billing NS Mismatch: Audit All Three Sides First

A debugging log for "Wrong response from the server" when changing nameservers in a billing system, which failed only for domains with leftover glue IP data, followed by a three-way audit of the registrar, billing system, and actual DNS zones before I felt confident changing anything.

#dns#cloudflare#networking#rest-api
9 min read·13 Oct
Ingesting Mail Without Disrupting Someone Else's Mailbox
Linux

Ingesting Mail Without Disrupting Someone Else's Mailbox

The mailbox the app fetches from is one people actually use every day — fetching with UNSEEN silently misses mail when the owner reads it on their phone before cron runs. The right approach is UID SEARCH paired with UIDVALIDITY and BODY.PEEK, with deduplication handled on our side.

#mail#imap#pop3#dovecot
10 min read·13 Oct
Reads from a Replica Must Show How Many Minutes Old the Data Is
Linux

Reads from a Replica Must Show How Many Minutes Old the Data Is

replication can report a lag of 0 seconds while the data is stuck at yesterday, because these two signals measure different things — how to set up a monitor that checks both Seconds_Behind_Master and business data freshness, catches anyone writing directly to the slave, and has another check watching the monitor itself

#mariadb#mysql#monitoring#alerting
9 min read·12 Oct
The App Uses UTC, People Use Thai Time — 7 Hours Off, No Errors
Linux

The App Uses UTC, People Use Thai Time — 7 Hours Off, No Errors

Scheduled for three in the afternoon, published at ten at night. No errors, nothing wrong in the logs, because the app stores UTC while the person entering the time thinks in Thai time. This article traces the issue from an env setting that has no effect to the single line that writes Thai wall-clock time into a column everyone reads as UTC.

#laravel#php#mariadb#debugging
9 min read·11 Oct
Egress Firewall Breaks Domain Checks — Tests as root Pass
Security

Egress Firewall Breaks Domain Checks — Tests as root Pass

The domain order page reported every name as unavailable, even names I made up. The cause was an outbound DROP policy with port 43 missing from the allowed port list after an operating system migration. The trap was that tests as root passed every time. Testing must use the account that actually runs the app.

#firewall#csf#security#networking
9 min read·10 Oct
Web vs. Worker — mkdir 0700 Disables Inherited ACL Permissions
Linux

Web vs. Worker — mkdir 0700 Disables Inherited ACL Permissions

One app runs as two users, with php-fpm handling web requests and a separate account running the queue. Even with default ACLs fully configured, they still cannot write to each other's directories. The cause is mkdir with mode 0700, which sets a mask that wipes out the effective ACL permissions, plus a deletion trap that requires permissions on the directory, not the file.

#linux#permissions#posix-acl#php-fpm
9 min read·9 Oct
The Service Is Running but Dead — systemd restart Cannot Help
Linux

The Service Is Running but Dead — systemd restart Cannot Help

A long network outage broke DNS. A daemon holding a session with an external API failed to log in 194 times, then hung halfway through the process. It never exited, so the system never restarted it — why I stopped checking whether a process was alive and started measuring data freshness instead.

#systemd#linux#monitoring#alerting
9 min read·9 Oct
tinker Sends Mail but the Web Can't — proc_open Disabled in FPM
DirectAdmin

tinker Sends Mail but the Web Can't — proc_open Disabled in FPM

Sending mail from tinker works perfectly, but submitting the actual form sends nothing, and the mail log has no entries — because CLI and PHP-FPM are not the same PHP environment. Includes a working SMTP AUTH configuration for DirectAdmin hosting and the From rule you must get right.

#mail#smtp#laravel#php
9 min read·8 Oct
Site Not Updating After Deploy — Clear All Five Cache Layers
DirectAdmin

Site Not Updating After Deploy — Clear All Five Cache Layers

The files are uploaded, but the site still shows the old version, even after Ctrl+F5 — because five cache layers sit between the files on disk and what people see, and each needs a different clearing method. This article covers the sequence I use regularly, along with the trap that leaves the site stuck on 500 after clearing the view cache.

#deployment#laravel#php#php-fpm
10 min read·8 Oct
route:cache Turns Every GET into 405 — The Sub-path Trap
DirectAdmin

route:cache Turns Every GET into 405 — The Sub-path Trap

Deployment finished, I ran optimize as the guides suggested, and every page returned 405 Method Not Allowed even though route:list confirmed the routes were correct — a route cache that breaks only on certain servers, how to isolate it in 30 seconds, and three other issues I found while deploying an app under a sub-path.

#laravel#php#deployment#apache
9 min read·7 Oct
php -l Passes, Web Pages Return 500 — A Compile Gate Before Commit
Linux

php -l Passes, Web Pages Return 500 — A Compile Gate Before Commit

Templates broke twice in one month even though every php -l check passed, because php -l does not understand Blade syntax. This article covers the six-stage gate I put before commit — compiling the views and then linting the compiled files, plus a script that sweeps every GET route through the Kernel.

#laravel#php#blade#testing
9 min read·7 Oct
Always Claim the State Before Sending Anything Out
Linux

Always Claim the State Before Sending Anything Out

A "read the state, then send" guard looks safe until a worker gets killed after sending but before writing to the database, and a retry sends the same thing to the customer twice. This article covers the full set of duplicate prevention measures I use, from atomic claims to retry_after and failed() handlers that must never overwrite a successful state.

#laravel#php#queue#worker
11 min read·6 Oct
One Tunnel Is the Only Way In—and a Single Point of Failure
Network

One Tunnel Is the Only Way In—and a Single Point of Failure

The machine sits behind NAT with no ports open to the internet. Every way in runs through one cloudflared tunnel — web, SSH, and remote desktop. This article covers the four ingress routes I set up, how to read the signs when it goes down, and the trap that keeps the tunnel from starting at boot.

#cloudflare#cloudflared#tunnel#zero-trust
10 min read·5 Oct
Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read
Linux

Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read

In one day, 28 events became 56 messages, and 57 percent of all events were repeat alerts every 30 minutes. This article covers recipient tiers, merging overlapping cycles, state that survives restarts, and why you must not add keepalive to a chat app session.

#monitoring#alerting#bot#logging
9 min read·5 Oct
Shared Neutrals Across Transformers: Why Phase Balancing Is Not Enough
Hardware

Shared Neutrals Across Transformers: Why Phase Balancing Is Not Enough

A case involving two transformers with interconnected neutrals and houses tapped into one side's main feeder, leaving phase A 30-40 A higher even on days off, plus a meter-reading trap that can suggest a 200 A difference if the CT ratio is not verified first, and a repair sequence that must be followed in order.

#hardware#electrical#power#engineering
9 min read·30 Sep
Diagnosing Feeder Overload from Melted Insulation and Duty Cycles
Hardware

Diagnosing Feeder Overload from Melted Insulation and Duty Cycles

My investigation of a 95 mm² main cable with insulation melting at scattered points even though the system seemed to work normally, from the nameplates of 6 washers and dryers totaling 234.7 kW to a minute-by-minute current model explaining why a 482 A load on a cable rated for 180 A had not caused a visible system failure.

#hardware#electrical#power#engineering
10 min read·28 Sep
A Single-Page PHP and SQLite Document Vault That Keeps Originals
Linux

A Single-Page PHP and SQLite Document Vault That Keeps Originals

A document photo storage app for government submissions, using photos taken on site with a phone. Each upload produces three files—the display image, the original, and saved crop settings—so adjustments can be made later without losing quality, with automatic migration from JSON to SQLite and web server restrictions on folders containing personal data.

#php#sqlite#document#security
9 min read·27 Sep
Printing Dozens of Government Forms from HTML with Correct Margins
Linux

Printing Dozens of Government Forms from HTML with Correct Margins

A government submission document system where users click Print in their browsers and every sheet must fit within its margins, from measuring the actual printable area in millimeters and repeating headers and footers without position fixed, to four image frame types for different documents and automated overflow checks with Playwright instead of inspecting every sheet.

#pdf#php#css#printing
10 min read·26 Sep
News Stuck for Three Days Because the Feed Wasn't Sorted by Date
Linux

News Stuck for Three Days Because the Feed Wasn't Sorted by Date

The news site's homepage showed the same day's stories for three days straight, even though scheduled jobs ran, translations finished, and every step reported success. The causes were an RSS feed sorted by relevance rather than date, string sorting of pubDate that actually sorted by weekday name, and requesting too few stories, leaving nothing after filtering before the rest even got a look.

#python#javascript#rss#automation
10 min read·25 Sep
Sending Automated Job Files to Telegram and the Bot API's 50 MB Limit
Linux

Sending Automated Job Files to Telegram and the Bot API's 50 MB Limit

Set up a small local service that takes files from scripts and sends them to me on Telegram, with outbound delivery only, no polling, and no incoming commands, along with the actual Bot API limits to know before writing it, including 50 MB per file, images over 10 MB that must be sent as documents, and the Thai text trap in curl on Windows.

#telegram#bot#automation#python
8 min read·24 Sep
CDP Still Responds but Commands Fail After Chrome Updates Itself
Linux

CDP Still Responds but Commands Fail After Chrome Updates Itself

I run several automation jobs on one machine that need a real browser, and ran into two problems at once — jobs fighting over tabs and getting each other's results, and Chrome overwriting its own files during an update while the process was still running, leaving the port responsive but Runtime.evaluate unusable. This article covers how I isolate jobs, detect the failure, and repair it automatically.

#cdp#chromium#automation#windows
9 min read·23 Sep
A 25-Hour Power Outage Left Scheduled Jobs Missing
Linux

A 25-Hour Power Outage Left Scheduled Jobs Missing

The machine running my automated jobs lost power three times in three days, with the longest outage lasting 25 hours. When it came back, I found that jobs scheduled directly through Task Scheduler had skipped their runs entirely, while jobs managed by a dispatcher that checked the time itself had all caught up. This article covers the setting to enable, what to avoid during recovery, and the traps that led me to misread the status.

#windows#scheduler#automation#powershell
9 min read·22 Sep
Automated Image Uploads Stopped for 4 Days: ssh Chosen by PATH
Linux

Automated Image Uploads Stopped for 4 Days: ssh Chosen by PATH

Write subprocess.run(["ssh", ...]) and leave the system to choose the binary from the caller's PATH. One day the caller changes, and the binary changes with it. File transfers fail silently for 4 days while reporting success every time. This article covers how I tracked it down, a helper that probes the actual binary before using it, and a completeness check before declaring the job done.

#ssh#python#automation#scp
9 min read·21 Sep
Tell a Broken Link from an ISP Outage Before Restarting Anything
Linux

Tell a Broken Link from an ISP Outage Before Restarting Anything

A typical watchdog restarts NetworkManager when ping fails, which is wrong half the time because no internet access may mean the ISP is down, not that the adapter has failed. This article distinguishes three states using nmcli, then escalates from gentle recovery to PCI remove and rescan without rebooting the machine.

#linux#networking#systemd#debian
9 min read·20 Sep
Borrowing a VLAN for Windows Tools That Need the Same L2
Network

Borrowing a VLAN for Windows Tools That Need the Same L2

Many manufacturers' device discovery tools use broadcasts that cannot cross L3, so connecting over a VPN still leaves devices invisible. A solution that leaves switch ports alone is to put a small Windows VM on Proxmox with its interface directly in that VLAN, with pitfalls around vlan-aware bridges, a full thin pool breaking other VMs, and virtio leaving Windows without a network adapter after installation.

#proxmox#kvm#vlan#networking
8 min read·19 Sep
Set VLANs on a Hikvision Switch Using Its Web UI API
Network

Set VLANs on a Hikvision Switch Using Its Web UI API

A smart managed switch whose Switch ISAPI endpoints return 403 can still be scripted if you inspect what its web UI sends. This article turns that API into a VLAN configuration script with a dry-run before apply, covering the traps I actually hit, including different field names for reads and writes, CCTV mode forcing 10 Mbps, and a calculated PoE load that exceeds the switch's budget.

#networking#vlan#switch#poe
9 min read·17 Sep
Using an NVR to Reach Inaccessible Cameras with ISAPI and SDK
Network

Using an NVR to Reach Inaccessible Cameras with ISAPI and SDK

The cameras sit behind the NVR's second interface, and none are directly reachable. This article works through what the NVR's ISAPI can actually do, with 26 of 38 tested endpoints working, retrieves CCTV snapshots and RTSP through the NVR, finds when each channel last recorded, and falls back to HCNetSDK through ctypes to read device logs when ISAPI runs out of options.

#cctv#hikvision#nvr#isapi
11 min read·16 Sep
Every Layer Reported Success While the Output Was Garbage
Linux

Every Layer Reported Success While the Output Was Garbage

In one day, four layers of the same automated pipeline failed together, but every layer reported success without a single error line. The person who caught it was a user who said the site was not updating. These are the rules I wrote afterward so that failing stages could speak up for themselves.

#automation#python#monitoring#debugging
10 min read·15 Sep
Static on DirectAdmin: GET Returns 404 but HEAD Returns 200
DirectAdmin

Static on DirectAdmin: GET Returns 404 but HEAD Returns 200

I uploaded a static site to public_html and could not open it. Small files worked, large files returned 404, but HEAD returned 200 for every file. The cause was Laravel's .htaccess at the docroot, funneling everything into the front controller. This article covers the diagnosis and three ways out.

#directadmin#cloudflare#apache#htaccess
9 min read·14 Sep
Auditing prod with real customer data without leaving a single row
Linux

Auditing prod with real customer data without leaving a single row

The system owner asked me to test the entire system on the live server holding customer data without leaving even a single junk row behind. My approach was to bootstrap Laravel myself and call controllers directly inside a transaction that I rolled back, with a COUNT of every table before and after as evidence.

#laravel#php#mysql#mariadb
11 min read·13 Sep
Backups Succeeded Nightly, but Offsite Uploads Failed for 3 Weeks
Backup

Backups Succeeded Nightly, but Offsite Uploads Failed for 3 Weeks

cron ran every night, local files were complete every day, and the log said done, but offsite storage had not changed for 3 weeks. The first 8 nights of backups were already gone for good because local rotation had deleted them first. This article examines the two gaps that let the failure stay silent for so long.

#backup#rclone#s3#cloudflare
10 min read·11 Sep
Generating Thai PDF Quotes with Chromium and the Pitfalls I Hit
Linux

Generating Thai PDF Quotes with Chromium and the Pitfalls I Hit

Building a system to generate PDF quotations from HTML with Chromium for multiple companies in a group and multiple templates, from an Executable doesn't exist error caused by looking for the wrong headless shell, to A4 layout and pagination, sanitizing Thai filenames, and a token-based document numbering system where an accidental slash broke the path.

#pdf#chromium#playwright#python
9 min read·31 Aug
Moving Computation from PHP to the Browser: 12 Seconds to 12 ms
Linux

Moving Computation from PHP to the Browser: 12 Seconds to 12 ms

A single page ran tens of thousands of subqueries and took 12 seconds to load. I fixed it by precomputing a summary table, sending a single 107 KB dataset, and moving the formulas to the browser, bringing computation down to 12 milliseconds, with a parity test that requires JS results to match PHP on every row.

#performance#php#javascript#laravel
9 min read·29 Aug
Automating Remote Windows Tasks with SSH + Task Scheduler
Linux

Automating Remote Windows Tasks with SSH + Task Scheduler

Move heavy workloads to a more powerful machine over SSH, then automate them with schtasks — with pitfalls that cost me real time, from landing in PowerShell 5 instead of bash over ssh, to backslashes getting swallowed when writing .bat files, to file transfers hanging until I switched to plink

#windows#ssh#powershell#automation
9 min read·26 Aug
Inventory Switch and AP Ports with SNMP Walk and MAC Tables
Network

Inventory Switch and AP Ports with SNMP Walk and MAC Tables

You can build your own Layer 2 inventory without a controller or NMS. Write an SNMP walk with pysnmp asyncio to retrieve sysDescr, ifTable, and the bridge FDB, then convert OID indexes back into MAC addresses to find which device connects to which port on which switch.

#snmp#networking#python#monitoring
9 min read·22 Aug
Read Device Uptime Without SNMP/SSH/API Using TCP Timestamps
Network

Read Device Uptime Without SNMP/SSH/API Using TCP Timestamps

Many IP cameras, door controllers, and embedded devices have no SNMP, SSH, or API for querying uptime. But if they return TCP timestamps, I can read their boot time remotely without logging in even once.

#networking#monitoring#linux#python
7 min read·20 Aug
The Web User's crontab Isn't the One Running Your Jobs
Linux

The Web User's crontab Isn't the One Running Your Jobs

Schedule changes saved through the web still leave jobs running at the old times because the script writes the web user's crontab, not the user’s crontab that cron actually reads. How to spot it in 3 commands, the sudoers drop-in I used, and how to design it so this cannot happen again.

#cron#linux#sysadmin#debugging
7 min read·19 Aug
Laravel Queues on Shared Hosting — Don't Share Redis with Cache
Linux

Laravel Queues on Shared Hosting — Don't Share Redis with Cache

If you're enabling queues for Laravel on shared hosting and planning to separate the queue from cache with different REDIS_DB numbers, that won't help. maxmemory and eviction apply to the whole instance, not individual DBs. This article explains why I chose the database driver instead, with complete systemd units to replace Supervisor.

#laravel#redis#queue#systemd
10 min read·17 Aug
Laravel Behind a Proxy Sees 127.0.0.1 — Silent Form, No Error
DirectAdmin

Laravel Behind a Proxy Sees 127.0.0.1 — Silent Form, No Error

A contact form on DirectAdmin says the sender is not human, sends no mail, and leaves every log silent — the cause is Laravel behind nginx-to-apache seeing $request->ip() as 127.0.0.1, failing the bot check and returning before the mail is sent, with an explanation of why TrustProxies must not use '*'

#laravel#php#nginx#php-fpm
9 min read·16 Aug
WAF Returns 406 for Thai Text in POST: Fixing It Without Root
Security

WAF Returns 406 for Thai Text in POST: Fixing It Without Root

ModSecurity on the host returns 406 for certain Thai sentences in the POST body, preventing forms from saving — but as a hosting tenant, I cannot change the rules myself. This article shows how to isolate the offending text through black-box testing, then work around it with JS encoding on submit + middleware decoding in Laravel.

#modsecurity#waf#laravel#php
9 min read·15 Aug
My Firewall Permanently Banned Googlebot and Hid Sites from Search
Security

My Firewall Permanently Banned Googlebot and Hid Sites from Search

I investigated why sites disappeared from search results and found Googlebot in my own firewall's permanent ban list. 408 of 500 entries came from the counter tallying WAF false positives. How I lifted the bans, wrote a script to fetch official bot IP ranges into the ignore list daily, and dealt with a grep trap and the side effects of blocking entire countries.

#csf#firewall#seo#googlebot
10 min read·14 Aug
MariaDB on defaults — a 128M buffer pool on a 55 GB RAM server
Linux

MariaDB on defaults — a 128M buffer pool on a 55 GB RAM server

A 16 core server with 55 GB RAM and an SSD, but MariaDB still had the default 128 MB buffer pool. The result was 50 million disk reads in 5 days. The full before-and-after settings, service name and config path traps on DirectAdmin/Debian, rollback steps, and another 500 MyISAM tables waiting for conversion.

#mariadb#mysql#performance#tuning
10 min read·13 Aug
AI Crawler Overloads ?add-to-cart= — Block It Without Hurting SEO
Security

AI Crawler Overloads ?add-to-cart= — Block It Without Hurting SEO

Server load triggered an email alert. The logs showed that 84% of the day's traffic came from one AI crawler following add-to-cart links for every product. I added layers of protection with robots.txt, rel=nofollow and nginx limit_req while keeping all product pages indexable.

#nginx#security#wordpress#seo
10 min read·12 Aug
Bulk Import Legacy .xls Files into Laravel with PhpSpreadsheet
Linux

Bulk Import Legacy .xls Files into Laravel with PhpSpreadsheet

Legacy .xls files with Thai headers encoded in tis-620 have unreadable column names, large files make PhpSpreadsheet exhaust memory and crash, and files users merge themselves contain duplicate rows — use column positions instead of names, add a read filter, and deduplicate with a natural key.

#php#laravel#excel#phpspreadsheet
9 min read·10 Aug
Repairing a Claude Code Session .jsonl File Too Large to Open at 38 MB
AI

Repairing a Claude Code Session .jsonl File Too Large to Open at 38 MB

A single session stretched over 8 days grew to 38.61 MB / 11,537 records until the UI could no longer open its history. I wrote a Python script to process it line by line, trimming only the parts that were already unreadable to humans, bringing it down to 17.41 MB while preserving all 8,171 message uuid values in their original order, with no orphans.

#claude-code#ai#python#automation
8 min read·8 Aug
Stolen Mailbox Password Sent 1,786 Spam Emails Unseen by CSF/LFD
Security

Stolen Mailbox Password Sent 1,786 Spam Emails Unseen by CSF/LFD

A real case of a stolen mailbox password used to send 50-75 spam emails a day, below a limit of 200, for 4 straight weeks without a single response from CSF/LFD/BFM because they only count failed authentication attempts, with the decisive question of whether an account was compromised or merely under attack, exim forensic commands, and a blocklist removal playbook

#mail#exim#security#csf
11 min read·7 Aug
Charset Mismatch Silently Kills JOIN Indexes — an utf8mb4 Mirror
Linux

Charset Mismatch Silently Kills JOIN Indexes — an utf8mb4 Mirror

The old database uses tis620 and the new one uses utf8mb4. JOIN across databases, and every index immediately becomes decorative without a single error — how to prove it with EXPLAIN and build a mirror that converts tis620 to utf8mb4 during the dump, plus the sed trap that silently breaks tables.

#mariadb#mysql#charset#utf8mb4
10 min read·6 Aug
One Query Hung for 54 Minutes, Causing Sitewide 504s via MyISAM Locks
Linux

One Query Hung for 54 Minutes, Causing Sitewide 504s via MyISAM Locks

A single 30 MB table caused 504s on every page across the site, even with 9.5 GB of RAM still free — I traced information_schema.PROCESSLIST to a single SELECT that had been stuck for 54 minutes on a MyISAM table, cascading into metadata locks across the server, with an immediate fix and ways to prevent it from happening again.

#mariadb#mysql#myisam#innodb
10 min read·6 Aug
Recover MariaDB Redo Log Corruption with force_recovery and Reinit
Linux

Recover MariaDB Redo Log Corruption with force_recovery and Reinit

MariaDB would not start after an unclean shutdown — a corrupted InnoDB redo log took the mysql.global_priv system table down with it. This is the recovery sequence I actually used, starting with innodb_force_recovery to make the data readable, then dumping each db before reinitializing the whole system, with one rule never to raise force_recovery higher than necessary.

#mariadb#mysql#disaster-recovery#debugging
9 min read·25 Jul
Porting Excel Formulas to the Web: Floating-Point Traps
Linux

Porting Excel Formulas to the Web: Floating-Point Traps

Porting engineering formulas from an Excel file to a web calculator produced differences around 1e-13 — not a math bug, but differences in how JS and Excel evaluate floating-point powers and multiplication order, with a method for reading formulas and values separately using openpyxl and building fixtures that actually catch regressions.

#automation#python#debugging#structural
8 min read·23 Jul
Control TP-Link Omada OC200 via REST API with Python
Network

Control TP-Link Omada OC200 via REST API with Python

A practical guide to logging into an OC200 controller through REST API v2 with Python — handling CSRF tokens, self-signed certs, fetching sites/devices/SSIDs/DHCP/port-forward rules, and writing back with PATCH, including the read-only fields that cause a nonzero errorCode

#omada#networking#api#python
8 min read·20 Jul
Screenshot Multiple Domains Directly from Origin, Bypassing Cloudflare
Network

Screenshot Multiple Domains Directly from Origin, Bypassing Cloudflare

Need screenshots of dozens of domains for inspection, or want to confirm that the origin server (not the CDN cache) renders correctly — but Chrome's screenshot flag has been broken since version 132, and AV reverts hosts edits? This article uses Playwright to override DNS at the process level, solving both problems.

#debugging#cloudflare#python#automation
7 min read·18 Jul
~30 Seconds Turns Tiny: Flarum's Litedown Subscript Trap
Linux

~30 Seconds Turns Tiny: Flarum's Litedown Subscript Trap

Write "~30 seconds" in a Flarum post and 30 turns into a tiny subscript — because Flarum's Litedown interprets ~ as a subscript opener. Thai users who use ~ to mean "approximately" run into this regularly. This article explains the cause, workarounds, and database fixes for content stored as s9e XML rather than raw markdown.

#flarum#markdown#debugging#php-fpm
6 min read·18 Jul
Mail Stopped After Moving to Cloudflare — Proxied Mail Records
Network

Mail Stopped After Moving to Cloudflare — Proxied Mail Records

After adding a domain to Cloudflare and pointing DNS at the host, incoming mail suddenly stops — the most common cause I see is leaving mail/smtp/pop records proxied (orange cloud), even though Cloudflare's proxy only handles HTTP/S. This article covers which records must be grey + token/SSL pitfalls.

#cloudflare#dns#mail#security
7 min read·17 Jul
nginx heap overflow spans 15 years—version checks aren't enough
Security

nginx heap overflow spans 15 years—version checks aren't enough

CVE-2026-42533 affects nginx versions 0.9.6 through 1.31.2, spanning 15 years, but actual exposure depends on the config, not the version. The problem is that news reports describe narrower conditions than the real ones. Checking against those reports can return "safe" even when the config still meets the conditions.

#security#nginx#cve#vulnerability
9 min read·16 Jul
Writing Scripts on Windows for Linux — Two File Cleanup Traps
Linux

Writing Scripts on Windows for Linux — Two File Cleanup Traps

Writing helper scripts on Windows and using scp to run them on Linux is routine work, but two things keep catching me out — d:\ paths embedded in scripts turn into strange filenames in /tmp that trigger the file monitor, and token/payload files get left behind on the server. This article lays out my rules for avoiding these mistakes.

#automation#windows#linux#sysadmin
6 min read·16 Jul
Reading Modbus RTU over an RS485 Gateway Without Phantom Values
Linux

Reading Modbus RTU over an RS485 Gateway Without Phantom Values

Connecting multiple Modbus RTU meters/sensors on an RS485 bus through a serial-to-Ethernet gateway can produce phantom values, incorrect readings, or devices that do not actually exist — because a transparent gateway uses RTU-over-TCP, which requires assembling frames and calculating CRC yourself, and validating every reply. This article covers the rules I learned from doing it.

#modbus#rs485#iot#python
9 min read·15 Jul
libmodsecurity3 + nginx — only ctl:ruleEngine exclusions work
Security

libmodsecurity3 + nginx — only ctl:ruleEngine exclusions work

ModSecurity v3 (libmodsecurity3) + nginx + OWASP CRS — the ctl:ruleRemoveById taught in the guides does nothing and fails silently. This article covers what I found after trying every approach: ctl:ruleEngine works, scoped with a Host+URI chained rule, and the exclusion file must load before CRS.

#modsecurity#nginx#security#firewall
9 min read·10 Jul
WordPress 500s, Empty debug.log — The Fatal Is in the Apache Log
DirectAdmin

WordPress 500s, Empty debug.log — The Fatal Is in the Apache Log

WordPress returns 500 only on portfolio pages, but debug.log is empty — because the fatal error during theme rendering happens before WP logging is ready. The answer is in the apache error log (proxy_fcgi), along with a fix for a child theme with a stale template copied from an older parent version.

#wordpress#php-fpm#debugging#logging
8 min read·8 Jul
AdSense Says ads.txt Is Missing: Tracing the Logs Layer by Layer
Network

AdSense Says ads.txt Is Missing: Tracing the Logs Layer by Layer

The AdSense Sites page reports ads.txt as "Not found" even though the file exists and is correct — a forensic investigation using curl + access logs to prove, layer by layer, whether Googlebot visited and what it received. Case closed: nothing was blocking it; Google's own status was stale.

#adsense#nginx#cloudflare#logging
9 min read·8 Jul
Recovering Old Website Images with Wayback Machine CDX API + Python
Backup

Recovering Old Website Images with Wayback Machine CDX API + Python

A website over a decade old had broken images throughout because it hotlinked to a dead CDN — I wrote a Python script to list images from the Wayback Machine CDX API and restore them to their original paths without touching a single character of HTML, with the full script I actually used.

#wayback#python#backup#disaster-recovery
9 min read·7 Jul
Legacy Apps Break After Migration — MariaDB Strict Mode Errors
Linux

Legacy Apps Break After Migration — MariaDB Strict Mode Errors

After moving a website from an old server to a new one, the registration form breaks even though the code has not changed — the cause is STRICT_TRANS_TABLES in the sql_mode of newer MariaDB versions, with fixes at 3 levels, from conf.d to per-app SET SESSION

#mariadb#mysql#migration#server-migration
7 min read·7 Jul
Lost the .com: Move to .net on DirectAdmin Without Data Loss
DirectAdmin

Lost the .com: Move to .net on DirectAdmin Without Data Loss

A .com domain expired and ended up with a domain squatter — how I moved the whole site (files/mail/DNS/SSL) to a new .net domain on DirectAdmin with a single CMD_CHANGE_DOMAIN command, plus 4 follow-up tasks that do not happen automatically

#directadmin#dns#ssl#letsencrypt
9 min read·6 Jul
Reading exim mainlog for Stuck Clients — Mail Forensics Part 1
Security

Reading exim mainlog for Stuck Clients — Mail Forensics Part 1

How I read /var/log/exim/mainlog to distinguish an office machine using the wrong password from an attacker running a brute force attack, with the commands I use when customers need answers fast.

#mail-forensics#exim#mail#smtp
12 min read·23 May
Moving DA from CentOS 7 to Debian 12, Part 4 — Cutover and 17 Issues
DirectAdmin

Moving DA from CentOS 7 to Debian 12, Part 4 — Cutover and 17 Issues

The actual cutover, where the near-0 downtime plan fell apart because of the DA Legacy license and slow qcow2 reads, ended with 1 hour of downtime and 17 issues in quick succession after the switch — critical web stack outages, medium issues affecting subsets of services, and minor debugging notes.

#cutover#migration#directadmin#qcow2
13 min read·20 May
CSF Auto-Unblock for Ghost Devices Amid Botnet Brute Force
Security

CSF Auto-Unblock for Ghost Devices Amid Botnet Brute Force

A real case on a DA shared host where a ghost device in an office got the entire office blocked by CSF every morning. I explain a 4-layer auto-unblock setup that distinguishes office devices from a botnet without accidentally unblocking attackers.

#csf#firewall#brute-force#security
11 min read·20 May