EST. 2026 — Notes from real server work SYSADMIN · SECURITY · AI · DIRECTADMIN Issue 01
THE SYSADMIN ALMANAC
Home / Tags / Best Practices
Best Practices

#Best Practices

39 articles
Workflow
Passing the Sandbox Doesn't Mean It Works — 7 False Signals
Linux

Passing the Sandbox Doesn't Mean It Works — 7 False Signals

The automated ordering system I wrote passed every command in a provider's sandbox, but almost everything the sandbox told me differed from reality, from response times and charges to accepted names and items that were created successfully but could not be found. Here are 7 false signals and how to write code that does not fall for them.

#api#rest-api#testing#debugging
9 min read·26 Oct
Cross-Model Audits of AI-Written Code — Process, Not Faith
AI

Cross-Model Audits of AI-Written Code — Process, Not Faith

After using AI to port a calculator from a 35-sheet engineering spreadsheet, how do I know it is correct? This article covers three independent audit rounds by an agent from another provider, the rules to set before starting, confidence levels that keep claims within the evidence, and the classes of errors AI kept making in every round.

#ai#llm#ai-agent#claude-code
9 min read·26 Oct
Silent Overwrites on Upload: Three File Copying Traps
Linux

Silent Overwrites on Upload: Three File Copying Traps

Upload two files with the same name together, and the second silently overwrites the first. No error, no failing exit code, and every page still returns 200. The English page just becomes a Thai page. These are three traps that let file copying cause damage without any warning.

#deployment#pscp#plink#putty
8 min read·24 Oct
Let Clients Pick Photos on a Web Page Without a Login
Web

Let Clients Pick Photos on a Web Page Without a Login

When the person choosing photos for a website is not comfortable with computers, adding registration and login means the work stops at the login screen. This article covers a photo picker with no authentication at all, but with a server endpoint that checks every value against an allowlist strict enough to leave nothing from arbitrary input, plus limits at every layer and a backup before every write.

#php#javascript#web-security#permissions
8 min read·23 Oct
Build APM on Windows, Part 5: Compile ImageMagick 7 for PHP imagick
Web

Build APM on Windows, Part 5: Compile ImageMagick 7 for PHP imagick

Part 5 of the Build APM on Windows series compiles ImageMagick 7.1.2-34 and around 40 delegate libraries from source with the official toolchain, including the channel mask patch from 64 to 32 that lets imagick build, and why HEIC cannot be written

#windows#php#devops#powershell
14 min read·22 Oct
Build APM on Windows, Part 4: Compile MariaDB 11.8 LTS
Web

Build APM on Windows, Part 4: Compile MariaDB 11.8 LTS

Part 4 of the Build APM on Windows series walks through compiling MariaDB 11.8.9 from source with CMake and Ninja, from preparing the tools and running the script to testing on port 3399 and explaining why collation must be pinned to utf8mb4_unicode_ci

#mariadb#windows#devops#powershell
14 min read·22 Oct
Scripting Government PDF Forms and Making Safer ID Copies
AI

Scripting Government PDF Forms and Making Safer ID Copies

Government PDFs distributed as fillable forms often cannot actually handle Thai text. Subdivided fields let digits run over the borders, and tone marks collide with vowels above the line. This article covers listing field names first, filling the fields and flattening them into page content, drawing Thai text with a font that includes all the vowel marks, and making actual-size ID copies with purpose-specific watermarks.

#pdf#python#document#printing
10 min read·22 Oct
Reverse Engineering Spreadsheet Constants into Formulas
AI

Reverse Engineering Spreadsheet Constants into Formulas

A sheet has a criteria table sitting beside the calculations, but no VLOOKUP connects them. People read the table and type values into cells themselves, and mistakes go unnoticed. This article covers separating formula cells from manually entered cells, then reconstructing a single formula that matches all 16 values in a multiplier column.

#spreadsheet#python#reverse-engineering#audit
10 min read·20 Oct
A 26-Page Bilingual Site: Why .php Broke Every Link
Web

A 26-Page Bilingual Site: Why .php Broke Every Link

The production page displayed PHP tags as raw text because the web server served .html as static files. Changing all 26 pages to .php broke links across the site. Then, after adding clean URLs, subpages bounced back to .php because the rewrite and 301 setup was only half done. This article includes the full .htaccess, eight cases for $base, and scripts that handle the whole site in one pass.

#htaccess#apache#nginx#php
9 min read·19 Oct
Making 20,000+ API Calls That Resume Without Leaving Corrupt Files
Linux

Making 20,000+ API Calls That Resume Without Leaving Corrupt Files

Generating 26,850 files through an API limited to 20 calls per minute requires three pieces—a system-wide throttle, resuming from what is already on disk, and a separate file validator—because files interrupted by network failures still have the right extension and a nonzero size.

#python#api#rate-limiting#automation
9 min read·19 Oct
A Cookie Banner That Loads GTM and GA4 Only After Consent
Web

A Cookie Banner That Loads GTM and GA4 Only After Consent

Most cookie banners look good but block nothing, because GTM and GA4 have already loaded from the first line of the page. This article covers a setup that loads the container only after acceptance, puts GA4 inside GTM to prevent double counting, hides AdSense slots initially, and checks that outgoing requests to those services stay at zero.

#seo#adsense#javascript#playwright
9 min read·18 Oct
A Blank Page with No Errors, All Because I Named a Function top
Web

A Blank Page with No Errors, All Because I Named a Function top

After deployment, the page was blank, the logs had no errors, and node --check found nothing wrong. But the browser stopped the entire script before the first line ran because a function named top collided with window.top. Here are the names to avoid, the script loading order trap, and the Playwright check every project should have.

#javascript#debugging#troubleshooting#playwright
8 min read·18 Oct
Writing to DirectAdmin Through Panel Forms When the API Only Reads
DirectAdmin

Writing to DirectAdmin Through Panel Forms When the API Only Reads

Notes on letting a back-office app create and edit hosting packages in DirectAdmin when the official CMD_API only provides read operations, requiring POST requests directly to control panel forms, with verification rules that prohibit regex and two layers of kill switches disabled by default.

#directadmin#shared-host#php#laravel
9 min read·14 Oct
Registrar and Billing NS Mismatch: Audit All Three Sides First
Network

Registrar and Billing NS Mismatch: Audit All Three Sides First

A debugging log for "Wrong response from the server" when changing nameservers in a billing system, which failed only for domains with leftover glue IP data, followed by a three-way audit of the registrar, billing system, and actual DNS zones before I felt confident changing anything.

#dns#cloudflare#networking#rest-api
9 min read·13 Oct
Let AI Read Documents, but Let Code Do the Math
AI

Let AI Read Documents, but Let Code Do the Math

The controller requires manually entered values to fall between 0 and 100, but the path where AI reads values from documents and feeds them in only checks whether they are numeric. A single negative value inflated the result to 5.25 times what it should have been without an error — a lesson in treating AI output as untrusted input, just like values entered by people.

#ai#llm#claude#claude-code
10 min read·12 Oct
The App Uses UTC, People Use Thai Time — 7 Hours Off, No Errors
Linux

The App Uses UTC, People Use Thai Time — 7 Hours Off, No Errors

Scheduled for three in the afternoon, published at ten at night. No errors, nothing wrong in the logs, because the app stores UTC while the person entering the time thinks in Thai time. This article traces the issue from an env setting that has no effect to the single line that writes Thai wall-clock time into a column everyone reads as UTC.

#laravel#php#mariadb#debugging
9 min read·11 Oct
Secrets Set to 600, but 0644 Logs Go Offsite Every Night
Security

Secrets Set to 600, but 0644 Logs Go Offsite Every Night

The .env is encrypted and permissions are locked down on every secret file, but customer data still leaks off the machine every night through something nobody classifies as confidential: logs that every account on the machine can read, bundled into backups uploaded to object storage. Here is how I stop it at the source by logging only result codes.

#security#logging#backup#offsite
10 min read·10 Oct
Web vs. Worker — mkdir 0700 Disables Inherited ACL Permissions
Linux

Web vs. Worker — mkdir 0700 Disables Inherited ACL Permissions

One app runs as two users, with php-fpm handling web requests and a separate account running the queue. Even with default ACLs fully configured, they still cannot write to each other's directories. The cause is mkdir with mode 0700, which sets a mask that wipes out the effective ACL permissions, plus a deletion trap that requires permissions on the directory, not the file.

#linux#permissions#posix-acl#php-fpm
9 min read·9 Oct
php -l Passes, Web Pages Return 500 — A Compile Gate Before Commit
Linux

php -l Passes, Web Pages Return 500 — A Compile Gate Before Commit

Templates broke twice in one month even though every php -l check passed, because php -l does not understand Blade syntax. This article covers the six-stage gate I put before commit — compiling the views and then linting the compiled files, plus a script that sweeps every GET route through the Kernel.

#laravel#php#blade#testing
9 min read·7 Oct
Always Claim the State Before Sending Anything Out
Linux

Always Claim the State Before Sending Anything Out

A "read the state, then send" guard looks safe until a worker gets killed after sending but before writing to the database, and a retry sends the same thing to the customer twice. This article covers the full set of duplicate prevention measures I use, from atomic claims to retry_after and failed() handlers that must never overwrite a successful state.

#laravel#php#queue#worker
11 min read·6 Oct
Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read
Linux

Too Many Alerts Mean No Alerts: Tuning a Bot People Will Read

In one day, 28 events became 56 messages, and 57 percent of all events were repeat alerts every 30 minutes. This article covers recipient tiers, merging overlapping cycles, state that survives restarts, and why you must not add keepalive to a chat app session.

#monitoring#alerting#bot#logging
9 min read·5 Oct
Food Factory Drawings: Sheets, Scales, and Paper Sizes
AI

Food Factory Drawings: Sheets, Scales, and Paper Sizes

A food factory application drawing set has 7 items, each with its own scale range, and some numbers to work out before drawing—how wide a plan will fit on A4 or A3 at 1 to 100, and how to split it across sheets without changing the scale if it will not fit.

#drafting#cad#sketchup#3d-modeling
10 min read·4 Oct
From Model to 6-Meter Steel Cutting Layout and Workshop Cut Sheet
AI

From Model to 6-Meter Steel Cutting Layout and Workshop Cut Sheet

Read part lengths from the model, consolidate duplicates, then arrange a first-fit decreasing cutting layout on standard 6-meter stock with a 3-millimeter saw kerf allowance, plus a notch-and-bend option for right-angle corners that reduces joints from 12 to 6

#structural#steel-design#sketchup#ruby
9 min read·3 Oct
Quantity Takeoff from SketchUp Models with Ruby Scripts
AI

Quantity Takeoff from SketchUp Models with Ruby Scripts

A finished model already contains every number a quantity takeoff needs. Give groups readable names from the start, then write a script to grep through them, with two traps to watch for that can double your areas.

#3d-modeling#sketchup#ruby#drafting
9 min read·3 Oct
Sizing Main Cables and Breakers by Installation Method
Hardware

Sizing Main Cables and Breakers by Installation Method

Notes on choosing the main cable and breaker for a new 253 A feeder, why the same size CV cable can carry 227 A or 364 A depending on how it is installed, and the option of downsizing to 95 and compensating with operating rules, at the cost of the main breaker tripping if anyone breaks those rules.

#hardware#electrical#power#engineering
10 min read·29 Sep
Sending Automated Job Files to Telegram and the Bot API's 50 MB Limit
Linux

Sending Automated Job Files to Telegram and the Bot API's 50 MB Limit

Set up a small local service that takes files from scripts and sends them to me on Telegram, with outbound delivery only, no polling, and no incoming commands, along with the actual Bot API limits to know before writing it, including 50 MB per file, images over 10 MB that must be sent as documents, and the Thai text trap in curl on Windows.

#telegram#bot#automation#python
8 min read·24 Sep
CDP Still Responds but Commands Fail After Chrome Updates Itself
Linux

CDP Still Responds but Commands Fail After Chrome Updates Itself

I run several automation jobs on one machine that need a real browser, and ran into two problems at once — jobs fighting over tabs and getting each other's results, and Chrome overwriting its own files during an update while the process was still running, leaving the port responsive but Runtime.evaluate unusable. This article covers how I isolate jobs, detect the failure, and repair it automatically.

#cdp#chromium#automation#windows
9 min read·23 Sep
A 25-Hour Power Outage Left Scheduled Jobs Missing
Linux

A 25-Hour Power Outage Left Scheduled Jobs Missing

The machine running my automated jobs lost power three times in three days, with the longest outage lasting 25 hours. When it came back, I found that jobs scheduled directly through Task Scheduler had skipped their runs entirely, while jobs managed by a dispatcher that checked the time itself had all caught up. This article covers the setting to enable, what to avoid during recovery, and the traps that led me to misread the status.

#windows#scheduler#automation#powershell
9 min read·22 Sep
Automated Image Uploads Stopped for 4 Days: ssh Chosen by PATH
Linux

Automated Image Uploads Stopped for 4 Days: ssh Chosen by PATH

Write subprocess.run(["ssh", ...]) and leave the system to choose the binary from the caller's PATH. One day the caller changes, and the binary changes with it. File transfers fail silently for 4 days while reporting success every time. This article covers how I tracked it down, a helper that probes the actual binary before using it, and a completeness check before declaring the job done.

#ssh#python#automation#scp
9 min read·21 Sep
Every Layer Reported Success While the Output Was Garbage
Linux

Every Layer Reported Success While the Output Was Garbage

In one day, four layers of the same automated pipeline failed together, but every layer reported success without a single error line. The person who caught it was a user who said the site was not updating. These are the rules I wrote afterward so that failing stages could speak up for themselves.

#automation#python#monitoring#debugging
10 min read·15 Sep
Auditing prod with real customer data without leaving a single row
Linux

Auditing prod with real customer data without leaving a single row

The system owner asked me to test the entire system on the live server holding customer data without leaving even a single junk row behind. My approach was to bootstrap Laravel myself and call controllers directly inside a transaction that I rolled back, with a COUNT of every table before and after as evidence.

#laravel#php#mysql#mariadb
11 min read·13 Sep
The Web User's crontab Isn't the One Running Your Jobs
Linux

The Web User's crontab Isn't the One Running Your Jobs

Schedule changes saved through the web still leave jobs running at the old times because the script writes the web user's crontab, not the user’s crontab that cron actually reads. How to spot it in 3 commands, the sudoers drop-in I used, and how to design it so this cannot happen again.

#cron#linux#sysadmin#debugging
7 min read·19 Aug
Encrypt Secrets in .env and Decrypt Them When Config Loads
Security

Encrypt Secrets in .env and Decrypt Them When Config Loads

How to store passwords and API keys as ciphertext in Laravel's .env and have config decrypt them on load, using a single helper that detects encrypted values, plus the APP_KEY mismatch across environments that can bring everything down if you get it wrong.

#laravel#php#security#encryption
9 min read·18 Aug
Auditing a Deployed Laravel Site — 3 Layers in Priority Order
Security

Auditing a Deployed Laravel Site — 3 Layers in Priority Order

After deploying Laravel to production, check security in 3 layers — probe externally for exposed endpoints, check files/permissions on the server, then grep for vulnerabilities in the code. This article provides commands and P0-P2 priorities for deciding what to fix first.

#security#hardening#php-fpm#cloudflare
8 min read·17 Jul
Writing Scripts on Windows for Linux — Two File Cleanup Traps
Linux

Writing Scripts on Windows for Linux — Two File Cleanup Traps

Writing helper scripts on Windows and using scp to run them on Linux is routine work, but two things keep catching me out — d:\ paths embedded in scripts turn into strange filenames in /tmp that trigger the file monitor, and token/payload files get left behind on the server. This article lays out my rules for avoiding these mistakes.

#automation#windows#linux#sysadmin
6 min read·16 Jul
Read-only Debian shared-host security audit — avoid banning yourself
Security

Read-only Debian shared-host security audit — avoid banning yourself

Audit the security of a Debian server running DirectAdmin in read-only mode first — but the first trap is that scan traffic itself triggers CSF's brute-force rules and gets your own IP banned. This article lays out a safe audit sequence + areas to check in 2024-2026

#security#hardening#csf#cve
9 min read·16 Jul
Mail Checks Ban the Whole Office—Fixing It with Login Auto-Whitelist
Security

Mail Checks Ban the Whole Office—Fixing It with Login Auto-Whitelist

Real mail clients behave exactly like bots — office machines keep retrying old passwords until CSF bans the whole office. This article covers an auto-whitelist system that watches for successful logins + Thai GeoIP, then grants a temporary allow for 24 hours, along with 5 pitfalls I learned the hard way.

#csf#firewall#mail#exim
9 min read·10 Jul
Managing Linux Servers Safely with AI over SSH
AI

Managing Linux Servers Safely with AI over SSH

How I actually use Claude to audit and fix Linux server security over SSH, from preparing my machine and storing secrets to hard rules before touching production

#claude-code#ssh#ssh-hardening#security
13 min read·12 Jun
Rebuilding DirectAdmin Backups with NFS and POSIX ACL in Production
Backup

Rebuilding DirectAdmin Backups with NFS and POSIX ACL in Production

A record of rebuilding backups on a DirectAdmin server after migrating to a new machine — from FTP push that broke at three layers to an NFSv4 hard mount + POSIX ACL, with kernel-level lessons learned along the way.

#backup#nfs#posix-acl#directadmin
13 min read·28 May