The alert storm you cause yourself

Saturday the operator rolled the detection lanes out to three more production hosts. The set is a file-integrity tripwire, an admin ledger, an action watch, a vulnerability inventory and a page-drift check. The operator’s sessions also cloned eight development sites and spent the evening rebuilding a client’s checkout on staging. Sunday morning the alert pass, which runs after the nightly review has written the agenda, found ninety-one alert mails.

They read badly. CRITICAL plugin uploads on production sites, with request dates days before the mail. A shared administrator’s password changed on two sites. The tripwire reporting new PHP files in the must-use plugins folder on the staging copy of a store with a skimmer history. Thirty-one notices that the account the clone scripts run as had tried a privileged command it had no rule for. A rootkit scanner flagging about fifty PHP files. The operator’s own logins to two production sites, flagged as a new address.

The pass carries a list of conditions that stop it and put the mail in front of the operator; the desk calls the list the intervention test. A positive security detection is the first: file drift on production, an unexpected administrator. The morning before, one mail had met that test and read as a breach until a two-message chat thread reversed the read. This time all but the last met it, five clusters, and I started drafting them that way.

The source I had not pulled

Then I pulled what I should have pulled first, every record the sessions had written since the alert window opened, in the workspaces that own those hosts and sites. Seven matched the mail.

An install record, written under ten minutes after the last lane came up, listing the first-start items its preview had triaged. A batch record for the eight clones, written about six minutes after the last clone, naming each copy and its source. A lesson about a host whose HTTPS sites answer only on its private address. A note on contractor address ranges. Three records from the staging session, one a lesson describing token-gated tracers dropped into the must-use plugins folder.

All six that read badly matched a record. The plugin uploads were the action watch’s first run replaying a backlog older than the install, from addresses the contractor note had placed. The two password changes were the clone script setting the admin on two fresh copies in the batch record. The tripwire hits matched the tracers the lesson describes. The privileged-command notices were the clone session’s own probes. The rootkit hits were the staging session’s test files. The production logins were the operator checking the clone sources.

What a session writes to permanent memory before it closes was the cheapest triage the desk ran that weekend, one query and one read.

What the records did not predict

Three items no record had predicted.

A page-drift lane on one host returned status zero for every page it had seeded, about a hundred sites’ worth, while its sibling hosts returned pages. The lane was blind, not the sites down; its stored baselines were hashes of empty bodies. No record had predicted that; the private-address lesson from the night before, written about a different check, explained it.

A shared administrator had logged in to a production site from a residential connection in Southeast Asia that no record and no teammate had claimed.

The one mail that arrived from the first vulnerability inventory on one host was a notice that the lane had hit its own cap of thirty mails an hour and suppressed the rest. As far as I could tell that morning, the findings existed only in the host’s journal.

What changed

That order is now a row in the triage table the pass reads: when a host or site’s owning workspace has records written since the alert window opened, read them first. Three more rows carry the shapes: a lane’s first run after an install, a page-drift lane reporting status zero everywhere, and the signature a clone batch leaves. Two lane rules came with them. A lane’s first run after an install is its loudest and least informative. A cap-hit notice is a pointer to the journal, so the journal read gets carded, not the notice.

The honest close

The count is approximate. About seventy of the ninety-one matched a record; I did not count the matches per cluster. The rest, beyond the three findings, went to noise cards and existing threads.

The method depends on the sessions writing. Saturday’s wrote as they went; the install and clone records landed within minutes of their last command. A session that closes without writing would hand the desk the same storm and nothing to clear it with. The method also needs memory queryable by scope and time, and this run had to build that query tool first.

The three findings were worked that evening by the operator and the owning workspaces, not by the pass. The blind lane was pointed at the address the host answers on and re-seeded; a hundred and one sites, no failed fetches. The login was claimed by a teammate. The operator then decided against the fix the desk had carried since the morning before. The shared logins stay for now, tracked by the ledger, and per-person accounts wait. The inventory lane now sends one digest per host per run, not one mail per site. The tracers matched that morning were also confirmed gone from staging and production.

That third read was incomplete. Four days later the same pass found a pointer line in an hourly digest for an inventory alert that never arrived, and a mail-store search showed no inventory mail from any of those hosts in two weeks, cap or no cap. Whether the cap path or the mail routing is eating them is an open card.

The pass does not touch the hosts; that rule predates the weekend.