Wholesale power-price analysis pipeline, North America

For every constraint the memo had to answer three questions in numbers: which price node moves, in which direction and by how much; whether the cause is load or thermal limits, a transmission outage, renewables, a neighbouring corridor or a border flow; and whether the day-ahead market already knew. What the day-ahead market has already priced is not an edge. Causes are classified in a table of six causes (the last being "no supporting cause") against three day-ahead columns, and "no supporting cause" is not a default: it is what remains after the other causes were tested and all failed.

The reference answers are the traders' past memos, written by hand. On August 11 I surveyed all of them: 1,482 memos and 4,853 cause lines.

The sample was small. Hours bound per constraint have a median of 7, and 0 of 192 exceed 100 hours. On August 28 I diagnosed that regression, ML and hierarchical models were out, leaving only deterministic decomposition, which needs no sample.

What I decided is recorded in the documents with dates. Outage history never trusts an old cache and is fetched fresh (Aug 17). The analysis starts only once the required inputs are in (Aug 23). Reference memos are not read by default (Aug 24). The definition of critical and the scoring rule (Sep 2). The pipeline is fixed on small windows first (Sep 7). And three on September 14: take from the client DB first, adopt experiment 16, which switched an input to the DB, and open the holdout once before merging hypothesis 04, then roll it back.

I built four things. The analysis pipeline: from one entry point, a day's collection and 73 calls, two LLM review steps (analyse, refute), an independent recomputation, 21 gate checks and a publish stamp, ending in a CSV. The memo, whose lint fails raw decimal notation. The hypothesis harness, which scores six memo segments separately. And a separate repository holding the pipeline for service migration: its README lists the pipeline at 15 steps, and it uses 89 constraint profiles as a regression baseline. The service itself was not started.

I also set the working rules. Only a person signs in or authenticates, the pipeline never writes into the system's memo field and only produces output, and fixes start on 4–10-case windows. These rules are written down as standing instructions.

On August 17 I listed and ran the steps by hand, and two CSVs that had not passed the gate were published. The rule was in a document but nothing checked it. I replaced it with the publish stamp and a two-way step-table check, and split the skill document, which had grown to 1,523 lines, into run order, rules with checks, and an incident log.

Things have passed while hollow. All 233 binding days were "no supporting cause" and the check still passed, the linter failed to load and reported "0 violations", and a travel-leg feed missing 5 days kept the same file name and flipped a threshold verdict. That led to the input-contract checks and the rule that a tool's silence is not the data's silence.

Explanations I wrote without checking were wrong twice in a day. I then set a rule to write down the whole state before answering. Running a big window (69 cases) repeatedly and being cut off by a limit led to the rule of fixing on small windows first.

Some things are still open: 8 open hypotheses, the decision on raising the wrong-answer cost to 2, critical recall for outages that stays at 9% on the production path, and experiment 49 with a dev net gain of 0.