An early-warning gauge for the semiconductor cycle

MVP  How the lean gauge runs — build spec

A monthly, resumable pipeline of five steps. A few agents, a fixed source list, and one SQLite file that every agent writes to as it works — so you can watch the run fill in live. This panel is the spec; it is detailed enough to hand to another agent and have it build the thing.

The five steps

Architecture at a glance

# repo layout — the source + extraction library
gauge/
  run.py                # orchestrator (python -m gauge.run)
  db/ schema.sql conn.py gauge.db
  sources/
    registry.yaml       # seeds the `sources` table
    base.py             # Extractor ABC + Observation
    capital_intent/  market_tracker/
    power_queue/     equipment/  target/
  extractors/ normalize.py  llm.py
  pipeline/ gather.py extract.py reconcile.py
            compute.py publish.py
  progress/ app.py  progress.html
  templates/ gauge.html.j2
  charts/  raw/

Agents write to the DB as they progress (real-time)

run_steps is the single source of truth for progress. Each step sets its row to running before work and done/error after, with a live rows_written count and a per-source sub-row. A tiny Flask view (or a polled progress.json) renders the step tree, updating every 3s while the run executes.

-- the live progress feed (DDL excerpt)
CREATE TABLE run_steps (
  run_id  TEXT, step TEXT,
  unit    TEXT DEFAULT '-',   -- source_id or '-'
  agent   TEXT,
  status  TEXT,               -- pending|running|done|error
  started_at TEXT, ended_at TEXT,
  rows_written INTEGER DEFAULT 0,
  message TEXT,
  PRIMARY KEY (run_id, step, unit)
);

// GET /api/progress?run_id=2026-06
{
  "run": {"status":"running","current_step":"extract"},
  "steps": [
    {"step":"gather","unit":"-","status":"done","rows_written":12},
    {"step":"gather","unit":"knight_frank","status":"done"},
    {"step":"extract","unit":"ercot_queue","status":"running"},
    {"step":"compute","unit":"-","status":"pending"}
  ]
}

Full schema (9 tables: runs · run_steps · sources · raw_documents · observations · series · metrics · regime_calls · review_queue) is in the spec. The regime_calls table is the seed of the track record — one logged call per run.

Step contracts

Resumable: a step is skipped if its run_steps row is done for the run. --only <source> re-runs one source; --resume <run_id> continues a crashed run.

The source & extraction library

Every source implements one interface, so the extraction agent never branches per source. Two kinds behind it: structured (FMP API / CSV / HTML table → deterministic) and document (PDF / press release → schema-locked LLM extraction that must keep the source snippet). Add a source = add a registry row + a parser class.

# sources/base.py — the one interface
@dataclass
class Observation:
    metric: str      # operational_capacity | capex | ...
    region: str
    basis: str       # it_load | grid_demand | capex_usd ...
    period: str      # 2025 | 2025Q4 | 2026-05
    value: float; unit: str       # GW | USD_bn
    confidence: str  # verified|strongly|weakly
    snippet: str     # exact source text (audit)

class Extractor:
    source_id: str; family: str
    def fetch(self) -> RawDoc: ...
    def parse(self, raw) -> list[Observation]: ...

Compute definitions & run

# run it
python -m gauge.run                # current month
python -m gauge.run --month 2026-06
python -m gauge.run --only ercot_queue
python -m gauge.run --resume 2026-06
python -m gauge.progress           # live view :8090

# build order: schema → conn → base+registry
# → 2 FMP extractors (nvda, msft) → gather
# → extract → reconcile → compute → publish
# → progress view → remaining extractors

MVP  The one-page gauge

Everything the MVP ships fits on one screen: the verdict, the chart behind it, which signals it watched, and whether the lead holds. Example run, June 6 2026.

Build-rate momentumcharts/regime_gauge.png

Which signals it watchedcharts/indicator_ladder.png

Does the build lead chips?charts/leadlag.png

Track record — past forecasts vs what landedcharts/realized_vs_forecast.png

The 2025 forecast (55.6 GW) came in low — actual was 60.0 GW. Forecasts have run ~5–8% below realized, so the gauge tags confidence medium and treats the current call as, if anything, conservative. This is what earns the regime call the right to be believed.

This run's read. Net additions are running ~4.2 GW/quarter and still climbing; every leading layer (power-queue requests, hyperscaler capex, construction starts) is rising in step. The build rate has historically led NVDA data-center revenue by ~5 quarters, so the signal points to continued AI-chip demand strength into 2027. Confidence is medium — the AI-era history is short, so treat the lead as directional, not precise.

Expansive  The full leading-indicator engine

The same idea, taken all the way: every layer of the indicator stack, global coverage, down to project-level stage tracking — with backtested lead/lag per indicator, a GW→dollar bridge, scenario forecasts, and a defensible audit trail. This turns the gauge from a read into a forecasting engine with attribution.

The full indicator stackcharts/indicator_ladder.png

GW → chip-$ bridgecharts/nowcast_bridge.png

Why expand. The aggregate trackers the MVP relies on update slowly and are pre-digested. Project-level tracking catches a slowdown earlier — as stalls, slipping RFS dates, and queue withdrawals — and it can attribute the move to a layer. The cost is real: a continuous multi-agent pipeline and a data layer that only gets accurate as it ages.

Expansive  How the full engine runs

A continuous, resumable, Codex-driven multi-agent pipeline writing to a database. Charts, the API, and the regime call regenerate on every run; the project/event/stage layer accumulates over time.

Source families

Cadence: continuous / weekly, fully resumable so a run can be killed and resumed without losing the data store. Cost scales with the source-family fan-out; the QA queues keep a human in the loop where confidence is low.

Expansive  The full chart pack + forecasting engine

The complete output: the levels and build rate, the forecast-vs-actual evidence, the leading-indicator panel with attribution, and the GW→$ nowcast. Example run, June 6 2026.

Build-rate momentumcharts/regime_gauge.png

Capacity over timecharts/trend.png

Build pipeline by regioncharts/pipeline.png

Forecast vs actual (by vintage)charts/forecast_revisions.png

Where capacity leaks outcharts/build_funnel.png

Track record — past forecast vs realized

The scoreboard. Third-party forecasts vs what landed, and — the gold standard — the gauge grading its own past calls. Both feed the confidence bands on every forward number.

Third-party: forecast vs realizedcharts/realized_vs_forecast.png

The gauge grading itselfcharts/forecast_track_record.png

*current estimate, not yet fully realized. The project-level version of this is RFS schedule adherence (forecast RFS date vs actual) — see the spec tiles below.

Does the build lead chips?charts/leadlag.png

GW → chip-$ nowcastcharts/nowcast_bridge.png

The leading-indicator panelcharts/indicator_ladder.png

Per-indicator lead/lag & current reading

Charts still on the spec (need the stage layer)

This run's read. Every layer of the stack is rising in step, and the earliest layer — power-interconnection requests — flashed first, which is the most leading signal we have. The build rate leads NVDA data-center revenue by ~5 quarters, and the GW→$ bridge says chip dollars rise faster than GW because intensity is climbing. Net: AI-chip demand strength looks well-supported into 2027, with the usual caveat that scheduled activations carry delay/cancellation risk concentrated in the steep 2027 slate.