# Data-Center Build Gauge — MVP Build Spec (hand-to-an-agent)

You are building the **MVP** of a gauge that measures how fast data centers are being built and turns
that into an early-warning read on the semiconductor cycle. This document is complete enough to
implement end-to-end without further questions. Build exactly this; do not expand scope.

---

## 0. Definition of done

- `python -m gauge.run` performs one full run: gather → extract → reconcile → compute → publish.
- Every agent **writes its progress to the database as it goes**, so a live progress view updates in
  real time during the run.
- Sources and their parsers live in an **organized, extensible library** (a registry + one adapter per
  source). Adding a source = add a registry row + a parser class. No core code changes.
- Output is a one-page `gauge.html` rendered from the DB: regime call, build-rate chart, indicator
  ladder, lead/lag check, track-record chart, one-paragraph read.
- Re-running the same month is idempotent (no double counting). A killed run resumes via `--resume`.

Out of scope for MVP (these are the Expansive build): per-project database, permit scraping,
satellite, cohort stage-tracking, global long-tail markets.

---

## 1. Tech stack

- Python 3.11. **SQLite in WAL mode** as the data store (single file, inspectable, supports concurrent
  reads while agents write — this is what makes real-time progress trivial).
- `httpx` (fetch), `pdfplumber` (PDF tables), `selectolax` (HTML), `pandas` (tabular), `matplotlib`
  (charts), `pyyaml` (registry), `jinja2` (dashboard templating).
- FMP API for structured financials (key in agent env). Optional Claude call for messy PDF/press
  extraction — always constrained to a strict JSON schema and always keeping the source snippet.
- Progress view: a ~40-line Flask app (`gauge.progress`) that reads the DB, **or** the orchestrator
  writes `progress.json` after every step update and a static `progress.html` polls it. Either is fine.

---

## 2. Repository layout (the source + extraction library)

```
gauge/
  __init__.py
  run.py                 # orchestrator / entrypoint (python -m gauge.run)
  db/
    schema.sql           # all DDL (section 3)
    conn.py              # connect(), WAL pragma, helpers
    gauge.db             # the SQLite file (gitignored)
  sources/
    registry.yaml        # canonical source list -> seeds the `sources` table (section 4)
    base.py              # Extractor ABC + Observation dataclass (section 5)
    capital_intent/      # hyperscaler capex extractors
      msft.py  amzn.py  googl.py  meta.py  orcl.py
    market_tracker/      # knight_frank.py jll.py cushman.py sightline.py goldman.py
    power_queue/         # ercot.py  pjm.py  dominion.py
    equipment/           # vertiv.py  eaton.py
    target/              # nvda_dc.py  (the validation target)
  extractors/
    normalize.py         # basis enum, unit->GW, region canonicalization, confidence rubric
    llm.py               # constrained Claude extraction helper (schema-locked)
  pipeline/
    gather.py  extract.py  reconcile.py  compute.py  publish.py
  progress/
    app.py               # Flask progress view (reads runs + run_steps)
    progress.html        # live step tree (polls /api/progress every 3s)
  templates/
    gauge.html.j2        # output dashboard template (rendered from DB)
  charts/                # rendered PNGs (regime_gauge, indicator_ladder, leadlag, track_record)
  raw/                   # raw_documents content store (hashed files)
tests/
  test_smoke.py  test_idempotent.py  test_schema.py
```

Adding a source later: drop a `<family>/<name>.py` adapter implementing `Extractor`, add a row to
`registry.yaml`. The loader (`sources/__init__.py`) maps `extractor:` → class by dotted path.

---

## 3. Database schema (`gauge/db/schema.sql`)

WAL + foreign keys on. Every table that an agent writes carries `run_id` so progress and provenance
are queryable.

```sql
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;

-- one row per pipeline run
CREATE TABLE IF NOT EXISTS runs (
  run_id        TEXT PRIMARY KEY,           -- e.g. 2026-06 or 2026-06-06T10:00
  started_at    TEXT NOT NULL,
  ended_at      TEXT,
  status        TEXT NOT NULL,              -- running | done | error
  current_step  TEXT,
  version       TEXT,                       -- code/spec version
  note          TEXT
);

-- live progress feed: the orchestrator and agents upsert here as they work
CREATE TABLE IF NOT EXISTS run_steps (
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  step          TEXT NOT NULL,              -- gather | extract | reconcile | compute | publish
  unit          TEXT NOT NULL DEFAULT '-',  -- sub-item, e.g. a source_id; '-' for whole step
  agent         TEXT,
  status        TEXT NOT NULL,              -- pending | running | done | error | skipped
  started_at    TEXT,
  ended_at      TEXT,
  rows_written  INTEGER DEFAULT 0,
  message       TEXT,
  PRIMARY KEY (run_id, step, unit)
);

-- organized source registry (seeded from registry.yaml; one row per source)
CREATE TABLE IF NOT EXISTS sources (
  source_id     TEXT PRIMARY KEY,
  name          TEXT NOT NULL,
  family        TEXT NOT NULL,              -- capital_intent | market_tracker | power_queue | equipment | target
  publisher     TEXT,
  url           TEXT,
  basis         TEXT,                       -- it_load | facility_power | grid_demand | capex_usd | revenue_usd
  region        TEXT,                       -- World_ex_CRO | United_States | ERCOT | ...
  cadence       TEXT,                       -- quarterly | semiannual | ongoing
  lead_quarters REAL,                       -- typical lead vs chip purchase (negative = leads)
  extractor     TEXT NOT NULL,              -- dotted path to Extractor subclass
  enabled       INTEGER NOT NULL DEFAULT 1,
  last_fetched_at TEXT,
  last_status   TEXT
);

-- raw captured documents (audit + re-extraction without re-fetch)
CREATE TABLE IF NOT EXISTS raw_documents (
  doc_id        TEXT PRIMARY KEY,           -- sha256(source_id + url + run_id)
  source_id     TEXT NOT NULL REFERENCES sources(source_id),
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  url           TEXT,
  fetched_at    TEXT NOT NULL,
  content_hash  TEXT,                       -- sha256 of body; dedupe across runs
  storage_path  TEXT,                       -- gauge/raw/<hash>.<ext>
  http_status   INTEGER
);

-- normalized extracted figures (staging output of the extraction agent)
CREATE TABLE IF NOT EXISTS observations (
  obs_id        TEXT PRIMARY KEY,
  doc_id        TEXT REFERENCES raw_documents(doc_id),
  source_id     TEXT NOT NULL REFERENCES sources(source_id),
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  metric        TEXT NOT NULL,              -- operational_capacity | under_construction | planned | capex | dc_revenue | book_to_bill
  region        TEXT,
  basis         TEXT NOT NULL,              -- it_load | facility_power | grid_demand | capex_usd | revenue_usd
  period        TEXT NOT NULL,              -- 2025 | 2025Q4 | 2026-05
  value         REAL,                       -- in canonical unit for the basis (GW or USD bn)
  unit          TEXT NOT NULL,              -- GW | USD_bn
  confidence    TEXT NOT NULL,              -- verified | strongly_inferred | weakly_inferred
  extractor     TEXT,                       -- structured | llm
  snippet       TEXT,                       -- source text the value came from (audit)
  captured_at   TEXT NOT NULL
);

-- promoted, reconciled series the compute step uses (one value per metric/region/basis/period)
CREATE TABLE IF NOT EXISTS series (
  metric        TEXT NOT NULL,
  region        TEXT NOT NULL,
  basis         TEXT NOT NULL,
  period        TEXT NOT NULL,
  value         REAL,
  unit          TEXT NOT NULL,
  source_id     TEXT,                       -- the winning source
  method        TEXT,                       -- how it was picked (e.g. "best-confidence same-basis")
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  PRIMARY KEY (metric, region, basis, period)
);

-- computed scalar/array outputs (build rate, acceleration, regime, lead/lag)
CREATE TABLE IF NOT EXISTS metrics (
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  key           TEXT NOT NULL,              -- build_rate_gw_q | yoy_pct | acceleration | lead_quarters | leadlag_corr
  period        TEXT,                       -- nullable for scalars
  value         REAL,
  unit          TEXT,
  computed_at   TEXT NOT NULL,
  PRIMARY KEY (run_id, key, period)
);

-- the call log: one regime call per run (this is the track-record seed)
CREATE TABLE IF NOT EXISTS regime_calls (
  run_id        TEXT PRIMARY KEY REFERENCES runs(run_id),
  as_of         TEXT NOT NULL,
  regime        TEXT NOT NULL,              -- accelerating | steady | decelerating
  build_rate_gw_q REAL,
  yoy_pct       REAL,
  lead_quarters REAL,
  confidence    TEXT,                       -- low | medium | high
  note          TEXT
);

-- low-confidence / conflicting rows for human review
CREATE TABLE IF NOT EXISTS review_queue (
  rq_id         TEXT PRIMARY KEY,
  run_id        TEXT NOT NULL REFERENCES runs(run_id),
  obs_id        TEXT REFERENCES observations(obs_id),
  issue         TEXT,                       -- conflict | low_confidence | stale | parse_failed
  detail        TEXT,
  created_at    TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS ix_obs_run    ON observations(run_id);
CREATE INDEX IF NOT EXISTS ix_steps_run  ON run_steps(run_id);
CREATE INDEX IF NOT EXISTS ix_raw_hash   ON raw_documents(content_hash);
```

---

## 4. Source registry (`gauge/sources/registry.yaml`)

Seed `sources` from this on every run (upsert). These 12 are the MVP fixed list. `lead_quarters` is
negative when the signal leads the chip purchase.

```yaml
- {source_id: msft_capex,   name: "Microsoft capex",      family: capital_intent, basis: capex_usd,    region: United_States, cadence: quarterly,  lead_quarters: -10, extractor: sources.capital_intent.msft.MsftCapex}
- {source_id: amzn_capex,   name: "Amazon capex",         family: capital_intent, basis: capex_usd,    region: United_States, cadence: quarterly,  lead_quarters: -10, extractor: sources.capital_intent.amzn.AmznCapex}
- {source_id: googl_capex,  name: "Alphabet capex",       family: capital_intent, basis: capex_usd,    region: United_States, cadence: quarterly,  lead_quarters: -10, extractor: sources.capital_intent.googl.GooglCapex}
- {source_id: meta_capex,   name: "Meta capex",           family: capital_intent, basis: capex_usd,    region: United_States, cadence: quarterly,  lead_quarters: -10, extractor: sources.capital_intent.meta.MetaCapex}
- {source_id: orcl_capex,   name: "Oracle capex",         family: capital_intent, basis: capex_usd,    region: United_States, cadence: quarterly,  lead_quarters: -10, extractor: sources.capital_intent.orcl.OrclCapex}
- {source_id: knight_frank, name: "Knight Frank DC forecast", family: market_tracker, basis: it_load,  region: World_ex_CRO,  cadence: semiannual, lead_quarters: 0,   extractor: sources.market_tracker.knight_frank.KnightFrank}
- {source_id: jll,          name: "JLL NA DC report",     family: market_tracker, basis: it_load,      region: North_America, cadence: semiannual, lead_quarters: -2,  extractor: sources.market_tracker.jll.JLL}
- {source_id: sightline,    name: "Sightline Climate",    family: market_tracker, basis: it_load,      region: World_ex_CRO,  cadence: ongoing,    lead_quarters: -6,  extractor: sources.market_tracker.sightline.Sightline}
- {source_id: goldman_us,   name: "Goldman US activations", family: market_tracker, basis: it_load,    region: United_States, cadence: ongoing,    lead_quarters: -4,  extractor: sources.market_tracker.goldman.GoldmanUS}
- {source_id: ercot_queue,  name: "ERCOT large-load queue", family: power_queue,  basis: grid_demand,  region: ERCOT,         cadence: quarterly,  lead_quarters: -11, extractor: sources.power_queue.ercot.Ercot}
- {source_id: pjm_queue,    name: "PJM load forecast",    family: power_queue,    basis: grid_demand,  region: PJM,           cadence: quarterly,  lead_quarters: -11, extractor: sources.power_queue.pjm.Pjm}
- {source_id: vertiv_b2b,   name: "Vertiv book-to-bill",  family: equipment,      basis: grid_demand,  region: Global,        cadence: quarterly,  lead_quarters: -4,  extractor: sources.equipment.vertiv.Vertiv}
- {source_id: nvda_dc,      name: "NVDA data-center revenue", family: target,      basis: revenue_usd,  region: Global,        cadence: quarterly,  lead_quarters: 2,   extractor: sources.target.nvda_dc.NvdaDc}
```

Prefer structured pulls where possible: `msft/amzn/googl/meta/orcl` capex and `nvda_dc` revenue come
from **FMP** (`statements`, `company` endpoints) — deterministic, no LLM. KF/JLL/Sightline/Goldman are
PDF/article → LLM extraction. ERCOT/PJM are PDF/XLSX → table parse. Vertiv is the earnings transcript
(FMP `earningsTranscript`) → LLM extraction of the book-to-bill remark.

---

## 5. Extractor library (`gauge/sources/base.py`)

Every source implements one interface. The extraction agent never special-cases a source.

```python
from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class RawDoc:
    url: str
    body: bytes
    content_type: str
    http_status: int

@dataclass
class Observation:
    metric: str          # operational_capacity | under_construction | planned | capex | dc_revenue | book_to_bill
    region: str
    basis: str           # it_load | facility_power | grid_demand | capex_usd | revenue_usd
    period: str          # 2025 | 2025Q4 | 2026-05
    value: float
    unit: str            # GW | USD_bn
    confidence: str      # verified | strongly_inferred | weakly_inferred
    snippet: str         # the exact source text the number came from (audit trail)

class Extractor:
    source_id: str
    family: str
    def fetch(self) -> RawDoc: ...                    # network only; deterministic
    def parse(self, raw: RawDoc) -> list[Observation]: ...   # raw -> normalized observations

def now() -> str:
    return datetime.now(timezone.utc).isoformat(timespec="seconds")
```

`extractors/normalize.py` provides: `to_gw(value, unit, basis)`, `canon_region(s)`, and a confidence
rubric (`verified` = official primary disclosure; `strongly_inferred` = reputable tracker;
`weakly_inferred` = single weak source / model). LLM extraction (`extractors/llm.py`) must return JSON
matching the `Observation` fields and must include the verbatim `snippet`; anything it can't ground in a
snippet is dropped to `review_queue` with `issue = parse_failed`.

Two extractor kinds, both behind the same interface:
- **structured** — CSV / FMP API / HTML table / XLSX → deterministic parse, `extractor='structured'`.
- **document** — PDF / press release → `extractors/llm.py` with a schema lock, `extractor='llm'`.

---

## 6. Pipeline & agent contracts (the runbook proper)

The orchestrator (`run.py`) owns the `runs` row and the `run_steps` feed. Each step updates
`run_steps` to `running` before work and `done`/`error` after, with `rows_written` and a `message`.
This is the contract that makes progress visible live.

```python
# run.py core loop (pseudocode)
run_id = ensure_run(month)                       # insert/resume runs row, status=running
seed_sources_from_registry()                     # upsert sources
for step in ["gather","extract","reconcile","compute","publish"]:
    if step_done(run_id, step): continue         # resume support
    mark(run_id, step, "running")
    try:
        n = STEP_FN[step](run_id)                 # the step writes its own per-unit run_steps rows too
        mark(run_id, step, "done", rows=n)
    except Exception as e:
        mark(run_id, step, "error", message=str(e)); set_run_status(run_id,"error"); raise
set_run_status(run_id, "done")
```

**Step contracts** — each is an agent or a deterministic function; inputs are DB queries, outputs are DB writes:

| Step | Agent | Reads | Does | Writes | Exit when |
|---|---|---|---|---|---|
| 1 gather | Gather agent | `sources` (enabled) | For each source: `fetch()` → store body to `gauge/raw/<hash>` and a `raw_documents` row; update `sources.last_*`; write a `run_steps(step='gather', unit=source_id)` row per source | `raw_documents`, `run_steps` | every enabled source has a raw_doc or an `error` row |
| 2 extract | Extraction agent | `raw_documents` for this run | For each doc: load adapter, `parse()` → `Observation[]`; normalize to GW/USD_bn; insert `observations`; ungroundable values → `review_queue` | `observations`, `review_queue`, `run_steps(unit=source_id)` | all docs parsed or queued |
| 3 reconcile | Reconcile fn | `observations` (+ prior `series`) | Promote one value per (metric, region, basis, period): pick best confidence, never average across bases; flag conflicts to `review_queue` | `series` | series populated for required metrics |
| 4 compute | Compute agent | `series` | build_rate_gw_q, trailing-4q avg, YoY %, acceleration, regime; lead/lag corr vs `nvda_dc`; write the run's `regime_calls` row | `metrics`, `regime_calls` | regime + lead/lag present |
| 5 publish | Publish fn | `metrics`, `series`, `regime_calls`, prior `regime_calls` | Render charts (regime_gauge, indicator_ladder, leadlag, track_record) and `gauge.html` from the template | `charts/*.png`, `gauge.html` | files written |

Resumability: a step is skipped if its `run_steps` row is `done` for this `run_id`. `--only <source_id>`
re-runs gather+extract for one source. `--resume <run_id>` continues a crashed run.

---

## 7. Real-time progress

`run_steps` is the single source of truth; WAL lets the progress view read mid-write. The view is a
step tree grouped by `step`, with per-source `unit` rows underneath, each showing status icon, elapsed,
and `rows_written`.

`gauge.progress` (Flask) exposes `GET /api/progress?run_id=` returning:

```json
{
  "run": {"run_id":"2026-06","status":"running","current_step":"extract","started_at":"..."},
  "steps": [
    {"step":"gather","unit":"-","status":"done","rows_written":12,"elapsed_s":31},
    {"step":"gather","unit":"knight_frank","status":"done","rows_written":1},
    {"step":"extract","unit":"ercot_queue","status":"running","rows_written":0},
    {"step":"compute","unit":"-","status":"pending"}
  ]
}
```

`progress/progress.html` polls this every 3s and re-renders the tree (green=done, blue pulsing=running,
grey=pending, red=error). If you don't want a server: the orchestrator writes the same JSON to
`progress/progress.json` after every `mark()` and `progress.html` polls the static file. Either way you
watch the run fill in live.

---

## 8. Compute definitions (no ambiguity)

- **build_rate_gw_q** = quarterly net additions of the promoted `operational_capacity / World_ex_CRO /
  it_load` series. If only annual points exist, distribute across quarters with a monotone spline and
  flag `confidence=weakly_inferred`. Also compute for `United_States`.
- **trailing4q** = 4-quarter trailing average of build_rate_gw_q.
- **yoy_pct** = (trailing4q[t] − trailing4q[t−4]) / trailing4q[t−4].
- **acceleration** = trailing4q[t] − trailing4q[t−1].
- **regime**: `accelerating` if yoy_pct > +10%; `decelerating` if yoy_pct < −10%; else `steady`.
  (Thresholds in `gauge/config.py`; tune after first backtest.)
- **lead/lag**: cross-correlate build_rate_gw_q against `nvda_dc` dc_revenue over all overlapping
  quarters; report `lead_quarters = argmax_lag(corr)` and `leadlag_corr = max corr`. With < 8 overlapping
  quarters, set `confidence=low` and label the chart "illustrative — short history".
- **confidence (regime_call)**: `high` if ≥12 quarters history and sources agree; `medium` if 6–11 or
  minor disagreement; `low` otherwise.

---

## 9. Run & schedule

```
python -m gauge.run                 # current month
python -m gauge.run --month 2026-06
python -m gauge.run --only ercot_queue
python -m gauge.run --resume 2026-06
python -m gauge.progress            # serve the live progress view on :8090
```

Schedule monthly (cron / the routine system). One run ≈ 12 fetches + ~7 LLM extractions: cheap.

---

## 10. Acceptance tests

- **test_schema** — `schema.sql` applies clean; all tables/indexes exist.
- **test_smoke** — a run with 2 stub sources completes; `runs.status='done'`; `regime_calls` has 1 row;
  `gauge.html` is written.
- **test_idempotent** — running the same month twice yields the same `series` (no duplicate promotion);
  `observations` may grow but `series` upserts.
- **test_progress** — during a (slowed) run, `/api/progress` shows at least one `running` step, then all
  `done`.

Build in this order: db/schema → conn → sources/base + registry + 2 structured extractors (nvda_dc via
FMP, msft_capex via FMP) → gather → extract → reconcile → compute → publish → progress view → remaining
extractors. You can demo end-to-end after the first two extractors.