home - read-only record of what was built, what did not go as planned, and which sources worked.
# GridArb build log
Public, read-only record of what was built, what did not go as planned, and which data sources
worked. Served at /build-log. No secrets are ever written here.
## 2026-10-12: Phases Q through V (consultant's-report features + feeds, alerts, statements)
Live at https://gridarb-api-production.up.railway.app
### What is live per phase
- **Q, exact site node.** A catalog of 14,070 PJM pricing-node names (one-hour snapshot of the
day-ahead dataset), a typeahead on the main page, and on-demand node price ingest
(`POST /admin/nodes`) that loads a node's 2024-to-date day-ahead and real-time history under
a budget guard and logs every row. Node or zone is accepted by the backtest, revenue stack,
report ("Node: X (zone PSEG, as entered)"), schedule and forecaster; node gaps fall back to
zone prices and are listed. Loaded: `NEWARK 26 KV LD1XFR`, a PSE&G 26 kV load bus:
$18,856 on 2025 day-ahead prices for 1 MW / 2 MWh against $18,495 for the PSEG zone.
- **R, grid-connection desk check.** Report section and `/api/gridcheck` with the site from
the US Census Geocoder, nearest substations from OpenStreetMap Overpass, circuit headroom and
nearby queued MW as manual fields with the utilities' and PJM's public URLs, connection type,
LDA and the standing warning.
- **S, fifteen-year view.** Capacity known through 2028/29, 2029/30 collar-bounded, then low
$175 / base $250 / high $500 per MW-day with ELCC sliding to 0.40 by 2035/36; arbitrage flat /
+5% / +10% per year (industry-outlook assumption); fade and augmentation; totals low <= base
<= high; the base path feeds the NPV. Default 1 MW / 2 MWh: fifteen-year net before capex
-$234k low, -$71k base, +$435k high.
- **T, weather + PJM feeds, forecaster v2.** Open-Meteo archived model forecasts (97,440 hourly
rows for four zone points, 2024-01-01 to yesterday) and live 48 h forecasts with issue time;
PJM daily outages (14,204 rows, all publishes, two regions); PJM hourly load forecast (one
15:45 UTC publish per day, catching up at up to 190 requests a day). Feature set v2 = v1 +
weather, degree-days, load forecast, outages, all as of 14:00 ET the day before, with the
no-look-ahead test extended to the feeds. The daily job falls back to v1 and flags the
schedule when a feed is missing. Results below.
- **U, alerts.** Per-owner price threshold (default $300/MWh, editable), be-full-by, PJM
emergency postings, job failures and missing feeds; owner-page bell with timestamps; email via
Resend only when a key is set (not set, by instruction); admin alert address set; duplicates
never re-created; a drill endpoint exercised the job-failure path on the Test Owner profile.
- **V, monthly statement.** `/owner/<token>/statement?month=YYYY-MM`: earned vs ceiling, DA-only
and fixed schedules, % of perfect, best and worst five days with one-line reasons, cycles vs
allowed, days delivered, scoreboard summary, other revenue not operated by GridArb; print CSS
and a Download PDF button; auto-generated on the 1st for every profile; the Test Owner's
partial October 2026 statement renders.
### Forecaster v1 (prices only) vs v2 (weather + PJM feeds), 2025 unseen year, 1 MW / 2 MWh
Money captured as % of the perfect-foresight real-time ceiling; DA-only and naive for context.
Verdict: v2 is NOT better on money (four-zone total 43.0% of perfect, $71,297, against 44.3%,
$73,322 for v1). MAE improved slightly in three zones and the top-4 hit rate barely moved. v1
stays live; v2 is kept for comparison and will be re-run once the load-forecast history is
complete.
| Zone | v1 RT-aware | v2 RT-aware | DA-only | Naive | v1 MAE | v2 MAE | v1 top-4 | v2 top-4 |
|---|---|---|---|---|---|---|---|---|
| AECO | 42.8% | 44.8% | 55.2% | 31.7% | 13.06 | 12.50 | 49% | 50% |
| JCPL | 50.1% | 47.4% | 59.0% | 30.8% | 12.81 | 12.40 | 50% | 50% |
| PSEG | 47.7% | 40.9% | 56.9% | 31.2% | 13.22 | 12.77 | 46% | 49% |
| RECO | 37.4% | 39.6% | 50.7% | 29.5% | 14.73 | 14.93 | 46% | 45% |
| **All four** | **44.3%** | **43.0%** | | | | | | |
Caveat on v2: the load-forecast history covers 110 of the 731 train/test days at the time of
the run (gridstatus caps requests at 250 per day); weather and outages are complete. The daily
job catches up the remaining load-forecast days and the comparison can be re-run with
`POST /admin/run-forecast-backtest?feature_set=v2`.
### What did not go as planned
- **gridstatus limits.** Beyond the 500,000 rows/month, the free plan enforces 600 requests per
hour and 250 requests per day. The one-request-per-day load-forecast backfill hit both. It
now paces itself, backs off on 429, stops on the daily 403, skips loaded days, and the daily
job spends leftover request budget catching up. Full load-forecast history needs about six
more days at this cap.
- **Node catalog ids** overflow a 32-bit integer (PJM pnode ids exceed 2^31); columns moved to
BigInteger after the first insert failed. Two failed snapshot inserts still cost 28k rows.
- **Hosting-capacity maps and PJM queue.** No public REST endpoint could be found for any of
the four NJ utilities' hosting-capacity maps, and PJM's queue export is not linked statically;
both are manual fields with public URLs. Overpass returned gateway errors during verification
and degraded gracefully.
- **NREL ATB.** nrel.gov was unreachable from the build environment; the OEDI data files were,
so the Phase P costs use the ATB 2025 workbook (OEDI submission 8759, downloaded 2026-10-12):
$234.01/kWh energy, $361.34/kW power, fixed O&M 4.0% of overnight cost, 2023 USD.
### Data sources
| Source | Status |
|---|---|
| gridstatus.io PJM DA/RT hourly LMPs (zones and nodes) | worked; row and request budgets tracked |
| gridstatus.io PJM load forecast (historical + live) | worked; history partial, catching up daily |
| gridstatus.io PJM daily outages | worked, complete |
| gridstatus.io PJM emergency postings | reachable; used for alerts |
| gridstatus.io PJM interconnection queue | no such dataset; manual field |
| Open-Meteo historical-forecast + forecast APIs | worked, complete |
| Open-Meteo previous-runs API | no data before 2025 for these points; not used |
| US Census Geocoder | worked |
| OpenStreetMap Overpass | live, down at verification; graceful |
| Utility hosting-capacity maps (PSE&G, JCP&L, ACE, RECO) | no public REST service found; manual field + URL |
| NREL ATB via OEDI | worked (2025 edition) |
| Resend email | not configured by instruction; owner page is the delivery |
### gridstatus rows used this month (October 2026)
303,406 of the 400,000-row self-imposed budget at the time of writing: 195,700 pre-tracking
estimate (history ingest and reruns), 42,210 node catalog snapshots, 48,652 node history,
2,640 load forecast, 14,204 outages. Live at `/api/usage`.
### Payback and NPV, default case (PSEG DA 2025, benchmark costs, with ITC)
- 1 MW / 2 MWh: capex $953,764 before ITC, $667,635 after; no payback within 15 years; NPV at
8% about -$650k.
- 1 MW / 4 MWh: capex $1,491,987 before ITC, $1,044,391 after; no payback; NPV about -$910k.
## 2026-10-12: Phase W, v3 dispatch strategy (tuned on 2024, confirmed on 2025)
Live at https://gridarb-api-production.up.railway.app (`/api/strategies`, `/track-record`,
owner pages, and the live study table at the bottom of this page).
### What was built
- **W1, five strategies as pure functions** (`app/strategies.py`): DA-only; shift-if-confident
(move a planned discharge hour to one whose p10 beats its DA price by MARGIN, a charge hour to
one whose p90 undercuts it by MARGIN); spread-gate (follow the forecast only on days whose
conservative spread, mean p10 of the top-k hours minus mean p90 of the bottom-k, beats the DA
spread by GATE); blend (optimise on w * DA + (1 - w) * p50); EV with band penalty (discharge
priced at p50 - LAMBDA * (p90 - p10), charge at p50 + LAMBDA * (p90 - p10)). Every dispatch is
settled by the shared simulator against ACTUAL real-time prices, dispatch fixed, cycle cap and
reserve / risk-hour floors enforced, SOC carried day to day per strategy.
- **W2, tuning** (`app/strategy_runs.py`): grid search per zone on 2024 only (MARGIN 0-100,
GATE 0-50 x k 2-4, w 0.5-1.0, LAMBDA 0-1), then the fixed parameters on 2025. The 2024
forecasts are out of sample (two folds: each half of 2024 predicted by a model trained on the
other half); the 2025 forecasts come from the model trained on all of 2024, as live. No
gridstatus download was made: the study reads the prices already in Postgres. 1,460
zone-days per year, 1 MW / 2 MWh, about 25 minutes on a laptop core.
- **W3, decision rule**: promote the best v3 to "recommended" only if it beats DA-only on 2025
by at least 2 percentage points of the perfect-foresight ceiling across the four zones.
- **W4, scoreboard**: the daily job now also writes a `kind="v3"` schedule (default batteries
and every owner profile, under the owner's floors and cycle cap); `/track-record`, the owner
scores and the monthly statement track recommended, DA-only and best v3 side by side.
- **W5, owner page**: a per-hour confidence hint (p50 with the p10-p90 band; "confident" under
$15 wide, "fair" under $40, "unsure" above) next to the RT forecast, and one plain-English
sentence on the recommended strategy.
- **W6, re-run hook**: `POST /admin/run-forecast-backtest?feature_set=v2&strategy=all` re-runs
the v2 forecast backtests and then the whole strategy study; the result is stored and this
page renders it from the database, so the log updates by itself when the load-forecast
history completes.
- **W7, tests**: each strategy on a toy day; reserve, cycle-cap and risk-hour floors under
deviations; tuning reads only the days it is given (never 2025); the decision rule; v3 in
the daily job, the scoreboard, the API and the owner page. 157 tests, ruff clean, CI green.
### Result: did v3 beat DA-only?
**No, not by enough to go live.** The best v3 strategy, shift-if-confident with a $30 margin
(JCPL $50), captured 52.0% of the perfect-foresight ceiling on 2025 against 51.9% for DA-only:
+0.1 points, +$44 over the year on four 1 MW / 2 MWh batteries, from 9 deviations in 1,460
zone-days. The rule needs +2.0 points, so **DA-only stays recommended** and shift-if-confident
is published as **beta** next to it, scored daily. The tuning year said the same: no v3
strategy beat DA-only on 2024 either, which is why the search settled on the most cautious
parameters it had.
### Phase W strategy study: v3 strategies tuned on 2024, confirmed on 2025, 1mw_2mwh
Feature set v1; run 2026-10-12 15:31 UTC. tuning forecasts: two-fold out-of-sample within 2024; confirmation forecasts: model trained on 2024 (hgb-v1-2024-12-31)
**Decision:** recommended = **DA-only**. Best v3 = shift-if-confident at 52.0% of perfect on 2025 vs DA-only 51.9%: gain +0.1 points; NOT promoted (promote only if best v3 beats DA-only by >= 2 points of the ceiling on 2025).
Money captured as % of the perfect-foresight RT ceiling. Parameters chosen on 2024 only.
| Strategy | Params (2024-tuned, per zone) | 2024 tune % | 2025 confirm % | 2025 $ | vs DA-only (pts) | vs v1 RT-aware (pts) | vs naive (pts) | deviation days | deviation gain $ |
|---|---|---|---|---|---|---|---|---|---|
| DA-only (recommended) | - | 52.4% | **51.9%** | $86,042 | +0.0 | +13.1 | +21.1 | 0 of 1460 | $0 |
| v1 RT-aware | - | 37.6% | **38.8%** | $64,322 | -13.1 | +0.0 | +8.0 | 1460 of 1460 | $-21,720 |
| shift-if-confident | AECO: margin=30; JCPL: margin=50; PSEG: margin=30; RECO: margin=30 | 52.4% | **52.0%** | $86,085 | +0.1 | +13.2 | +21.2 | 9 of 1460 | $44 |
| spread-gate | AECO: gate=0, k=2; JCPL: gate=0, k=2; PSEG: gate=0, k=2; RECO: gate=0, k=2 | 52.4% | **51.9%** | $86,042 | +0.0 | +13.1 | +21.1 | 0 of 1460 | $0 |
| blend | AECO: w=0.9; JCPL: w=1; PSEG: w=1; RECO: w=1 | 52.4% | **51.3%** | $85,078 | -0.6 | +12.5 | +20.5 | 117 of 1460 | $-964 |
| EV with band penalty | AECO: lam=0.1; JCPL: lam=0.2; PSEG: lam=0.1; RECO: lam=0 | 38.9% | **38.5%** | $63,849 | -13.4 | -0.3 | +7.7 | 1401 of 1460 | $-22,192 |
Per zone, 2025 confirmation (% of perfect):
| Zone | DA-only | v1 RT-aware | naive | shift | spread-gate | blend | EV band | best v3 |
|---|---|---|---|---|---|---|---|---|
| AECO | 52.1% | 38.4% | 31.7% | 52.2% | 52.1% | 49.7% | 40.8% | shift-if-confident 52.2% |
| JCPL | 56.3% | 45.2% | 30.8% | 56.1% | 56.3% | 56.3% | 42.4% | spread-gate 56.3% |
| PSEG | 53.8% | 40.7% | 31.2% | 54.0% | 53.8% | 53.8% | 39.8% | shift-if-confident 54.0% |
| RECO | 46.4% | 32.0% | 29.5% | 46.3% | 46.4% | 46.4% | 32.0% | spread-gate 46.4% |
Reading the table:
- The forecaster's bands are too wide to earn deviations. A conservative spread (p10 of the
expensive hours minus p90 of the cheap hours) never exceeded the DA spread on any 2024 day,
so spread-gate chose GATE = 0 and still never fired; it is DA-only in disguise.
- Trusting the point forecast costs money: v1 RT-aware (38.8%) and EV-with-band (38.5%) sit 13
points below DA-only. Penalising the band width does not rescue a point forecast whose shape
is wrong; LAMBDA tuned to 0-0.2.
- Blend tuned to w = 1.0 (pure DA) in three zones and 0.9 in AECO, and the AECO deviations lost
$964 on 2025.
- The 2024-tuned parameters held on 2025 (52.4% tune vs 52.0% confirm for shift), so the
procedure is honest; the forecast simply does not carry enough extra information.
- DA-only is 51.9% here against about 55% in the Phase L/M table because this harness starts
every day's plan from the nominal SOC and carries the settled SOC forward for all four zones
together; the ranking is unchanged.
What would change the answer: a sharper forecaster (v2 once the load-forecast history is
complete, re-run through the W6 hook), or intraday re-planning after the morning RT prints,
which no D-1 strategy can use.
### Data sources
No new source. Prices from the existing gridstatus history in Postgres; no rows downloaded for
this phase (usage unchanged at `/api/usage`).
## 2026-10-12: Feed history import, database incident and cleanup, v2 re-run
### Load forecasts from PJM Data Miner 2
Six "Historical Load Forecasts" CSV exports (2022, 2023, 2024 twice, 2025, 2026; about 719k rows
a year across 11 forecast areas, about ten issues per day). Mapping
(`scripts/import_load_forecast.py`): forecast_area -> area (RTO and MIDATL; PJM publishes no
AECO / JCPL / PSEG / RECO forecast in this dataset, the NJ zones sit inside MIDATL),
evaluated_at_utc -> publish_time (the issue time), forecast_hour_beginning_utc -> ts_utc,
forecast_load_mw -> mw. Only the issue a target day's features can use is kept: the last one at
or before 14:00 ET on D-1, and the last at or before 13:00 ET (the forecaster's cutoff). PJM
publishes nothing between 13:00 and 14:00 ET, so they are the same issue. Upsert on
(area, publish_time, ts_utc) dedupes overlapping files.
| File | Rows read | Kept | Dropped (not the as-of issue) | Range (UTC) |
|---|---|---|---|---|
| 2022 | 717,893 | 17,424 | 113,102 | 2022-01-01 .. 2023-01-01 |
| 2023 | 713,878 | 17,326 | 112,470 | 2023-01-01 .. 2024-01-01 |
| 2024 | 719,279 | 17,424 | 113,354 | 2024-01-01 .. 2025-01-01 |
| 2024 (duplicate export) | 719,279 | 17,424 (deduped) | 113,354 | same |
| 2025 | 562,859 | 13,632 | 88,706 | 2025-01-01 .. **2025-10-13** |
| 2026 | 562,199 | 13,632 | 88,586 | 2026-01-01 .. 2026-10-13 |
Coverage now: 363 / 361 / 363 full days for 2022 / 2023 / 2024, 284 for 2025 and 2026.
**Gap: the 2025 export stops on 2025-10-12**, so 13 October to 31 December 2025 has no load
forecast. Re-export that range from Data Miner, or the daily job's catch-up fills it from
gridstatus at one request per day from tomorrow.
### Weather and outages, 2022-2023
- Open-Meteo archived forecasts, 2022-2023: 70,080 rows (four zone points), complete.
- PJM outages, 2022-2023: not loaded yet. The two gridstatus requests (about 10k rows) hit the
250-requests/day cap; the daily job now fetches any missing whole year first when request
budget is free, so it lands on a coming run.
- **2022-2023 LMPs** come from gridstatus on 2026-11-01, when the monthly row budget resets.
Until then the forecaster trains on 2024 only, so the 2022-2023 feed history is stored but
not yet used.
### What did not go as planned: the database filled up and went down
The first import ran into the Postgres volume, then 500 MB. Postgres' default max_wal_size
(1 GB) is larger than that volume, so the sustained upserts filled the disk with write-ahead
log; the server crashed ("could not write to file pg_wal/xlogtemp: No space left on device")
and could not finish crash recovery, so the site lost its database for about half an hour. The
owner grew the volume to 5 GB in the Railway dashboard (the CLI cannot resize). Then:
- `max_wal_size` capped at 128 MB and `min_wal_size` at 32 MB (ALTER SYSTEM).
- load_forecast pruned to the as-of issue per day (391,148 -> 52,222 rows before the rest of
the import); the as-of reader returned identical data before and after on 43 sampled days.
- Recomputable data: the backtest cache was truncated (26 rows, rebuilt on demand). There were
no forecasts or models for non-live versions and only one node-catalog snapshot. The
strategy-study decision row (6 KB) and the stored forecast backtests were kept: they drive
the live beta schedule and the comparison below.
- VACUUM FULL on the pruned tables, VACUUM ANALYZE on the rest.
- `/health` now reports database, WAL and volume usage; an hourly job raises an admin alert at
70%, emailed through Resend (now configured) together with job-failure and missing-feed
alerts. A test email was delivered.
- Heavy studies (year backtests, the strategy study) now refuse to run against production from a
laptop; they run on Railway via `POST /admin/run-forecast-backtest`.
| | Before | After |
|---|---|---|
| Database `railway` | 187 MB | 130 MB |
| load_forecast (table + indexes) | 68.2 MB, 391,148 rows | 11.8 MB, 79,438 rows (complete import) |
| WAL | 224 MB | 128 MB |
| Volume (Railway's figure) | 310 MB of 500 MB, then down | 310 MB on Railway's (lagging) meter, 280 MB measured in Postgres, 5.6% of 5,000 MB |
### v2 forecaster re-run on 2025 (run on Railway), 1 MW / 2 MWh
Money captured as % of the perfect-foresight RT ceiling, model trained on 2024 (now with
complete 2024 load forecasts), tested on 2025.
Whole year, before vs after:
| Zone | v1 | v2 before (110 of 731 days of load forecast) | v2 after | v1 MAE | v2 MAE before | v2 MAE after |
|---|---|---|---|---|---|---|
| AECO | 42.8% | 44.8% | 44.2% | 13.06 | 12.50 | 18.53 |
| JCPL | 50.1% | 47.4% | 45.0% | 12.81 | 12.40 | 14.68 |
| PSEG | 47.7% | 40.9% | 43.5% | 13.22 | 12.77 | 15.87 |
| RECO | 37.4% | 39.6% | 40.7% | 14.73 | 14.93 | 17.66 |
| **All four** | **44.3%** ($73,322) | **43.0%** ($71,297) | **43.2%** ($71,651) | | | |
The whole-year MAE got worse because of the export gap: from 13 October 2025 the load-forecast
features are missing, and a model trained with them complete mispredicts badly (v2 MAE 22-38
$/MWh in that window against 12-17 for v1). The fair comparison is the window where the feed
exists:
1 January to 12 October 2025 (285 days, feeds complete):
| Zone | v1 | v2 | DA-only | v1 MAE | v2 MAE | v1 top-4 | v2 top-4 |
|---|---|---|---|---|---|---|---|
| AECO | 46.5% | 45.5% | 56.4% | 13.28 | 12.97 | 51% | 50% |
| JCPL | 53.1% | 45.6% | 61.3% | 12.89 | 12.53 | 51% | 50% |
| PSEG | 53.5% | 44.7% | 60.7% | 12.96 | 12.55 | 48% | 50% |
| RECO | 40.9% | 44.2% | 54.8% | 14.21 | 13.75 | 49% | 47% |
| **All four** | **48.3%** ($65,098) | **45.0%** ($60,614) | 58.2% | | | | |
**Verdict: v2 is still not better on money.** With complete load forecasts it is more accurate
hour by hour (lower MAE in all four zones) but captures 3.3 points less of the ceiling, so v1
stays live. Both remain well below DA-only. Re-run with the hook once the October-December 2025
gap is filled; the v2 money from 13 October onwards in the whole-year table is not meaningful.
---
## 2026-10-12 - Product redesign v2: the chat-first battery analyst (stages 1-3)
### URLs (all on https://gridarb-api-production.up.railway.app)
| URL | What |
|---|---|
| `/` | New landing page (switched last, after `/app` and the old links passed on production) |
| `/app` | The analyst: chat, sidebar, Profile / Results (Tomorrow for owners) / Activity |
| `/plans` | Plans from `app/plans.py`; Request access form |
| `/tools` | The original calculator (all old query parameters work; reports link back here) |
| `/admin`, `/admin/analyst` | Status dashboard with a new Analyst card; users, plans, AI cost, flags, questions, AI switch |
Old links checked on production after the switch: `/report`, `/schedule`, `/track-record`,
`/owner/<token>` (200 for a real token), `/build-log`, `/backtest`, `/compare`, `/api/*`,
`/docs`, `/admin` and `/admin/*` (401 without the token), `/health`.
### Keys still missing and what each unlocks
| Key | Without it (today) | With it |
|---|---|---|
| `CLERK_PUBLISHABLE_KEY`, `CLERK_SECRET_KEY` | "Sign-in opens soon - request access" (saves a lead, emails the admin). The admin cookie acts as a Pro user for testing. | Email and Google sign-in, saved sites and reports per user, the one-time role question, plan limits per account |
| `ANTHROPIC_API_KEY` | Guided mode only: one question at a time with quick replies; free-typed questions it cannot parse are logged as unanswered | Free-typed chat with Claude (Sonnet 5.5 signed in, Haiku 5.5 for anonymous visitors and short questions), streaming, AI write-ups after each workflow, all number-guarded |
| `PAYMENT_LINK_PRO`, `PAYMENT_LINK_SITE_REPORT`, `PAYMENT_LINK_OWNER` | Request access form (lead + admin email); plans set by hand on `/admin/analyst` | Subscribe / Buy open the checkout; `app/billing.py` is the place for a webhook to set plans |
| Resend verified domain | Owner plans: "Copy for my operator" and CSV only | Emailing the operator directly |
### What was built
- Stage 1 (no AI needed): landing page, plans, sign-in fallback, users table and server-side
entitlements, `/app` in guided mode, owner Tomorrow view with risk setting and Approve plan
(confirmation, operator copy, CSV), `/privacy`, `/terms`, `/contact`, delete-my-data.
- Stage 2: the Claude layer (Messages API, streaming, tool use), the plan gate, the
update_profile guard (Claude can only write values found in the user's own words), status
answers from the activity log, the number guard (one regeneration, then the engine-rendered
answer; every catch logged), per-plan message limits, the $20 monthly budget, cost per user
per day, "This looks wrong" flags, question logging.
- Stage 3: `/admin/analyst`, the Analyst card on `/admin`, the payment hook structure, a design
pass from desktop and phone screenshots, and the switch of `/`.
- Tests: 202 passing (21 new: pages and old links, anonymous daily limit, entitlements, guided
site check with source tags and engine numbers, re-run on edit, plan / skip / stop, status
questions, ownership refusals, owner plan and approval, Request access, number guard catch /
regenerate / fallback, budget, AI switch and message-limit fallbacks, AI failure fallback,
admin page, retry re-arm).
### The landing page
A centred headline, "Your battery analyst for New Jersey.", with the sub-line, the chat box as
the hero and three starter chips. Behind it a quiet band draws the real PSEG day-ahead curve once
on load, with the default battery's charge hours in teal and sell hours in amber. Below: a live
strip with the latest full day's widest NJ spread, How it works in three steps, an example answer
computed live from stored prices (PSEG 1 MW / 2 MWh on 2025 prices), the plans as a ruled list,
why the numbers can be checked, the line "GridArb gives recommendations. It does not bid into PJM
or sign anything for you.", a short FAQ, and a footer with Privacy, Terms, Contact. Public Sans
with tabular figures in paper, navy and grid blue; teal and amber appear only for charge and
sell. On a phone the header keeps Plans and Check a site, and the composer and chips stack.
### One full sample conversation (production, guided mode, real 2024-2025 prices)
> **You:** I want to build a 5 MW, 4-hour battery in South Jersey
> *(Profile: power 5 MW and duration 4 hours "from you"; zone AECO "from you" with the note
> "from 'south jersey'"; GSESP "calculated: eligible".)*
>
> **GridArb:** How will it connect to the grid? *[Transmission] [Distribution] [Behind the meter] [Not sure]*
>
> **You:** Transmission
>
> **GridArb:** Do you have a cost quote, or should I use the NREL benchmark?
>
> **You:** Use the NREL benchmark
>
> **GridArb:** Site check for 5 MW / 20 MWh: 1. Compare the four NJ zones on 2024 and 2025
> prices; 2. Look for an exact pricing node; 3. Compare 2-hour and 4-hour; 4. Add capacity,
> GSESP, ITC and the NREL cost; 5. Work out payback and NPV, run the grid desk check; 6. Run
> 15-year scenarios. *[Proceed] [Edit plan] [I'll do it myself]*
>
> **You:** Proceed *(six steps, about 3 seconds with the warmed cache)*
>
> **GridArb:** My recommendation: put the 5 MW / 20 MWh battery in AECO. It would have earned
> about $472,523 a year on 2025 prices: $122,579 from energy arbitrage and $349,944 from
> capacity. After the 30% ITC it costs about $5,221,955, so simple payback is never within 15
> years and the NPV at 8% is -$4,539,102.
> Why: RECO earns the most for 5 MW / 20 MWh: $479,923 a year on 2025 prices. Next is AECO at
> $472,523. You picked AECO; RECO would earn $7,400 a year more. 2-hour has the best NPV
> (-$3,244,045 at 8%): the extra energy costs more than the extra revenue it earns. No exact
> pricing node is loaded for AECO, so I use the AECO zone price.
> Fifteen-year net revenue before build cost: low -$1,440,319, base $4,532,459, high
> $10,997,309. Against $5,221,955 net cost, cumulative cash ends at -$6,662,274 (low),
> -$689,496 (base) and $5,775,354 (high). These are scenarios, not a forecast.
> Not modelled: interconnection upgrade costs, permits and real vendor quotes. Backtests on past
> prices are not guarantees.
> Next step: get a vendor quote and ask the utility about connection at your site, then make a
> site report. This is a recommendation; you decide.
An anonymous visitor gets the short version of the same answer: best location, rough annual
revenue, simple payback, then "Sign in free to save this site, see the 15-year scenarios and
keep going". The owner flow on production ("I own a battery: plan tomorrow. 2 MW, 2 hours in
PSEG", balanced) produced a 24-hour plan (charge 03, 12 and 22; sell 07, 18 and 19; expected
margin $244.65 at day-ahead prices) and an approved plan with operator copy and CSV.
### What did not go as planned
- **Today's daily job lost its retries.** The 19:35 UTC attempt was pending because the
gridstatus cap was spent (375 of 250 requests). Its hourly retries lived in scheduler memory,
and the deploys during this build restarted the process, so no retry ran. Fixed: on startup
the scheduler re-arms the next retry when the last attempt is still pending for tomorrow and
inside the window (`pending_retry`, tested). Until the cap resets, tomorrow's day-ahead prices
are not loaded, and the owner plan says plainly that it uses today's prices.
- **The backtest cache had been emptied** by the storage cleanup, so the first site checks would
have run live LPs. The analyst now warms the 16 default 1 MW runs on startup (about 2 seconds
each on Railway; done once, then cached).
- **The AI and Clerk paths have not run live** because the keys are not set. Both are covered by
tests: a scripted fake Claude, and stubbed Clerk token verification.
- **The 5 MW / 4-hour example does not pay back at NREL benchmark cost.** That is the engine's
real answer (ATB year-2024 costs plus the small-project premium), shown as is.
- **The live strip** says "latest full day loaded" instead of "yesterday" when yesterday's prices
are incomplete; today it shows 2026-10-10.
- **Judgment calls:**
- `/report` stays public because old links must keep working, so the Pro-only PDF is
enforced in `/app` only.
- "Assistant messages" counts AI replies; guided mode is not counted.
- Free accounts get the site-check summary and the scenario chart. The comparison tables,
stand-alone compare workflows, own cost quote, PDF and CSV are Pro.
- Owner approval needs the Battery owner plan.
- Anonymous answers use the engine text, not an AI write-up.
- The Claude prices in `app/plans.py` (Sonnet $3/$15, Haiku $1/$5 per million tokens) are
assumptions used only for the budget meter.
- **Leads no longer store raw IP addresses** (rule 5). Old values were cleared in migration
0015, and the per-IP rate limit now keys on a salted hash held in memory.
---
## 2026-10-19 - Why there were no schedules from Oct 15 to Oct 19
Checked after the redesign: the Oct 16 daily job did not produce the Oct 17 schedules. Neither
did any job since Oct 15; the last schedules are for 2026-10-14. Three causes, two of them bugs:
1. **gridstatus refuses every request.** Each 19:35 UTC run ends "pending" with
`Error 403: API requests limit reached. Usage: 375, Limit: 250`. The counter has not moved
in eight days, so this is not the daily cap the code assumes; it is a monthly (or longer)
limit, spent on Oct 12 by the load-forecast and outage backfill. Nothing in the app can fix
that: `app/sources/pjm.py` is a stub and there is no PJM key. It needs the gridstatus cap to
reset (Nov 1 at the latest) or a plan upgrade.
2. **Hourly retries never ran, since Phase U (de382e7).** `run_daily_job` imported
`date, timedelta` inside its `except` branch, which made `timedelta` a function-local name;
on every pending run the retry line raised `UnboundLocalError`, so only attempt 1 ever
recorded. The imports are module-level now, with a regression test. Verified live: after the
deploy attempt 2 ran at 23:29 UTC and retry 3 was scheduled.
3. **The 10 PM summary said "All good" every night.** 10 PM ET is 02:00 UTC, and
`job_time_today` used the UTC date, so the dashboard thought the job was not due yet and
reported the pre-job green line. It now uses the Eastern day, and from 10 PM ET "tomorrow not
ready" is red whether or not a retry is pending, with the job's own failure reason in the
line. The admin is also emailed at once when the price fetch itself fails (once per day) and
again if the last attempt still has no schedule.
Tonight's summary will read: "Needs attention - tomorrow's schedule was not created (fetch
failed: ... API requests limit reached ...)". It will stay red until the cap resets.
---
## 2026-10-19/20 - Launch build: the developer version, ready to sell
Six stages, each committed and deployed in turn (5778e58, d83ec51, 50f2fcc, cd33b71, and the
1ca0949), every stage verified on the public URL. 232 tests pass; CI green on 1ca0949.
### URLs
| URL | What |
|---|---|
| `/` | "Check a New Jersey battery site before you spend on it." Get a report ($500), Try a free check, the chat box, the live example, plans, FAQ. No owner messaging. |
| `/plans` | Free check, Site report $500 (reviewed, delivered within 24 hours of payment), Pro $149. The owner plan is hidden behind `OWNER_PLAN_PUBLIC` in `app/plans.py`; owner pages keep running. |
| `/get-report` | The order form: site (zone or address), MW, hours, connection, optional cost and interconnection estimates, email, company. |
| `/methods` | How we calculate: every source with dates and links, the optimizer and the forecaster, the capacity, GSESP, ITC and Regulation assumptions, the limits, and what a report is and is not. Linked from every report and the footer. |
| `/about` | Who builds GridArb (one person, Sayreville NJ, since October 2026), why, the sample report, contact, the disclaimer. |
| `/index`, `/index/2026-09`, `/index/2026-10` | The NJ Battery Index: per zone, 1 MW / 2 MWh and 1 MW / 4 MWh, energy arbitrage (perfect foresight and 80% achievable), capacity and Regulation pro-rated, best and worst days, a summary and a "Copy for LinkedIn" button. Built on the 1st at 07:30 ET; the current month refreshes daily as "so far". |
| `/admin/reports` | The review queue: engine numbers, preview, note, Approve and deliver, Mark paid. The status line on `/admin` goes yellow while a report waits. |
| `/r/<token>` | A customer's private report page with the reviewer's note (404 until approved). |
| `/admin/analyst` | Users and plans with AI cost this month next to what they pay (flagged above 30%), automatic AI budget with override, AI switch. |
### What changed in the 5 MW / 4-hour AECO example (live numbers)
- Revenue is now shown as a range: **$472,523 a year base, $457,201 low to $487,845 high**
(2025 prices; low and high are the achievable factor 0.70 to 0.90 around the capacity line).
- **Regulation: not in the number yet.** The line is in the stack, the method is on `/methods`,
the tests cover low / base / high, the scenarios and the payback change, but the figure needs
PJM's Regulation market results, which need the PJM key or the CSV export below. Until then the
line reads "Not loaded yet" and the total is unchanged. The moment the data is imported the
line fills in on every report and the AECO payback moves (for scale: at a post-redesign average
of $9/MW-hour, a 5 MW 4-hour battery regulating in its 16 idle hours earns about $263,000 a
year base, which would lift the AECO example from $472,523 to about $735,000 and bring simple
payback inside 15 years; that is an illustration of the formula, not a result).
- **Cost sensitivity:** capex after ITC $5.22M as shown; $4.18M at cost -20% (NPV -$2.89M);
$6.27M at cost +20% (NPV -$6.19M). Payback stays outside 15 years in all three at the NREL
benchmark, which is the engine's honest answer for this size at that cost.
- **Interconnection estimate:** entering $250,000 adds it to capex (no ITC): net capex $5.47M,
NPV -$4.87M. The report now also shows PJM's deposits for 5 MW ($75,000 + $20,000 readiness =
$95,000, Manual 14H), the typical 2-3 year timeline, a link to count queue projects within 10
miles on interconnection.fyi, and the list of what GridArb cannot know.
- **Capacity assumptions** are printed in every report: 2026/27 $329.17, 2027/28 $333.44,
2028/29 $325.00 per MW-day (all at the cap), the ELCC table, the collar ending after 2029/30,
and the 2-hour-as-energy/4 assumption to confirm in PJM's ELCC calculator.
### Data sources (stage 1)
- `app/sources/pjm.py`: PJM Data Miner 2 for day-ahead and real-time LMPs (zones and pricing
nodes) and `reg_market_results`, paged, paced to 6 requests a minute, backing off on 429,
every request logged under a `dm2:` prefix so it never counts against gridstatus. Built and
tested against recorded responses (`tests/fixtures/pjm/`), so it works the moment the key
arrives. Two things to confirm on the first real call, each in one place: the date-range
filter format (`_range`) and the column names (`LMP_FIELDS`, `REG_FIELDS`); a mismatch raises
an error that lists the columns the feed returned.
- `DATA_SOURCE=auto`: PJM when `PJM_API_KEY` is set, gridstatus otherwise. Enable with
`railway variables --service gridarb-api --set PJM_API_KEY=<key>` (and the same line in `.env`).
- `scripts/import_pjm_csv.py`: Data Miner CSV exports from `data/imports/` for the same feeds.
- gridstatus: the limit is per billing period. `GRIDSTATUS_MONTHLY_REQUEST_LIMIT` (250) and
`GRIDSTATUS_RESET_DAY` (1); `/admin` shows requests against the period; the first refusal
records the hit, emails once, and the daily job stops retrying until the reset day, then
resumes and backfills by itself (`catch_up_prices`; `POST /admin/backfill-prices` on demand).
- Backfill since Oct 14: not possible yet (no working source); it runs automatically on the
first successful fetch.
### The Data Miner export still needed (Regulation)
Data Miner 2 -> Markets & Operations -> "Regulation Market Results" (feed `reg_market_results`),
date range **01/01/2024 to today**, all columns, CSV. Save it as
`C:\Users\Rudra\gridarb\data\imports\reg_market_results.csv` (several files named
`reg_market_results_1.csv`, `_2.csv` also work). Columns that matter: `datetime_beginning_utc`,
`datetime_beginning_ept`, `rmccp`, `rmpcp`, `regd_mileage`, `rega_mileage`, `rto_perfscore`,
`total_mw`, `service`. Then run `uv run python scripts/import_pjm_csv.py`; it reports rows read,
kept and the date span, and lists any column it could not map. If Data Miner shows a separate
post-redesign Regulation feed, export that one too under its own feed name and send me the
column list. Day-ahead and real-time LMP exports (`da_hrl_lmps`, `rt_hrl_lmps`, Oct 14 to today,
type ZONE) go through the same script and would close the price gap without a key.
### Keys still missing (KEYS.md has the full table)
`PJM_API_KEY` (prices and Regulation from PJM; the Oct 2026 gap fills on the next run),
`ANTHROPIC_API_KEY` (typed chat; guided mode until then), `CLERK_PUBLISHABLE_KEY` and
`CLERK_SECRET_KEY` (sign-in and per-account limits), `PAYMENT_LINK_SITE_REPORT` and
`PAYMENT_LINK_PRO` (checkout; until then "Request report" saves a lead and emails you, and plans
are set by hand), and a verified Resend domain (emailing customers the report link and PDF).
### Launch checklist: the things only you can do
1. Export the Regulation CSV (above) and run the import; or get the PJM key and set it.
2. Create the $500 Stripe or Lemon Squeezy payment link and set `PAYMENT_LINK_SITE_REPORT`.
3. Verify a sending domain in Resend and set `RESEND_FROM`, so customers can be emailed.
4. Confirm the 2-hour ELCC rating in PJM's ELCC calculator (the report says you will).
5. Confirm the Manual 14H deposit figures ($75,000 + $4,000/MW) before quoting them.
6. Check the Claude prices in `app/plans.py` against Anthropic's price list.
7. Decide whether the gridstatus plan is worth upgrading or whether PJM replaces it.
8. Delete (or keep as a sample) the self-test report #1 "GridArb self-test (delete me)" in
`/admin/reports`; it was the end-to-end check of the delivery flow and sent you one email.
9. Sign in to `/admin` on your phone once so the review-queue status line is handy.
### What did not go as planned
- Two commits reached CI red: the Stage 3 commit carried `index_pages.py` before its
line-length ignore existed (fixed in the Stage 4 commit), and one deploy went out with a test
still red because a `tail` masked pytest's exit code; the gate now runs pytest directly.
- The Stage 3 commit message was amended after a push (a stale message file); force-pushed with
lease, no one else pushes to this repo.
- The PJM source has never seen a real response; the parts most likely to differ are isolated
and documented. The regulation revenue rule (idle hours, RegD-style price formula) is a
GridArb assumption until checked against a real operator's numbers.
- Regulation data, and so the Regulation line and the "how payback changes" answer, are blocked
on the export or the key.
---
## 2026-10-20 - Regulation data loaded; the 5 MW / 4-hour AECO example re-run
Six Data Miner 2 exports imported into `regulation_hourly` (now 24,383 hours, 2024-01-01 to
2026-10-13): `reg_zone_prelim_bill` 2024, 2025 and 2026 to Oct 12 (RMCCP, RMPCP and the hourly
credit totals) and `reg_market_results` 2024, 2025 and 2026 to Sep 30 (mileage, performance
score, procured MW). From the October 2025 redesign the market feed is half-hourly with the
RegD columns blank; the importer averages half-hours to the hour and uses the single product's
mileage column.
**A formula error was caught and fixed on the way.** The first pass used the absolute signal
mileage (about 30) in RMPCP x mileage x performance, which overstated 2024-25 prices about
threefold. The billing feed settles the question: each hour's `total_pjm_rmccp_cr` equals RMCCP
x MW regulated exactly, and `total_pjm_rmpcp_cr / (rmpcp x MW)` is the performance multiplier PJM
actually paid: about 2.5 in 2024, 2.4 in 2025, 1.0 in 2026. The credit per MW-hour is now
RMCCP + RMPCP x that paid multiplier (stored per hour as `credit_factor`); the fallback is the
RegD / RegA mileage ratio x performance. A second error, found in the first re-run: where
regulation beats the arbitrage margin the battery regulates all day, so the arbitrage line must
be forgone; it was being counted on top. Fixed in the stack total and the 15-year paths.
**What the data says (PJM RTO, $ per MW-hour of regulation):** 2024 average $31.6; 2025 before
the redesign $42; since the redesign $116 (Q4 2025 $82, 2026 so far $127). The 2026 average
leans on spike hours: the median hour pays $36.8 and the 90th percentile $271. PJM's whole
regulation requirement is 250 to 390 MW. These three cases, applied to a 5 MW / 4-hour battery
that regulates all day because every case beats the arbitrage margin:
| | Low (pre-redesign $36.22/MW-h) | Base (post-redesign $116.16) | High (2026 Q1 $143.68) |
|---|---|---|---|
| Regulation per year | $1,586,261 | $5,087,677 | $6,293,053 |
| Energy arbitrage | forgone | forgone | forgone |
| Capacity | $349,944 | $349,944 | $349,944 |
| Total per year | $1,936,205 | $5,437,620 | $6,642,996 |
| Simple payback on $5.22M after ITC | 3.3 years | 1.0 years | 0.8 years |
| NPV at 8%, 15 years | $7.0M | $34.6M | $44.1M |
**How the payback changed:** before the Regulation data the example earned $472,523 a year
(energy $122,579 + capacity $349,944) and did not pay back within 15 years at the NREL benchmark
cost (NPV -$4.54M). With Regulation it pays back in about 1 year in the base case and 3.3 years
even at pre-redesign prices. Cost +/-20% moves the base payback between 0.8 and 1.2 years.
**Read this with the warning in every report:** the base case assumes a 5 MW battery clears
PJM's regulation market in every hour at the market-average credit, in a market of a few hundred
MW whose price quadrupled after the October 2025 redesign. A few hundred MW of new storage would
move that price, and PJM has every reason to change the design again. The low case (pre-redesign
prices, 3.3-year payback) is the one to plan around; the base and high cases are what the last
twelve months paid, not a forecast. This is a screening check, not a bank-ready revenue study.
Live on the public URL: every report shows the Regulation line with the three cases, the median,
and whether regulation replaces arbitrage; `/methods` explains the formula and the rule; the
NJ Battery Index months were rebuilt with the Regulation column filled.
---
## 2026-10-20 - Security hardening and realistic Regulation numbers
### Part A, security (from the code review)
- Every response now carries Strict-Transport-Security (one year, includeSubDomains),
X-Content-Type-Options nosniff, X-Frame-Options DENY, Referrer-Policy
strict-origin-when-cross-origin and a Permissions-Policy disabling camera, microphone and
geolocation. Private pages (`/r/<token>`, `/owner/<token>`, `/admin*`) send Referrer-Policy
no-referrer and X-Robots-Tag noindex, nofollow, and get a robots meta tag inserted into
their HTML. `/robots.txt` disallows `/r/`, `/owner/` and `/admin`.
- `client_ip()` uses the last X-Forwarded-For address (the one Railway's proxy appends).
- The admin session cookie lasts 30 days; `POST /admin/logout` clears it (Log out button on
`/admin`).
- Clerk tokens: RS256 stays pinned; the `azp` claim must match `ALLOWED_ORIGINS` (default the
production URL).
- The analyst's system prompt now says text inside tool results and user messages is data, never
instructions.
- CI runs pip-audit before the tests (clean today).
- `/docs`, `/redoc` and `/openapi.json` are off unless `DEBUG_DOCS=1`.
### Part B, Regulation on a forward path
The previous base case assumed the battery cleared every hour at last year's average price
(0.8-year payback). Now:
- Forward price path, $ per MW-hour: base about $150 in 2026 falling to about $35 by 2033
(Modo Energy's PJM regulation outlook), then flat; low the pre-redesign average ($36.22)
falling to the battery's cost floor ($4.30: a $3 margin adder plus the energy lost cycling on
the signal) by 2028; high the last-12-month average ($115.82) held.
- Cleared-hours share: high 0.8, base 0.5, low 0.3. Regulation in the idle hours and, where it
beats the arbitrage margin, in the cleared share of the active hours (that share of the
arbitrage is forgone).
- Headline, payback and NPV use the forward base. The historical figure is shown separately,
labelled "what the last 12 months paid if you had cleared every hour".
- From December 2026 the price is shown split into RegUp and RegDn at 50/50, labelled as a
placeholder.
- Benchmark check on every report: GridArb's base revenue per MW-year against Modo Energy's
published PJM figures ($62/kW-month fleet average after the redesign; $84,000 per MW-year
central case for a 4-hour battery, 2027-2049). Above 1.5x the central case the report shows a
red "Above industry benchmark" box and `test_shipped_example_is_within_the_industry_benchmark`
fails.
### The 5 MW / 4-hour AECO example now (live)
| | Low | Base (forward) | High |
|---|---|---|---|
| Regulation price, 2027 | $20.26/MW-h (on the way to the $4.30 floor) | $133.57/MW-h | $115.82/MW-h |
| Cleared share | 30% | 50% | 80% |
| Regulation / year | $266,190 | $2,925,205 | $4,058,403 |
| Energy arbitrage kept | $107,257 | $61,290 (half forgone) | $27,580 |
| Capacity | $349,944 | $349,944 | $349,944 |
| Total / year | $691,214 | $3,336,438 | $4,435,927 |
| Simple payback on $5.22M after ITC | never within 15 years | 1.7 years | 1.3 years |
| NPV at 8% | -$2.8M | $18.0M | $26.7M |
Context only: the last 12 months, cleared every hour, would have paid $5,073,004 a year.
**The benchmark check fires, by design.** With the brief's own parameters the base case is
$667,270 per MW-year, 7.9x Modo's $84,000 central case, so the report carries the red warning
and the benchmark test fails (CI red on this commit until the parameters are reconciled). The
two cannot both hold: 50% of 8,760 hours at even the $35 floor is $153,000 per MW-year from
regulation alone, above the whole central case. Ways to reconcile, for a decision: (a) compare
against Modo's fleet average ($744,000 per MW-year) for the near term and the central case for
the 15-year average; (b) cut the base cleared share to about 10-15% at the floor; (c) drop the
base path to the floor by 2029 instead of $35 by 2033. GridArb will not quietly pick one.
---
## 2026-10-20 - Benchmark decision: the assumptions fixed, not the numbers
The 7.9x benchmark conflict is resolved by changing what GridArb assumes, as decided:
1. **First operating year (COD), not today.** Every stack, payback, NPV and the fifteen-year
view start in the COD year. New input `cod_year` on `/backtest`, `/report`, `/api/finance`
and `/api/scenarios`; default 2029 for a transmission-connected project (PJM's cycle-based
interconnection) and 2028 for a distribution-connected one (utility WMPA route). After COD the
Regulation line follows its forward path year by year; arbitrage and capacity are held flat
in the payback model as before.
2. **Cleared-hours share from competition:** `min(0.8, 750 MW requirement / expected battery
fleet)`. Fleet path (PJM's ESR paper and Modo Energy): 600 MW in 2026, 2,500 in 2027, 4,000
in 2028, 5,000 in 2029, 7,000 from 2030. Low case 1.5x that fleet, high case 0.5x. Base
share: 80% (2026), 30% (2027), 19% (2028), 15% (2029), 11% from 2030.
3. **Price paths, $/MW-h:** base $150 in 2026 falling to the cost floor ($4.30) by 2029, then
flat; low the pre-redesign average to the floor by 2028; high the last-12-month average
($115.82) falling to $35 by 2033 (Modo Energy).
4. **Benchmark, apples to apples:** the first operating year's total per MW-year against Modo's
post-redesign fleet average ($62/kW-month, $744,000 per MW-year), and the 15-year average
total per MW-year against Modo's central case ($84,000 per MW-year). The red box on the
report and the failing test fire only when the 15-year average is above 1.5x the central
case. The stack's 15-year average uses the scenarios' own formulas (capacity price and ELCC
paths, base arbitrage growth, forward regulation, usable-energy fade; GSESP excluded) and a
test checks the two agree.
5. **Both numbers shown:** "Expected from COD" is the headline; "If this battery were running
today" (this year's prices and competition) sits beside it, labelled context only.
Also: the methods page, the analyst card ("If running today (not the headline)"), the NJ
Battery Index (each month at its own year's price and share) and the citations (`modo_pjm`,
`pjm_esr`) follow. 247 tests, all green; deployed as 9163259.
### The 5 MW / 4-hour AECO example at COD 2029 (live)
| | Low | Base | High |
|---|---|---|---|
| Regulation price 2029, $/MW-h | $4.30 (floor) | $4.30 (floor) | $81.18 (falling to $35 by 2033) |
| Cleared share 2029 (fleet) | 10% (7,500 MW) | 15% (5,000 MW) | 30% (2,500 MW) |
| Regulation, year 1 | $12,556 | $18,834 | $1,066,758 |
| Energy arbitrage kept | $107,257 | $122,579 | $137,902 x 0.7 = $96,531 |
| Capacity (2028/29 price) | $349,944 | $349,944 | $349,944 |
| **Total, first operating year** | **$469,757** | **$491,357** | **$1,513,233** |
| Simple payback on $5.22M after ITC | never within 15 years | never within 15 years | 3.9 years (scenario path) |
| NPV at 8%, 15 years | -$7.0M cumulative, never positive | -$2.8M (scenario path); -$4.4M flat model | $11.2M cumulative after capex |
Benchmark: first operating year $98,271 per MW-year = 0.13x Modo's fleet average; 15-year
average $72,461 per MW-year = 0.86x Modo's central case. No warning; the benchmark test passes.
Context only, shown beside the headline: if the same battery were running today (2026, $150/MW-h
and 80% of hours cleared) it would show $5,630,460 a year in the base case ($1,640,404 low,
$4,435,927 high); the last 12 months at every hour would have paid $5,073,004.
What this means: at the brief's assumptions a 5 MW / 4-hour battery in AECO starting in 2029
does not pay back on energy arbitrage plus capacity plus a saturated regulation market; it needs
the high regulation path, a GSESP award, a lower build cost, or an earlier COD. Cost -20% still
does not reach payback (NPV -$2.8M in the flat model). The report says so in plain numbers.
Known quirk of the paths: in 2026 the base price ($150) sits above the high price (the
last-12-month average, $115.82), so the running-today high case is below the base case; from
2027 on the order is low < base < high. Changing it would mean changing the brief's paths.
---
## 2026-10-20 - Three adjustments: base path to $35, high never below base, GSESP award case
1. **Base Regulation price path:** $150/MW-h in 2026 falling to $35 (Modo Energy's long-run
level) by 2029, then flat. The cost floor ($4.30) applies to the low case only.
2. **High is never below base** in any year; where the high path (last-12-month average falling
to $35 by 2033) would dip under base, it is set equal to base for that year. In 2026 that lifts
the running-today high case to the base price.
3. **"With GSESP award" case** for sites of 5 MW and more: a second headline line (revenue,
payback, NPV) and a low / base / high table without and with the award at the Tranche 1 range
($60,000 / $75,000 / $81,840 per MW-year for 15 years), labelled "only if awarded; Tranche 2
decision expected December 2026". The no-award case stays the default everywhere: the
fifteen-year scenarios no longer carry a GSESP award in their default paths (the award case is
under `with_gsesp` in `/api/scenarios` and `/api/finance`). `/api/finance` now also returns
low / base / high payback and NPV (`cases`) on the same forward-path revenue series as the
base case.
248 tests, all green. Commit aa4446b. Railway answered "Deploys have been paused temporarily" at
the first attempt; the deploy is retried automatically and the numbers below come from the same
engine run on the live inputs (perfect-foresight arbitrage $30,644.79 per MW-year, Regulation
history through 2026-10-13).
### The 5 MW / 4-hour AECO example at COD 2029
| | Low | Base | High |
|---|---|---|---|
| Regulation price 2029, $/MW-h | $4.30 (floor) | $35.00 | $81.18 (to $35 by 2033) |
| Cleared share 2029 | 10% | 15% | 30% |
| Regulation, year 1 | $12,556 | $229,950 | $1,066,758 |
| Energy arbitrage kept | $107,257 | $104,192 (15% forgone) | $96,531 (30% forgone) |
| Capacity (2028/29 price) | $349,944 | $349,944 | $349,944 |
| **Total, year 1, no award** | **$469,757** | **$684,086** | **$1,513,233** |
| Simple payback, no award | never within 15 years | never within 15 years | 11.6 years |
| NPV at 8%, no award | -$4.59M | -$3.29M | -$0.88M |
| GSESP award per year (only if awarded) | $300,000 | $375,000 | $409,200 |
| **Total, year 1, with award** | $769,757 | **$1,059,086** | $1,922,433 |
| Simple payback, with award | never within 15 years | 8.5 years | 4.6 years |
| NPV at 8%, with award | -$2.22M | -$0.34M | +$2.35M |
Build cost $7.46M, $5.22M after the ITC. Cost -20% without an award: payback 14.9 years, NPV
-$1.64M. Benchmark: first operating year $136,817 per MW-year = 0.18x Modo's fleet average;
15-year average $97,334 per MW-year = 1.16x Modo's central case (under the 1.5x line; no
warning). Running today (context only): $5,630,460 base, $5,633,524 high (high lifted to the base
price in 2026), $1,640,544 low.
What this means: without an award the project does not pay back within 15 years in the low and
base cases and takes 11.6 years in the high case. A GSESP award at the base figure brings payback
to 8.5 years with the NPV still slightly negative at 8%; the high award case reaches 4.6 years.
The report says so, with the award line marked "only if awarded".
---
## 2026-10-20 - NJ Battery Index: observed lines only in the headline, Regulation on its own line
- The headline total and the zone ranking use only the observed lines, energy arbitrage (80% of
the perfect-foresight schedule) plus capacity (pro-rated to the month). Regulation is no
longer inside any total and no longer changes the arbitrage line.
- Regulation has its own column and sentence, labelled "PJM-wide estimate, if the battery
cleared X% of hours this month", where X is the competition-based share for the month's year
(min(80%, 750 MW requirement / expected battery fleet); 80% in 2026) at the month's forward
prices, low / base / high.
- The old "30/50/80%" wording is gone from the index; the figure note states the rule and the
fleet figure for the year.
- The summary is three sentences: best and worst zone for arbitrage with the dollars (and the
4-hour figure), the capacity share of the best zone's total, and one line on Regulation as
upside. The "Copy for LinkedIn" text is that summary followed by the month's link.
- September and October 2026 rebuilt on production through `POST /admin/build-index` once the
deploy landed (Railway had deploys paused for a while this afternoon; see below).
---
## 2026-10-20 - Grid connection check v2
The weakest part of the report now reads the utilities' own data and PJM's public lists.
### 1. Utility hosting-capacity maps (read automatically: all four, one with dated data)
Found by opening each app item on ArcGIS Online, its web map, and the web map's
operationalLayers, then testing a radius query (`app/grid_sources.py`):
| Utility | Public map | Layer read | What it gives | Data date |
|---|---|---|---|---|
| PSE&G | Experience c28232df... | NM_HCM_ServicePoints/FeatureServer/1 (1.5 M service points) | circuit availability kW, voltage, station | layer edited 2026-09-01 |
| JCP&L | webappviewer d43cf248... | Solar_Accommodation_Limit_2020/FeatureServer/0 (219 k points) | remaining kW AC, floor "less than 100 kW" (no voltage) | layer edited 2026-06-30 |
| Atlantic City Electric | the ACE page is a JavaScript shell with no map reference | PHI HostingCapacity/FeatureServer/1, layer "ACE" (326 feeder lines), found under the PHI owner on ArcGIS Online | allowable kW, kV, per-feeder update date | layer last edited 2022-11-16: dated, labelled as such |
| Rockland Electric | O&R Experience 11601d9b... -> storage app 9a48659b... | ORU_EV_Storage_Prod/FeatureServer/1 and /0 (90 k feeder segments, NY and NJ) | feeder discharge and charge capacity for storage (MW), kV, update date | 2026-10-01 |
One small radius query per site (300 to 1,500 m), cached 30 days per 100 m cell
(`hosting_lookups`); the layers' data dates refreshed monthly (`hosting_layers`, job on the 2nd);
only the 326-feeder ACE layer is pulled whole and stored with geometry (`hosting_feeders`), so it
answers without a request. The report shows the nearest feeder's figure with the map link and
the data date; an unreadable layer shows the map link and "check the map manually".
### 2. PJM queue and upgrade costs
- Queue: PJM's cycle service request status page renders its grid through a same-origin POST
handler that returns the whole cycle-process list (1,792 projects, AE1 onward; 92 in NJ: JCPL
36, AEC 35, PSEG 21) in one response. Imported weekly (Monday 07:00 UTC) into
`pjm_queue_projects`. No coordinates in the public list, so active projects with the zone's
transmission owner are matched by name to the OpenStreetMap substations within 10 miles and
shown at the nearest substation, within 5 miles and within 10 miles; unmatched ones are
counted, not placed.
- Upgrade costs: the TC1 Phase 2 and 3 and TC2 Phase 1 and 2 executive summaries (public HTML)
are parsed weekly into `pjm_upgrades` (TO, RTEP ID, title, time, total cost), `pjm_upgrade_allocations`
("Cost Allocation details for <RTEP ID>": project, percent, allocated cost) and `pjm_cycle_projects`.
The report shows "network upgrade cost per MW allocated to recent projects in this zone" as a
warning signal (yellow above $150,000 per MW, red above $500,000; GridArb thresholds).
TC2 Phase 3 and Cycle 1 are tried each week and skipped until published.
### 3. Queue Scope
Linked with a short how-to; a site-report customer attaches the Excel export on the private
report page (`POST /r/<token>/attach`, .xlsx/.xls/.csv up to 5 MB, stored in `report_attachments`).
### 4. Jurisdiction and process
New input `voltage_kv` (report URL and the order form). Below 69 kV: distribution route,
N.J.A.C. 14:8-5 Level 3 with the utility plus a PJM WMPA; 69 kV and above: PJM cycle route with
the Manual 14H deposits ($75,000 plus $4,000 per MW). Each route lists its steps, fees and typical
timeline (GridArb notes, not commitments).
### 5. Pre-application request
"Request a pre-application report" on every report opens `/report/pre-application` with a
filled request letter to the right utility (JCP&L: [email protected]; the
others via their interconnection pages) and an authorization letter to sign, with the $300
default fee and the note that NJ's PAVE process may not be live at every utility yet.
### 6. Score
Green / yellow / red with the reasons listed (feeder kW against the battery's kW at 1.0 and 0.5;
nearest substation at 3 and 10 km; queued projects at the same substation 1 and 3, within 5 miles
3 and 8; the upgrade-cost signal) and a "what only a study can tell you" box (exact feeder
capacity, upgrade costs, capacity rights, protection requirements, the in-service date).
### 7. Methods, citations, tests
`/methods` has a Grid connection check section; citations `hosting_capacity`, `pjm_queue`,
`pjm_cycle_summaries`, `queue_scope`, `njac_14_8_5`. Recorded fixtures from the real responses in
`tests/fixtures/grid/` (one or two features per utility layer, 30 queue rows, a TC1 Phase 3
excerpt with the project list, an upgrade table and two allocation tables); tests for each source,
the cache and monthly refresh, the name matching, the route, the score, the letters and the
report section. Migration 0019 adds the eight tables. Admin hook `POST /admin/refresh-grid?what=all|hosting|queue`.
Which layers could be read automatically: PSE&G, JCP&L and Rockland Electric (current data) and
Atlantic City Electric (the PHI layer, last edited November 2022, shown with its date). None
needs the manual link today; the manual link appears automatically if a layer stops answering.
---
## 2026-10-20 - The stuck site check, live progress, navigation
### Part A: why the Bridgewater check sat at "Working on it"
Production job 4 (5 MW / 20 MWh, JCPL, distribution, 760 East Main Street, Bridgewater): six
steps, five of them under a second, and `payback_and_grid` 91.0 seconds. The logs show the cause:
the grid desk check asked OpenStreetMap's Overpass API for nearby substations and all three
mirrors timed out at 30 seconds each. The run finished at 93 seconds; the chat showed only
"Working on it" because step progress lived in the Activity tab. The intake also asked "Where is
the site?" twice: the first message carried the address, but a location only counted when a zone
was set. The second Proceed (18:10:06) was already ignored with a message.
Fixes:
1. **Hard time limit per step: 20 seconds** (`run_step_with_timeout` in `app/analyst/workflows.py`).
A step runs in a worker thread with a private copy of the context; past the limit it is marked
skipped with the note "Skipped after 20 seconds: '<step>' did not finish in time (an outside
source was slow or blocked). The rest of the check continued without it." A late finish cannot
change the results. Outside calls were shortened too: Overpass 6 seconds per mirror (was 30),
geocoder 5 seconds (was 8), utility map layers 6 seconds per request (was 40).
2. **Live progress in the chat**: the progress list is shown in the chat for every run (not only
the compact widget); each step reads waiting, running (with seconds elapsed, live), done,
skipped or failed with its time; done steps show their one-line result as they complete, and the
results card refreshes after every step. Steps carry `started_at`; the job carries `updated_at`
and `step_timeout_s`.
3. **A second Proceed while a run is active** is ignored with "The site check is already running
(step N of 6), so I ignored this Proceed. Press Stop to cancel it."
4. **Guided intake reads the first message**: MW and hours were already parsed; an address now
resolves to a zone (a place word in the address, else the Census geocoder's county through
COUNTY_ZONE, labelled "from the address"; if neither works the check compares all four zones)
so the zone is not asked again, and "NREL benchmark" in a message answers the cost question.
### Part B: navigation
"Check a site" (/app) is the first item in the header menu on every page (the layout pages, the
app, the calculator, the track record and the schedule pages). The GridArb logo goes to /app for
signed-in users (the app stores a flag the other pages read) and to the front page otherwise.
"Back to the analyst" sits at the top of Plans, Track record, Calculator, Index, About and Get a
report. The header is sticky when scrolling, on mobile too; on small screens the menu keeps its
first item, which is now Check a site.
Tests: `tests/test_analyst_progress.py` (the limit skips a hanging step and the run continues;
progress fields; second Proceed ignored; the Bridgewater first message fills MW, hours, address and
zone and asks only about the connection; the header, logo flag and back links). 269 tests, all
green.
---
## 2026-10-20 - One basis on the analyst card; the Bridgewater check re-run; parser for every summary layout
- **Analyst card and replies on one basis.** The headline and the numbers are the expected
first operating year (COD 2029 transmission, 2028 distribution): revenue, cost after ITC, and
payback and NPV from COD. Today's figure is a separate labelled line, "If running today:
$X (context only)". When the user chose the zone (or gave an address that pins it), the card
says "Checked in JCPL" instead of "Best location: JCPL". The anonymous teaser keeps the same
lines. Tool summaries no longer say "on 2025 prices" for the total; energy arbitrage is
labelled as 2025 prices held flat.
- **An address pins the site.** The zone derived from an address now counts as the user's choice,
so the check stays in that zone instead of moving to the best-earning one.
- **Bridgewater re-run on production** (after the deploy at 22:42 UTC): the first message filled
MW, hours, address and zone; one question (connection); plan; Proceed. Job done in
**20.5 seconds** (was 93): five steps under a second, the grid step 19.1 seconds because all
three Overpass mirrors time out from Railway (6 s each then). Outside calls are now 4 s each,
so that step is at most about 12 s plus the geocoder. The score and the rest of the grid
section rendered; the JCP&L layer had no mapped point within 300 m (radius now 800 m).
- **PJM summaries: header-driven parser.** TC1 Phase 2 (captioned project list, per-project
"Cost Summary" allocation tables), TC1 Phase 3 and TC2 Phase 2 ("Cost Eligible for
Discounting" upgrade column, full transmission-owner names) all parse: 206 / 130 / 270
projects, 391 / 247 / 418 upgrades, 2,060 / 701 / 438 allocations; 16 NJ-owner projects in
TC2 Phase 2. TC2 Phase 1 as published is rendered by script and yields nothing; the weekly job
records that. Columns are picked by header name, never by position (a cost column headed
"Transmission Owner Interconnection Facilities ($)" had been read as the owner).
- The monthly layer refresh waits up to 30 s per request (the PHI service is slow; the first
production refresh timed out on ACE at 6 s). 270 tests, all green.
---
## 2026-10-27/28 - CI fixed and pinned; honesty and trust pages; the ITC choice, tariff scenario and NJ moratorium tracker
### Part A: the failed check, the pins, the deploy, the refresh
- **CI run 62 failed in pip-audit, not in the tests.** pip-audit looked the project itself up on
PyPI (gridarb 0.1.0, installed editable, never published) and PyPI answered 503. The step now runs
with `--skip-editable` and three attempts 20 s apart. The first green run also surfaced two real
advisories the Windows laptop never installs: certifi 2026.7.22 (PYSEC-2026-4320, now 2026.10.23)
and brotli 1.2.0 (PYSEC-2026-4322, now 1.2.1). CI runs 63 (brotli) and 64 (green) on 803fad2 and
16b8182.
- **Runners pinned to ubuntu-24.04** in both workflows (GitHub moves ubuntu-latest to 26.04 from
2026-10-19); actions/checkout v7, astral-sh/setup-uv v10.3.2 (no floating v10 tag exists) and
upload-artifact v6, all on Node 24.
- **Deployed** (Railway: builds working, only new signups paused after their October 21 incident),
then on production: grid refresh (PJM queue 963 projects at 02:49 UTC; layers PSEG 2026-09-01,
JCPL 2026-06-30, AECO 2022-11-16, RECO 2026-10-01), September and October index months rebuilt
(02:48 UTC), and the **Bridgewater check: 15.2 seconds** end to end (five steps under a second,
the grid step 13.7 s because the Overpass mirrors still time out from Railway). The teaser read
"Checked in JCPL ... first operating year (COD 2028) ... If running today: $5,630,146 a year
(context only)".
### Part B: honesty and trust
- **The first-check line** ("GridArb is a fast first check, not a bank-ready study. It tells you
which sites are worth a $75,000 study deposit and a consultant, and which to drop today.") sits
under the landing headline, at the top of every report (and so of every printed PDF), in the
report's method section and footer, and on the order form. `plans.FIRST_CHECK`.
- **/sources, Data sources and terms** (footer link): every source with the dates GridArb holds,
read live from the database (price range, Regulation span, layer data dates, queue fetch), and the
terms it is used under. PJM data is used under PJM's terms and is not redistributed; there are no
raw hourly PJM price downloads and no CSV exports for anonymous visitors (none existed; the page
says so). Flagged for later: Open-Meteo's free tier is for non-commercial use; the owner forecasts
move to a paid key before that product sells.
- **/assumptions, Assumptions and dates** (linked from /methods, every report and the footer): 20
rules from the code itself (Regulation redesign and split dates, price paths, cleared share and
fleet path, capacity prices at the cap, collar, ELCC by delivery year, GSESP range and award case,
ITC rate and FEOC treatment, cost benchmarks, cost range, tariff scenario, capture rate, COD
defaults, benchmarks, grid thresholds and deposits, the moratorium tracker), each with the date a
person last checked it (`assumptions.CHECKED`, moved only by a re-check).
- **/admin data-freshness panel**: the latest date held for day-ahead and real-time prices,
Regulation, capacity (the constants' check date), the queue list, the cycle summaries and each
hosting-capacity layer (our monthly check, with the utility's own data date beside it). Anything
older than 35 days is marked and turns the one-line status yellow.
### Part C: scenarios nobody else offers
- **Tax credit, the user's choice, on every report**: "30% ITC" or "No ITC (fails the foreign-entity
rule)", default 30%, switched by a link on the report (`itc=`), on the $500 order form (stored
with the order) and on the private report page. The inputs table says "your choice" either way.
- **Import tariff scenario** next to the +/-20% cost range: battery cost +0% / +25% / +80%, labelled
"import tariff scenario, not a forecast" (`TARIFF_SCENARIO_PCTS`; in /api/finance too).
- **NJ moratorium tracker** (`app/restrictions.py`, table `municipal_restrictions`, migration 0020,
public list at /moratoriums, admin add and remove at /admin/restrictions). Research on
2026-10-27 found one confirmed municipality: **Egg Harbor Township** (Atlantic County), unanimous
township committee vote on September 16, 2026, no new battery storage applications or
installations through December 31, 2026 (NJ.com September 19, 2026; Hoodline; Press of Atlantic
City agenda). Other hits were data-center bans (Bayonne, Cherry Hill, Clark, Medford, Millville,
Monroe) or out of state (New York, Michigan, Massachusetts); Beach Haven is contesting an existing
ACE battery, not banning new ones. The Census geocoder now returns the municipality (county
subdivision), so "Egg Harbor Township" and "Egg Harbor City" are told apart; a match puts a red
"local rules" reason at the top of the connection score and a red box on the report; every other
report carries one line saying the tracker found nothing for that town and when it was last
searched.
- Tests: `tests/test_trust.py` (ten tests). 280 tests, all green.
### Live strategy study (from the database): v3 strategies tuned on 2024, confirmed on 2025, 1mw_2mwh
Feature set v1; run 2026-10-12 15:31 UTC. tuning forecasts: two-fold out-of-sample within 2024; confirmation forecasts: model trained on 2024 (hgb-v1-2024-12-31)
**Decision:** recommended = **DA-only**. Best v3 = shift-if-confident at 52.0% of perfect on 2025 vs DA-only 51.9%: gain +0.1 points; NOT promoted (promote only if best v3 beats DA-only by >= 2 points of the ceiling on 2025).
Money captured as % of the perfect-foresight RT ceiling. Parameters chosen on 2024 only.
| Strategy | Params (2024-tuned, per zone) | 2024 tune % | 2025 confirm % | 2025 $ | vs DA-only (pts) | vs v1 RT-aware (pts) | vs naive (pts) | deviation days | deviation gain $ |
|---|---|---|---|---|---|---|---|---|---|
| DA-only (recommended) | - | 52.4% | **51.9%** | $86,042 | +0.0 | +13.1 | +21.1 | 0 of 1460 | $0 |
| v1 RT-aware | - | 37.6% | **38.8%** | $64,322 | -13.1 | +0.0 | +8.0 | 1460 of 1460 | $-21,720 |
| shift-if-confident | AECO: margin=30; JCPL: margin=50; PSEG: margin=30; RECO: margin=30 | 52.4% | **52.0%** | $86,085 | +0.1 | +13.2 | +21.2 | 9 of 1460 | $44 |
| spread-gate | AECO: gate=0, k=2; JCPL: gate=0, k=2; PSEG: gate=0, k=2; RECO: gate=0, k=2 | 52.4% | **51.9%** | $86,042 | +0.0 | +13.1 | +21.1 | 0 of 1460 | $0 |
| blend | AECO: w=0.9; JCPL: w=1; PSEG: w=1; RECO: w=1 | 52.4% | **51.3%** | $85,078 | -0.6 | +12.5 | +20.5 | 117 of 1460 | $-964 |
| EV with band penalty | AECO: lam=0.1; JCPL: lam=0.2; PSEG: lam=0.1; RECO: lam=0 | 38.9% | **38.5%** | $63,849 | -13.4 | -0.3 | +7.7 | 1401 of 1460 | $-22,192 |
Per zone, 2025 confirmation (% of perfect):
| Zone | DA-only | v1 RT-aware | naive | shift | spread-gate | blend | EV band | best v3 |
|---|---|---|---|---|---|---|---|---|
| AECO | 52.1% | 38.4% | 31.7% | 52.2% | 52.1% | 49.7% | 40.8% | shift-if-confident 52.2% |
| JCPL | 56.3% | 45.2% | 30.8% | 56.1% | 56.3% | 56.3% | 42.4% | spread-gate 56.3% |
| PSEG | 53.8% | 40.7% | 31.2% | 54.0% | 53.8% | 53.8% | 39.8% | shift-if-confident 54.0% |
| RECO | 46.4% | 32.0% | 29.5% | 46.3% | 46.4% | 46.4% | 32.0% | spread-gate 46.4% |