Big experiment: 118-bus, 8760 scaling

.gitattributes

@@ -0,0 +1,4 @@
+# Preserve the pinned upstream source byte-for-byte; its comments contain
+# upstream trailing whitespace covered by the recorded SHA-256 hash.
+experiments/case118_annual_hierarchy/source/*.m -whitespace
+experiments/case118_annual_hierarchy/source/PGLIB_LICENSE -whitespace

.gitignore

@@ -277,6 +277,13 @@ experiments/dnlp_vs_pypower/*.fls
 experiments/dnlp_vs_pypower/*.fdb_latexmk
 experiments/dnlp_vs_pypower/*.synctex.gz
 
+# Local research and presentation material retained outside the public repository
+/experiments/ecosystem_positioning/
+/presentations/
+
 # Local agent skills and configuration
 .agents/
 skills-lock.json
+
+# Generated outputs
+/outputs/

AGENTS.md

@@ -0,0 +1,4 @@
+# Codex agent instructions
+
+Read and follow [`CLAUDE.md`](CLAUDE.md) before making changes in this
+repository. It is the authoritative developer guide for all AI coding agents.

README.md

@@ -8,25 +8,44 @@ lossy DC OPF (convex QP), and single-node DC dispatch (convex QP).
 
 ## Motivation
 
-Grid resiliency events rarely happen in an instant. The most dangerous
-scenarios unfold over days: solar suppressed by sustained weather systems,
-load elevated beyond seasonal norms, and battery storage depleted by
-controllers that optimize for the current hour. Studying the 
-behavior of the modern grid under these conditions and developing optimal
-control policies requires an optimization framework that is simultaneously
-time-aware and physically grounded, that is able to plan dispatch strategy 
-across a full multi-day horizon and able to enforce the AC network
-constraints that determine whether a plan is actually executable. 
-
-`cvxopf` is designed with this application in mind. It formulates optimal power
-flow problems using CVXPY, supports nonlinear AC-OPF, a convex lossy DC relaxation,
-and single-node economic dispatch from a single entry point (with more to
-come), and handles multi-step
-optimization with time-varying load, battery storage, and nondispatchable generation
-(wind, solar, hydro) natively. The intended use case is resiliency research:
-studying how battery controllers should behave under adverse multi-day
-conditions, how much temporal foresight matters, and how well convex
-approximations track AC feasibility across extended horizons.
+Grid resilience events rarely happen in an instant. The most consequential
+scenarios can unfold over days, months, or longer: weather suppresses renewable
+generation, demand remains elevated, geographically concentrated resources are
+damaged, recovery is gradual, and short-sighted controllers deplete storage
+before the system reaches its most constrained period. Studying these events
+requires an optimization framework that is both time-aware and physically
+grounded: it must coordinate decisions across long horizons while retaining
+the AC network constraints that determine whether a plan is realizable.
+
+`cvxopf` is designed for that research problem. From one modeling framework it
+supports nonlinear AC-OPF, convex lossy DC OPF, and single-node economic
+dispatch, together with multistep load, generation, storage, and transmission
+models. It is not intended merely as another OPF wrapper. It is the foundation
+for a scientifically coherent method for studying long-duration, uncertain,
+and compound resilience events without giving up access to nonlinear network
+physics.
+
+Long horizons preserve modeled storage, resource, damage, and recovery states
+across sequential events. Convex formulations make planning and broad scenario
+screening tractable. Selected nonlinear AC-OPF intervals can then redispatch
+active power, reactive support, and voltage state within the AC feasible set,
+rather than merely checking a fixed coarse-model dispatch.
+
+The larger research program is organized around the chain
+
+```text
+rare-event uncertainty
+        -> long-horizon adaptive planning
+        -> convex ensemble screening
+        -> nonlinear AC realization
+        -> audited resilience conclusions
+```
+
+The implemented package already provides the multi-fidelity component models,
+intertemporal storage, hierarchical DC-to-AC state handoff, nonlinear-solver
+recovery, and independent residual audits that support this direction.
+General stochastic investment planning and GPU-batched uncertainty ensembles
+remain research extensions rather than current package claims.
 
 Storage is treated as an intertemporal network device, not as a sequence of
 independent power injections. A multi-step solve co-optimizes the complete
@@ -49,38 +68,40 @@ terminal state; the causal greedy controllers are terminal-blind. Their lower
 dispatchable-energy totals are not improvements where they accompany unserved
 load.*
 
-Because it is built on CVXPY, the problem structure is transparent and
-composable. Researchers can modify objectives, add contingency constraints,
-or experiment with formulations — including multi-forecast Model Predictive
-Control — without rewriting solver interfaces.
+Because it is built on CVXPY, the mathematical structure is transparent and
+composable. Researchers can add device models, objectives, and operating
+constraints or study alternative network formulations without rewriting
+solver interfaces.
 
 ## Overview
 
 `cvxopf` formulates optimal power flow problems using CVXPY and solves them
 with appropriate solvers. It is designed to:
 
-- Run MATPOWER/Pypower test cases out of the box
-- Support multiple OPF formulations from a single entry point
-- Support single-shot optimization over multiple time steps
-- Accept time-varying nodal load as pandas DataFrames
+- Run MATPOWER/Pypower test cases out of the box.
+- Support multiple OPF formulations from a single entry point.
+- Support single-shot optimization over multiple time steps.
+- Accept time-varying nodal load as pandas DataFrames.
 - Model storage as a first-class intertemporal device with state-of-charge
-  coupling and configurable terminal policies
+  coupling and configurable terminal policies.
 - Model nondispatchable generators (wind, solar, run-of-river hydro) with
-  curtailable output and reactive power support
+  curtailable output and reactive power support.
 - Model loads as first-class, identity-aligned devices with optional
-  single-solve shedding and energy-not-served reporting
+  single-solve shedding and energy-not-served reporting.
 - Coordinate long-horizon convex battery planning with audited short-horizon
-  AC execution through the public hierarchical controller
+  AC execution through the public hierarchical controller.
 
 ### Methodology
 
 Many individual capabilities exposed by `cvxopf`, including multi-period OPF
 and intertemporal storage, also appear in other power-system optimization
 packages. The central contribution here is their organization within a
-[disciplined convex programming (DCP)](https://www.cvxpy.org/tutorial/dcp/) and [disciplined nonlinear programming
-(DNLP)](https://www.cvxpy.org/tutorial/dnlp/index.html) methodology: device dynamics, costs, and operating sets remain convex
-wherever the model permits, while the nonconvexity of the full AC formulation
-is confined to the network-flow physics.
+[disciplined convex programming (DCP)](https://www.cvxpy.org/tutorial/dcp/)
+and [disciplined nonlinear programming
+(DNLP)](https://www.cvxpy.org/tutorial/dnlp/index.html) methodology. Device
+dynamics, costs, and operating sets remain convex wherever the model permits,
+while the nonconvexity of the full AC formulation is confined to the
+network-flow physics.
 
 This separation supports the implemented hierarchical solve structure. A
 globally solvable, long-horizon convex layer determines intertemporal energy
@@ -102,6 +123,33 @@ affine extension of the load feasible set with a high linear value-of-lost-load
 cost in the original optimization problem; it is not a lexicographic pass, an
 anonymous balance slack, or a second feasibility-restoration solve.
 
+#### Economic decisions from modeled primitives
+
+CVXOPF constructs economic dispatch from explicit physical and economic
+primitives:
+
+- generator cost curves;
+- load and renewable availability;
+- network limits and either physical AC losses or a documented DC loss proxy;
+- storage dynamics, cycling cost, and terminal policy (with ideal efficiency
+  in the current `StorageUnitIdeal` model);
+- load-shedding cost; and
+- the evolving intertemporal system state.
+
+Exogenous electricity-price trajectories are not first-class inputs to the
+current model. Dispatch costs and scarcity consequences are represented
+directly, while marginal values arise endogenously from the optimization. This
+distinction is especially important in black-sky studies: a historical price
+series reflects a different network state, asset fleet, market design, and
+damage condition. Using that scalar signal to stand in for widespread outages,
+physical scarcity, customer consequences, and months of recovery would ask it
+to reconstruct interactions that the model had omitted.
+
+This does not imply that tariffs, contracts, or other explicit economic rules
+can never be modeled. When they are part of the scientific question, they
+should enter transparently as defined costs or constraints rather than serve
+as substitutes for available physical structure.
+
 AC voltage magnitudes and reactive dispatch are currently governed by their
 physical bounds and network equations but are generally not assigned an
 operating preference in the objective. Reactive variables can therefore reach
@@ -186,7 +234,7 @@ bus: no branch flows, no transmission limits, no losses, no reactive power —
 just total generation equals total load. It is the classic economic dispatch
 problem, useful as a fast baseline and for large-horizon energy planning.
 
-### Hierarchical DC to AC control
+### Hierarchical DC-to-AC control
 
 `solve_hierarchical_opf()` implements the project's reviewed two-layer
 workflow. The outer `lossy_dc` problem plans the full remaining horizon. Each
@@ -246,17 +294,18 @@ failure, and provenance contracts.
 References:
 
 - AC OPF: *Disciplined Nonlinear Programming*,
-  https://stanford.edu/~boyd/papers/dnlp.html,
-  https://github.com/cvxgrp/dnlp-examples/blob/main/nlp_examples/power_flow.ipynb
+  [paper](https://stanford.edu/~boyd/papers/dnlp.html) and
+  [power-flow example](https://github.com/cvxgrp/dnlp-examples/blob/main/nlp_examples/power_flow.ipynb).
 - Lossy DC OPF: *Convex Optimization with Smart Grid Examples*,
-  https://doi.org/10.2172/3018252
+  [technical report](https://doi.org/10.2172/3018252).
 
 ## Prerequisites
 
 `cvxopf` requires the IPOPT nonlinear solver system library. This must be
 installed before running `pip install cvxopf`.
 
 **Ubuntu / Debian**
+
 ```bash
 sudo apt-get update
 sudo apt-get install -y coinor-libipopt-dev liblapack-dev libblas-dev gfortran
@@ -269,11 +318,13 @@ sudo apt-get install -y coinor-libipopt-dev liblapack-dev libblas-dev gfortran
 > with a linker error (`cannot find -llapack`, `cannot find -lblas`).
 
 **macOS**
+
 ```bash
 brew install ipopt
 ```
 
 **Windows** (conda recommended)
+
 ```bash
 conda install -c conda-forge ipopt
 ```
@@ -440,8 +491,8 @@ and OPF configurations. The results should look something like this:
 
 ## Multi-step example
 
-Time-varying load is passed as a DataFrame — one row per timestep, one
-column per bus. This is the foundation for resiliency studies: feed in
+Time-varying load is passed as a DataFrame — one row per time step, one
+column per bus. This is the foundation for resilience studies: feed in
 a multi-day solar and load profile and the optimizer plans dispatch
 across the full horizon in a single solve.
 
@@ -548,9 +599,9 @@ and [`case9_multistep_load_shedding.py`](examples/case9_multistep_load_shedding.
 
 ## Battery storage example
 
-Battery state-of-charge evolves across timesteps, coupling decisions made
+Battery state of charge evolves across time steps, coupling decisions made
 at hour 1 to feasibility at hour 72. This intertemporal coupling is why
-multi-step optimization matters for resiliency: the optimizer can see that
+multistep optimization matters for resilience: the optimizer can see that
 conditions worsen on day 3 and hold reserves accordingly rather than
 depleting storage on day 1.
 
@@ -712,12 +763,16 @@ src/cvxopf/           Core package
   storage.py          Storage component: data, injections, constraints, cost
   nondispatchable.py  ND component: data, injections, and constraints
   hvdc.py             HVDC component and MATPOWER dcline conversion
+  load.py             Fixed and explicitly sheddable load component
+  hierarchical.py     Hierarchical DC-to-AC controller and audit records
   testcases/          Built-in MATPOWER test cases (case9 — case118)
 tests/                Pytest test suite
 tests/fixtures/       Committed Pypower reference outputs (static)
 scripts/              Fixture and test case generation scripts
 notebooks/            Interactive marimo notebooks
 examples/             Runnable example scripts
+experiments/          Reviewed scientific studies and retained protocols
+plans/                Milestone plans and implementation records
 ```
 
 ## Development
@@ -793,8 +848,17 @@ package environment.
 - [x] Single-node equivalent "copper plate" model
 - [ ] SOCP network model
 - [x] Extend battery parameters: terminal equality/shortfall constraints and linear/quadratic terminal costs
-- [ ] Implement cvxpy parameters for problem data
-- [ ] Vectorize time constraints (currently built with iterative loop)
+- [ ] Extend CVXPY parameterization for faster repeated solves
+- [ ] M14 time-vectorized multistep formulations: the explicit time-last
+  lossy-DC path is integrated into the Case118 `big-experiment` branch; its
+  conditioned 24/168/720 prefix ladder and 8,760-hour annual outer are
+  accepted. The retained
+  stepwise builder remains the default. The historical stepwise/CPP profiling
+  mismatch and the certificate-backed tight-tolerance disposition are both
+  tracked; vectorized/SCIPY with CLARABEL is the authoritative Case118 annual
+  realization. A non-promotional default-solver matrix found no accepted
+  alternative annual arm (see
+  `plans/milestone-14-time-vectorization.md`).
 - [ ] Full lossy HVDC (sign-switching converter losses via charge/discharge split) and reactive power support
 - [x] Unify grid component model patterns (dispatchable generators, storage, nondispatchable → first-class composable components)
 - [x] M16+ typed component adapters and shared formulation assembly (see `plans/milestone-16-plus-component-adapters.md`)
@@ -818,3 +882,13 @@ package environment.
   nonuniqueness and local-solver selection, then add only scientifically
   justified AC operating preferences (see
   `plans/milestone-20-ac-voltage-reactive-regularization.md`)
+- [ ] Nonconvex load-group penalties: model interactions such as mutually
+  exclusive customer-group shedding using relaxation, deterministic rounding,
+  and fixed-policy polishing (see
+  `plans/milestone-22-nonconvex-load-group-penalties.md`)
+- [ ] Unit commitment: add opt-in relaxed generator commitment to the convex
+  `lossy_dc` and `singlenode_dc` formulations, construct a fixed schedule with
+  a deterministic relax–partial-round–resolve–final-round–polish procedure,
+  and pass that schedule with polished SoC signposts into an explicitly
+  configured AC realization (see
+  `plans/milestone-23-unit-commitment.md`)

experiments/case118_annual_hierarchy/.gitignore

@@ -0,0 +1 @@
+results/

experiments/case118_annual_hierarchy/FIVE_MINUTE_TIMEOUT_POLICY.md

@@ -0,0 +1,69 @@
+# Case118 five-minute primary-attempt policy
+
+## Decision
+
+For subsequent execution of the frozen Case118 annual experiment, each
+target-constrained primary AC attempt receives an explicit 300-second solver
+wall-time budget. If the primary does not return an accepted primal within that
+budget, it is retained with its timeout/status evidence and the unchanged
+causal recovery sequence begins. The target-free solve must be accepted before
+its solution can initialize the copied hard-target solve. All existing
+accepted-status, residual, first-accepted stopping, and state-advancement rules
+remain unchanged.
+
+This is an experiment policy, not a general `cvxopf` default. It is a bounded
+tail-latency safeguard for this scenario and solver stack, not a claim that 300
+seconds is an optimal timeout or that it saves time on every difficult window.
+
+## Evidence
+
+The completed S3 month contained three accepted primary attempts that exceeded
+300 seconds. A post-hoc extreme-window replay reconstructed each exact archived
+initial state, outer signpost, immediately preceding causal controller, named
+shifted start, profiles, policy, and solver configuration. Both recovery solves
+were paid for and independently passed the unchanged hard-target acceptance
+gate.
+
+| Interval | Archived primary (s) | Target-free (s) | Copied hard-target (s) | Immediate pair (s) | 300 s + pair (s) | Fallback saving (s) |
+|---:|---:|---:|---:|---:|---:|---:|
+| 479 | 546.17 | 55.37 | 93.56 | 148.93 | 448.93 | 97.24 |
+| 569 | 535.63 | 34.91 | 104.19 | 139.11 | 439.11 | 96.52 |
+| 603 | 469.42 | 24.93 | 115.88 | 140.80 | 440.80 | 28.62 |
+
+The replay retained all 9,120 IPOPT starting coordinates for every solve
+(6,876 model-owned and 2,244 reduction-introduced) and verified exact named
+start and target reconstruction. Timings exclude model construction and start
+assignment and include CVXPY canonicalization, complete IPOPT `x0`
+construction/verification, and IPOPT execution.
+
+These windows were selected because their S3 primaries were slow. The result
+supports this tail safeguard but does not estimate general runtime savings.
+
+## Implementation gate before S4b/S5 AC execution
+
+1. The primary solve is actually stopped at its typed, explicit, retained
+   300-second budget, after which recovery begins.
+2. Timeout classification, the complete primary evidence, and every recovery
+   attempt remain in the audit tree.
+3. The budget is part of frozen experiment configuration and provenance, never
+   a hidden constant.
+4. A focused synthetic test exercises timeout into target-free and copied
+   recovery without waiting five minutes.
+5. Reporting separates the consumed primary budget, target-free time, copied
+   time, construction/canonicalization time where available, worker/restart
+   overhead, and total window latency.
+6. Existing acceptance, causal-source, first-accepted stopping, and exact-once
+   state-advancement rules remain unchanged.
+
+## Deferred performance work
+
+- prospective evaluation over larger samples;
+- adaptive budgets based on horizon, recent solves, or solver progress;
+- speculative parallel recovery;
+- construction and worker-restart overhead;
+- machine- and solver-specific calibration;
+- median, upper-quantile, and worst-case latency; and
+- alternative warm starts and future formulations.
+
+These questions belong in a dedicated performance milestone and do not block
+adoption of the frozen Case118 safeguard.