Big experiment: 118-bus, 8760 scaling - #22
Open
bmeyers wants to merge 86 commits into
Open
Conversation
Freeze the PGLib-rated network study, matched effectively-unlimited counterfactual, deterministic scenario rules, and gated scaling ladder. Require a reviewed streaming reference runner with nominal, recovery, checkpoint, memory, and public-controller equivalence gates before week- or year-scale execution.
Pin and verify the PGLib IEEE-118 source, convert it to cvxopf's MATPOWER contract, and create the matched rateA-only control. Freeze quantized annual profiles, deterministic electrical-distance device siting, and the predeclared storage/renewable pilot grid. Add complete M17-compatible AC/DC probe audits, scientific drift regressions, and the initial S0 characterization record.
Execute the rated and matched-unlimited lossy-DC and AC pilot cases from a clean source checkpoint. Retain complete public results, M17-compatible audits, timings, dimensions, solver statistics, peak-memory observations, and source provenance in one atomic artifact.
Retain the original midnight run as a separate scientific record after it showed only tolerance-level rated storage movement. Freeze one common exogenous net-load window across the pilot grid and require both rating-scaled instantaneous movement and capacity-scaled throughput in rated AC before selecting a scenario.
Select the lowest qualifying storage scenario, record the repository case118 comparison, and preserve the amended six-hour results. Cap S1 resource use and mark direct 24-hour AC as unauthorized after the six-hour solve reached a practical runtime and memory boundary.
Freeze the 24-hour S1 protocol and add a resource-supervised runner for lossy-DC outer plans and bounded AC endpoint realizations. Preserve complete audit records, worker-local provenance, explicit failure classifications, and the S0 prohibition on direct 24-hour AC.
Document the accepted 24-hour outer plans and bounded AC endpoint studies, including matched-window storage behavior, resource use, and provenance. Preserve the direct 24-hour AC safety exclusion and distinguish economically selected cycling from correction required for feasibility.
Define the P0 protocol and a strict archive/checkpoint schema for the three-hour hierarchical runner. Validate attempt lifecycles, solver evidence, AC results, residual gates, causal state and controller chains, hourly timing, and outer-plan targets before allowing resume.
Add deterministic case9 fixtures, public outer/window construction, causal initialization helpers, and complete policy and solver-config integrity gates for the case118 streaming runner.
Add the frozen nine-slot recovery lifecycle, verified IPOPT starting-point capture, strict causal and physical state handoffs, exact-once state advancement, and integrity-bound outer-plan signposts.
Serialize complete build-free outer and window evidence, atomically persist immutable artifacts, normalize unavailable scalar results, and advance verified checkpoints only after successful state execution.
Run complete hierarchical trajectories with archive-first state advancement, causal recovery, current-RSS monitoring, immutable resource chunks, and crash-safe resume. Strictly validate outer plans, checkpoints, resource chains, and archived controller state before continuing execution.
Compare the public and streaming hierarchy across frozen 6- and 24-hour scenarios, including outer plans, complete attempt registries, IPOPT starts, executed actions, realized states, accounting, and trajectory summaries.
Exercise the complete nine-slot recovery and termination matrix through coherent synthetic solver outcomes, and verify that public and streaming controllers retain identical audit trees, actions, state transitions, and termination diagnoses.
Exercise safe stop and resume, corruption rejection, atomic checkpoint recovery, live-build release, and build-free causal reconstruction before the final consolidated P0 gate.
Bind the authoritative copied-target-free recovery event to a compact tracked record, preserving its complete slot, start, IPOPT, audit, result, and provenance evidence for local re-derivation and CI validation.
Record the 16 GiB resource boundary, verified restart, and exploratory continuation that completed all 168 intervals. Preserve the independent scientific audit, compact artifact metadata, provenance context, and the worker-recycling gate for S3.
Add the reviewed pre-S3 protocol for comparing no recycling against 32- and 16-interval worker recycling. Define shared outer-plan provenance, restart lifecycle, warm-start evidence, resource measurements, and reproducible reference artifacts. Clarify that cadence is the sole intentionally varied factor while fixed-order machine effects remain observational limitations. Specify valid partial-run continuity and polling-based restart-overhead measurement without introducing automatic decision gates.
Add the supervised 64-interval comparison for never, 32-interval, and 16-interval recycling. Track the validated S2 reference, preserve restart, memory, provenance, and causal warm-start evidence, and report matched observational differences without introducing an automatic S3 gate.
Promote the 192-interval observational result and document that planned recycling preserved the causal trajectory. Record the measured memory, runtime, and restart-cost evidence, and update the pre-S3 plans and experiment status.
Freeze the 720-interval global restart schedule, checkpoint handoff rules, and reviewed abnormal-stop semantics for the Case118 month study.
Define the 720-hour scenario, global 16-interval recycling schedule, resource limits, and reviewed continuation rules. Add a hash-bound fixture and semantic tests while keeping scientific S3 solves outside implementation validation.
Add the global 16-interval worker lifecycle, provenance and outer/checkpoint binding, cumulative resource supervision, reviewed continuation, and independent compact-result reconstruction. Cover the pre-execution contract with synthetic lifecycle and corruption tests without running the scientific month.
Promote the authoritative fresh-process M14a.1 result and record its formulation-specific decisions. Lossy DC and single-node DC select their qualified leaf boxes, while AC retains explicit inequalities pending production-structure evidence.
Freeze the M14a.1 representation decisions and record focused qualification gates for storage, nondispatchable, HVDC, and load-shedding boxes before component leaf-bound migration.
Add typed time-last schemas, mixed-temporality box normalization, frozen representation decisions, and one-call component hooks with shape, convexity, and DCP validation. Preserve the stepwise path and explicitly cover boundary states and multistep T=1 behavior.
Integrate component costs and contributions once per horizon, publish time-last variables and expressions without rebuilding per-step objects, and bind convex vectorized solves to explicit SCIPY canonicalization provenance.
Preserve vectorized variable, expression, boundary-state, and integrated-cost schemas through assembly and project time-last values back to the existing public result contract. Validate required sources and dimensions even without a primal while leaving stepwise extraction unchanged.
Compare explicit inequalities with leaf bounds for storage, nondispatchable generation, load shedding, and lossy-DC HVDC using shared CLARABEL/SCIPY probes. Retain binding behavior, complete public results, independent audits, canonical structure, solver failures, pair-local fallback decisions, and clean fresh-process provenance for the authoritative run.
Promote the authoritative SCIPY/CLARABEL comparison and bind its execution commit and artifact hash. Record all nine tested lossy-DC and single-node component-box decisions as leaf-qualified, close the M14b gate, and hand the frozen assembly contract to M14c without extending the result to AC or making runtime/RSS claims.
Bind the authoritative execution and promoted result, freeze all nine tested lossy-DC and single-node leaf-bound decisions, and mark M14b complete with M14c next. Preserve explicit AC boxes and make no runtime or RSS claim.
Promote the M14b-qualified component boxes into the typed leaf-bound registry and freeze M14c’s objective, tolerances, resource limits, execution sequence, equivalence boundaries, and big-experiment integration gate.
Implement the explicit time-last lossy-DC path with typed component bindings, qualified leaf bounds, SCIPY canonicalization, stable result and failure schemas, and static-input broadcasting without horizon-sized Python copies. Preserve the stepwise default and qualify the new path through T=1, short-horizon, full-component, physical-audit, failure-schema, and full-suite regression gates.
Merge the reviewed M14c implementation into big-experiment without changing the retained stepwise default, and route the frozen S4 outer seam through vectorized assembly with SCIPY canonicalization. Bind the exact merge lineage, scenario, policy, solver, representation, and execution authority; this checkpoint permits only the ordered 24/168/720 prefix ladder and keeps annual execution blocked pending separately reviewed evidence.
Add supervised 24/168/720-hour vectorized lossy-DC gates with immutable execution provenance, independent analysis, and interruption-safe resource and failure lifecycle records. Permit promotion only for a complete reviewed ladder while retaining annual execution as explicitly unauthorized.
Freeze a non-promotional stepwise+CPP comparison against the accepted vectorized+SCIPY 24/168/720 artifacts. Define its evidence, resource, timing, and scientific gates, and keep annual execution blocked pending review.
Add the frozen 24/168/720 stepwise-CPP profiling runner and independent analyzer for comparison with the accepted vectorized-SCIPY ladder. Bind historical provenance and artifacts, retain resource, timing, cost, and failure evidence, and keep the study non-promotional with annual execution explicitly blocked.
Report lossy-DC loss cost and objective accounting in the prefix analysis, then add a separate fresh-process diagnostic comparing stepwise-CPP and vectorized-SCIPY at tighter CLARABEL tolerances. Retain native solver gaps, full bounds audits, and exact production provenance while keeping annual execution blocked.
Add a small fleet-wide quadratic generator cost to reduce representation sensitivity. Track the exact diagnostic and its selection rationale, update the conditioned S4 evidence and hashes, and keep annual execution blocked pending the conditioned prefix ladder.
Promote the independently reconstructed 24/168/720-hour conditioned vectorized ladder. Record accepted scientific audits and resource evidence while retaining the annual-execution block pending review and an explicit authority update.
Point the stepwise/CPP profile at the approved conditioned 24/168/720 reference tree and hashes. Preserve the failed first attempt as environment-only evidence and keep annual authority disabled.
Rebind the 24/168/720 tight-CLARABEL diagnostic to the completed conditioned profile. Retain native primal/dual certificates and report whether each representation's objective separation is covered by the combined solver gaps.
Preserve the original profile mismatch while binding the tight-tolerance diagnostic and its certificate-supported scientific disposition. Select the vectorized trajectory for the annual study, record weakly identified storage and flow paths, and open a fresh supervised S4 execution target.
Promote the accepted 8,760-hour S4 outer result and retain the post-hoc MOSEK, OSQP, SCS, and HiGHS characterization with exact runner, version, and artifact bindings. Document that the matrix is non-promotional and CLARABEL remains authoritative for the frozen Case118 study.
Mark the vectorized lossy-DC milestone complete and record the promoted annual S4 outer as the accepted basis for S4b planning.
Define the deterministic charging/mid-SoC annual boundary rule, canonical manifest identities, timeout and process-tree resource scopes, and bounded qualification gates. Distinguish partition-policy effects from fresh-process equivalence and record the prior S4 memory hold as resolved.
Verify the accepted annual outer archive and signpost identities, then encode the frozen boundary rule in a noncircular canonical manifest. Retain every candidate score and identity-aligned shard state with deterministic reconstruction and corruption tests while leaving timeout, worker, merger, qualification, and numerical execution unauthorized.
Expand the solver-matrix report with canonical-form evidence and solver-specific interpretation. Clarify that SCIPY still produces solver-dependent representations, distinguish equality rows from leaf bounds, and document the limits and follow-up implications of the observed results.
Reserve the root outputs directory for intentionally untracked scratch artifacts and generated deliverables.
Mark immutable manifest derivation as reviewed and committed while preserving the explicit block on S4b execution.
Add the five-minute primary-attempt timeout and causal recovery path, supervised shard workers, explicit checkpoint/resume handling, deterministic merging, and independent qualification analysis. Bind execution evidence to the frozen manifest and outer plan while keeping annual shard execution unauthorized.
Define an opt-in relaxed commitment model for lossy-DC and single-node formulations, followed by deterministic relax–partial-round–resolve–final-round–polish and fixed-schedule AC realization.\n\nDocument the mathematical model, typed result and audit contracts, verification program, and MVP boundaries, then register the milestone in README and CLAUDE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.