Update Ralph/LISA work in calmarg - #16
Merged
oshaughnessy-junior merged 21 commits intoJun 21, 2026
Merged
Conversation
Continue the multi-stage LISA-RIFT import by adding the next LISA-only support modules from the original branch: utility transforms, injection helpers, PSD generation, an initial-grid Fisher estimator, and the paper-era LISA template ini. Keep the import isolated from shared ILE/pipeline files; modernize script entry points enough to make package imports CI-safe, with smoke coverage for the new contracts.
Continue the staged LISA-RIFT import with the Sangria PyCBC-to-RIFT converter and the small LISA PSD response table. Modernize the converter so it is import-safe in CI: remove local sys.path edits, defer the optional pycbc import until frame reading is requested, and add smoke coverage for the missing-dependency path.
Prefer gwpy for Sangria frame reads in the LISA converter while keeping the older pycbc-named helpers as compatibility aliases. Add a focused LISA CI script and GitHub Actions job covering import safety, auxiliary helpers, response imports, lalsimutils compatibility, and the CEPP helper contract without yet gating on the heavier synthetic likelihood test.
Add a checked-in demo/rift/lisa scaffold that uses helper_LISA_Events.py to generate a LISA zero-likelihood CEPP bundle and render the BasicIteration DAG without submitting it. Cover the demo with a contract test and include it in the lightweight LISA CI script so the helper path has a promotable infrastructure test surface.
Add a demo/rift/lisa synthetic-input builder and direct LISA ILE runner that generates FD HDF5 A/E/T data, cache, flat XML PSDs, helper contract files, and a small nonzero likelihood output. Wire the helper for real data products by letting custom cache/channel/PSD arguments replace defaults, carrying an explicit LISA integration window, and requesting time marginalization. Cover the path with tests, including a float sample-rate check and a direct tiny ILE run.
… into rift_O4d_junior_ralph
Add a --lisa-known-sky path in util_RIFT_pseudo_pipe.py that delegates to helper_LISA_Events.py and renders a hyperpipeline CEPP DAG with integrate_likelihood_extrinsic_batchmode_lisa. Expose LISA-specific cache, channel, PSD, ecliptic sky, frequency, float sample-rate, and small-run controls without entering the LDG event helper path. Cover the contract in the LISA CI gate.
Add a demo analytic LISA PSD generator that writes A/E/T XML PSD products without checking generated artifacts into the tree. Introduce test/pp_lisa as the PP-style home for LISA synthetic bundles. The initial known-sky driver builds frames, cache, analytic PSDs, and renders util_RIFT_pseudo_pipe.py --lisa-known-sky; the LISA gate now covers this contract.
Let helper_LISA_Events.py treat ecliptic sky as an intrinsic hyperpipeline parameter by omitting --lisa-fixed-sky and seeding a small sky grid when --vary-sky is requested. Thread the mode through util_RIFT_pseudo_pipe.py and test/pp_lisa via --lisa-vary-sky / RIFT_PP_LISA_VARY_SKY, with contract coverage at the helper, pseudo_pipe, and PP-LISA surfaces.
Add an import-safe LISA run-check module for current ILE/all.net-style columns. The diagnostic summarizes max lnL, high-likelihood point counts, MC error, effective sample count, masses, and ecliptic sky at the best point. Cover the module and JSON CLI in the LISA gate as the modernized seed of the old paper-branch run-check plotting routine.
Add demo/rift/lisa/run_lisa_end_to_end.sh, which builds synthetic LISA A/E/T data, analytic XML PSDs, renders util_RIFT_pseudo_pipe.py --lisa-known-sky through CEPP, runs a real direct LISA ILE by default, and summarizes the output with the LISA run diagnostic. Document render-only and variable-sky toggles, and add CI-safe render coverage while leaving the full ILE path available for heavyweight local runs.
…oop)
First full cluster run of demo/rift/lisa exposed a chain of gaps that prevented
the iterative ILE->CIP DAG from running. These fixes get iteration 0 to recover
the injected MBHB (~0.1%) on the CIT pool, with ILE converging in a container via
condor file transfer (CPU, AV sampler).
integrate_likelihood_extrinsic_batchmode_lisa:
- implement --sim-grid: read the hyperpipeline grid, select row[--event], set
m1/m2/spin/ecliptic (the CEPP/DAG handoff the fork lacked).
- write hyperpipeline-format output (header + lnL/sigma_lnL cols) when
RIFT_HYPERPIPELINE_FORMAT is active, so util_CleanILE_hyperpipeline can read it.
helper_LISA_Events.py:
- 2-D (m1,m2) initial grid (was a 1-D line, degenerate for the CIP 2-D fit).
- ILE: AV sampler + --force-adapt-all (loud signals need adaptation; --no-adapt
collapses eff_samp to 1); converging n-eff/n-max/n-chunk; LISA d-min/d-max;
in-band fmin default (1e-4); ~600s data-integration window default (a 16s
window mis-marginalizes the long LISA signal and biases the lnL peak).
- CIP: only fit ecliptic when --vary-sky; AV sampler + --internal-use-lnL (loud
lnL overflows the default sampler/lsoda); MBHB mc/mtot ranges tied to the grid;
M-max-cut/sigma-cut/lnL-offset for MBHB scale.
make_synthetic_lisa_inputs.py: write the REAL analytic LISA PSD on the data's own
frequency grid (was a flat PSD).
util_CleanILE_hyperpipeline.py: honor RIFT_ILE_SIGMA_CUT for the consolidation cut.
util_RIFT_pseudo_pipe.py: default --lisa-data-integration-window-half to 300 (600s).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With these, the known-sky demo runs a clean multi-iteration ILE->CIP loop that
RECOVERS the injected MBHB and refines it (iter0 m1/m2 ~+0.6%/-0.7%, iter1
~+0.27%/-0.42% of 1e5/8e4), DAG status 0 with no failed/futile nodes.
helper_LISA_Events.py:
- CIP --eta-range bracketed to the grid (analogue of force-eta-range). Without
it the posterior drifts to the eta floor (0.01) -> extreme q -> garbage m1
(mtot = mc/eta^0.6 blows up); the truth eta ~0.25 is near the ceiling.
- write args_puff.txt (--parameter mc/eta, --puff-factor, grid-tied mc/mtot/eta
bounds) for the inter-iteration puffball.
util_RIFT_pseudo_pipe.py: pass --puff-exe/--puff-args/--puff-cadence/--puff-max-it
to the CEPP for the known-sky path when n-iterations>1 (gate with --lisa-no-puff).
The puffball perturbs the (very tight) CIP posterior so the next grid is not a
near-degenerate cluster, which otherwise makes the CIP refit ill-conditioned.
convergence_test_samples.py: derive mc/eta from m1/m2 when absent, so the
hyperpipeline posterior (m1/m2 columns) does not crash the test
("ValueError: no field of name mc").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container delivery for the known-sky LISA path should flow through the existing dag_utils machinery (write_ILE_sub_simple), not LISA-specific code. The LISA pseudo_pipe branch simply never forwarded --use-singularity, so that whole block was skipped. Now --lisa-use-singularity passes --use-singularity (and the CEPP --cache-file, which write_ILE_sub_simple's singularity path requires) through to create_event_parameter_pipeline_BasicIteration. With SINGULARITY_RIFT_IMAGE (osdf:// staged image, so dag_utils file-transfers it) and SINGULARITY_BASE_EXE_DIR (dir of the LISA ILE inside the image) set, the rendered ILE.sub then gets the native wiring with NO LISA special-casing: transfer_executable=False, MY.SingularityImage="./<image>", the osdf image added to transfer_input_files, and exe=$SINGULARITY_BASE_EXE_DIR/<ile>. This assumes a container with the rift_O4d_junior_ralph code baked in (the build kit clones the junior fork); a worktree-code overlay is a separate dev-only path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both align the demo with the SNR-800 reference run / A. Jan's guidance and are verified by a clean 4-iteration cluster loop whose posterior refines toward the injected MBHB (mean m1/m2 within ~0.1% of 1e5/8e4; sd collapses 170 -> 10). make_synthetic_lisa_inputs.py: generate the PSD from the LISA Data Challenge noise model (per-channel SciRDv1, via RIFT.LISA.utils.get_ldc_psds) when the optional `ldc` package is present -- this is the PSD people use for inference. Fall back to the analytic sky-averaged sensitivity curve (a mismatch tool) when ldc is absent; the two agree to <~1% across the sensitive band, so the fallback is an adequate CI proxy. PSD stays on the data's own frequency grid either way. helper_LISA_Events.py: CIP --fit-method default rf (random forest), matching the paper's cip-fit-method; robust to non-quadratic lnL surfaces. (Quadratic still works; rf verified end-to-end.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… race) StatusRecord.write() truncated status.json in place, so a concurrent reader (parallel marg jobs share one inline archive) could observe an EMPTY file -> json.JSONDecodeError in transition(). Write a per-PID/thread temp then os.replace() it in (atomic on POSIX), mirroring IndexAppend._write_all; .read() retries on a transient decode error. Surfaced running the popsynth_hyperpipe toy adaptive loop with explode-marg-jobs>1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fit the LISA ecliptic sky as ordinary CIP coordinates (phi/theta) read from the NAMED hyperpipeline columns ecliptic_longitude/ecliptic_latitude, instead of the eccentric branch's positional all.net column-shift hack (which bakes in technical debt). Adds the reflected (secondary) sky mode as an in-loop DAG step. Both validated end-to-end on a CIT cluster smoke run (4-iteration ILE<->CIP loop, file-transfer container): vary-sky recovers masses+sky to the truth; the reflect node maps the primary mode to its secondary at iteration 2 and the next ILE consumes it cleanly. - util_ConstructIntrinsicPosterior_GenericCoordinates.py: ingest/fit/emit sky by named column (use_sky), prior_map/prior_range_map phi/theta, --phi-range/--theta-range. - helper_LISA_Events.py: vary-sky grid jitters sky off-lattice (breaks sky<->mass collinearity), CIP --parameter phi/theta + ranges, ILE per-row fixed sky. - convert_primary_sky_mode_to_secondary (new): hyperpipeline-aware sky reflection. - create_event_parameter_pipeline_BasicIteration / util_RIFT_pseudo_pipe.py: reflected-sky DAG node (gated before the target iteration's ILE) + forwarding. - LISA contract tests updated to the corrected design (named-column CIP params, per-point fixed sky, off-lattice sky spread); all 12 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allow util_RIFT_pseudo_pipe.py --use-ini to build the LISA known-sky surface, instead of hard-exiting. The generic [rift-pseudo-pipe] parser already maps every CLI arg by name (so all lisa-* scalars/algorithm options just work); a small _lisa_data_products_from_ini() fills the per-channel data products (channels, PSDs) from the conventional [data]/[lalinference] sections. No LDG data-find is invoked; the LISA branch stays self-contained. The ini path is a thin front-end over the validated CLI machinery: with matched config it renders a BYTE-IDENTICAL workflow (args_ile/args_cip_list/args_test and DAG, incl. the reflected-sky node). Verified against the CLI render that was itself validated end-to-end on a CIT cluster run. - bin/util_RIFT_pseudo_pipe.py: replace the --use-ini LISA hard-exit with _lisa_data_products_from_ini(); flat [rift-pseudo-pipe] lisa-* keys still win. - test/test_lisa_ini_contract.py: render ini vs CLI, assert byte-identical args + matching DAG/reflect node (locks the equivalence). - demo/rift/lisa/BBH_lisa_demo.ini: toy template (IMRPhenomD, vary+reflected sky). - demo/rift/lisa/run_lisa_ini_demo.sh: build inputs + instantiate + render. - demo/rift/lisa/README.md: document the production-ini path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bustness)
The DAG sizes per-iteration MARG chunks from n-samples-per-job, but the grid an
adaptive iteration actually places (e.g. a puffball) can be SMALLER, so the tail
chunks ran off the end and the marg driver raised SystemExit ("index range
[a,b) exceeds grid size N"). Clamp the range to the grid instead; a fully
out-of-range chunk yields an empty (header-only) output the consolidation
ignores. Surfaced running an adaptive Rapster (popsynth_hyperpipe) loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…0 = inf)
fit_rf used sample_weight=1/y_errors**2; a zero error (placeholder rows can leak
into the accumulated marg net with sigma=0) makes the weight infinite, which
sklearn rejects ("Input sample_weight contains infinity"). Floor sigma at 1e-3.
Surfaced in an adaptive Rapster (popsynth_hyperpipe) EOS-posterior fit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Mostly ok. Most danger is the rf fit in EOSPosterior adding an error floor, which is dangerous to assume - we would be limiting future work potentially by limiting the assumed accuracy of inputs. Unlikely to be an issue since MARG errors are unlikely to be smaller than 1e-3 realistically. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cascades the updated Ralph/LISA branch into rift_O4d_junior_calmarg_in_loop.\n\nReview notes:\n- This is the active LISA line; previous Ralph/LISA PR #7 was merged, but this branch has advanced again.\n- Current branch has 21 commits not in calmarg, including LISA workflow modules, known-sky .ini flow, LISA demos/checks, and hyperpipe/simulation-manager robustness fixes.\n- Merge-tree check into current calmarg tip was clean.\n- Keep an eye on CI because this touches workflow/test surfaces and LISA support code.