Fix destination reachability and add shared-basis evaluation metrics - #3
Merged
Conversation
The Hypatia-derived fstate path collapsed per-GS visibility to the single nearest satellite before routing, which made the destination a fixed satellite rather than the ground station itself. Under sparse connectivity that satellite is frequently in an unreachable component while another visible one is reachable, so link-state reported failure on deliverable pairs: in a Ring topology it delivered 80 of 408 deliverable pairs instead of all of them. Thread the full per-GS visibility list (ground_station_satellites_in_range) through algorithm_free_one_only_over_isls and the fstate calculation so the destination egress is whichever visible satellite minimises path length plus GSL length. The single-attachment fallback is kept for callers without a precomputed visibility list; it understates reachability and is logged as such.
The previous stretch baseline was a shortest path to whichever egress satellite the algorithm being measured happened to reach, so each algorithm was graded against a different target over a different subset of pairs, and neither the subset nor its size was reported. An algorithm that fails on hard pairs was rewarded, because only its successes reached the average. Compute reachability from the topology before any algorithm is consulted, via _best_reachable_egress: the best end-to-end route to any satellite the destination can see, hop-optimal and distance-optimal egress tracked separately since they need not coincide. Every algorithm is now scored over the same pairs against the same lower bound. - delivery_* columns: deliverable pairs, delivered pairs, delivery rate, forwarding failures, and non-delivery causes (no source visibility, no destination visibility, graph disconnection) - stretch_hop_shared / stretch_dist_shared graded against the shared lower bound; legacy stretch_hop / stretch_dist kept for continuity - delivery_non_optimal_egress_rate: share of delivered pairs exiting through a non-optimal egress, what separates the two families A shortest-path algorithm scores 1.0 on the shared basis by construction, so link-state doubles as a correctness check on the metric. Metric semantics documented in docs/evaluation.md.
Runs a config x algorithm x ISL-scenario matrix as parallel docker jobs, one single-threaded python process per job with concurrency bounded by JOBS. BLAS threading is pinned to 1 inside the container: without that numpy oversubscribes and 24 jobs fight over the machine. - skips jobs whose timestep_metrics.csv and metadata.json already exist, so an interrupted campaign resumes by rerunning the command - appends per-job wall-clock and exit code to job_timings.csv - heaviest constellations enqueued first so the long pole starts early - per-algorithm flags (distance mode, segment refresh/egress mode) resolved from env with the defaults used by the paper runs JOBS=24 ./scripts/run-matrix-parallel.sh /path/to/output
Backfill the Unreleased section for the baseline-removal refactor and the changes on this branch: the any-visible-satellite routing fix, the shared stretch basis and delivery accounting, and the parallel matrix runner.
CI lint runs black --check on leopath and tests. Two files drifted: - plot_seam_robustness.py: pre-existing since main; old magic-trailing- comma style on two multi-line calls - metrics.py: the parenthesised _best_reachable_egress call and the non_optimal_egress_rate entry from this branch collapsed differently than black 25.x prefers now that the expressions fit on one line No semantic changes; 147 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two coupled problems fixed, plus tooling for running the evaluation campaign:
Changes
Routing fix (
6678dc6)ground_station_satellites_in_range) throughalgorithm_free_one_only_over_islsand the fstate calculation, so the destination egress is whichever visible satellite minimises path length plus GSL length.Metrics (
f669f87)_best_reachable_egress: the best end-to-end route to any satellite the destination can see (hop-optimal and distance-optimal egress tracked separately, since they need not coincide). Every algorithm is scored over the same pairs against the same lower bound.delivery_*columns: deliverable pairs, delivered pairs, delivery rate, forwarding failures, and the separate causes of non-delivery (no_src_visibility,no_dst_visibility,disconnected).stretch_hop_shared/stretch_dist_sharedgraded against the shared lower bound; legacystretch_hop/stretch_distkept for continuity with earlier runs.delivery_non_optimal_egress_rate: share of delivered pairs exiting through a non-optimal egress — the quantity that separates the two stretch families.docs/evaluation.md.Tooling (
03e7959)scripts/run-matrix-parallel.sh: config × algorithm × ISL-scenario matrix as parallel Docker jobs,JOBS-bounded, BLAS pinned to 1 thread per container, completed jobs skipped on rerun (interrupted campaigns resume), per-job wall-clock and exit code injob_timings.csv.Verification
tests/network_state/test_algorithm_free_one_only_over_isls.py: 4 passed.Chores
dbbed6abackfills the CHANGELOGUnreleasedsection for the baseline-removal refactor (ad05c47, previously unrecorded) and this branch.