Skip to content

Preserve precision in quantiles, interquartile ranges, and quantile axes - #142

Merged
jmh530 merged 5 commits into
libmir:masterfrom
jmh530:codex/integral-quantile-precision
Oct 3, 2026
Merged

jmh530 merged 5 commits into
libmir:masterfrom
jmh530:codex/integral-quantile-precision

Conversation

@jmh530

@jmh530 jmh530 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Quantile calculations could lose information by converting observations to floating point too early. This affected selection algorithms on large integers, interpolation between integral endpoints, and interquartile ranges where a small spread was obscured by a large common offset. Quantile-derived histogram axes could likewise collapse distinct integral boundaries.

This PR preserves observation precision through selection and interpolation and carries exact integral boundaries into quantile axes.

Quantiles and interquartile ranges

  • Keep selected integral observations exact for selection algorithms.
  • Preserve small differences between large integral values when calculating interpolated quantiles and interquartile ranges. For example, the midpoint of long.min and long.max is -0.5, and adjacent integers above 2^53 retain their spread.
  • Calculate ranks with at least double precision, including when probabilities are supplied as float.
  • Preserve wider floating-point observations during interpolation and convert to the requested result type afterward.
  • Handle mixed-sign subnormal midpoints without losing precision through separate rounding of each endpoint.
  • Restore support for integer probability arguments to interquartileRange.

Quantile axes and histograms

  • Retain integral boundaries from selection quantiles and caller-provided boundaries.
  • Include the maximum integral endpoint in the final ordinary bin without requiring a representable successor.
  • Preserve bin membership when integral observations are mapped to floating-point boundaries, and validate representability when coordinates are inserted into integral axes.
  • Add regression tests and documentation examples across GC, RC, and custom allocation.

Implementation and performance

Scalar quantiles and integral interquartile ranges share the selection implementation. Scalar calls return their result directly, while integral IQR calculations retain the selected endpoints and interpolation fractions needed to calculate the spread accurately.

Benchmarks and generated-code comparisons guided this refactoring. The direct scalar path removed the observed small-input LDC overhead from intermediate results. Some DMD floating-point cases remain slower in exchange for the numerical safeguards. The final mixed-precision and IQR compatibility fixes produced unchanged assembly in the tested Windows LDC interpolation and default IQR probes.

Validation

  • Full DMD and LDC test suites passed with DIP1000 and optimized builds.
  • After the final probability-local qualifier adjustment, focused optimized DIP1000 regression tests passed on both compilers.
  • Added regression coverage for integral extremes, adjacent large integers, subnormal values, mixed observation/result precision, integer probabilities, and quantile-axis boundaries.

Improve two-point interpolation and compute integral IQR differences before rounding away common offsets. Fix selection-algorithm support in IQR and Freedman-Diaconis bin selection.

Retain exact integral quantile boundaries across GC, RC, and custom factories, including maximum integer endpoints. Preserve bin membership when converting integral observations to floating-point coordinates.

Add regression tests and document boundary and conversion behavior.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Add opposite-sign operands before halving at the midpoint to avoid rounding subnormal values separately.

Strengthen IQR forwarding tests and enforce tighter attributes in separate GC and custom-allocation tests. Skip coordinate correction at compile time when integral-to-floating conversion is always exact.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Select scalar or endpoint/fraction results at compile time within the shared quantile selection routine. Preserve numerical safeguards and integral IQR precision while recovering small-input LDC performance.

Add floating-point regressions for subnormals, extreme values, and infinities with @safe pure nothrow @nogc attributes.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Include the observation type in interpolation working precision so narrower result types round only after interpolation. Normalize IQR probability arguments while preserving floating-point precision.

Add regression coverage for mixed observation/result precision and integer probability arguments with @safe pure nothrow @nogc attributes.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Verify type-6 endpoint clamping preserves the exact spread between adjacent integers above 2^53, covering the previously untested path.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
@jmh530
jmh530 merged commit 4166994 into libmir:master Oct 3, 2026
23 checks passed
@jmh530
jmh530 deleted the codex/integral-quantile-precision branch October 3, 2026 02:01
@jmh530 jmh530 mentioned this pull request Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant