Repository navigation
Preserve precision in quantiles, interquartile ranges, and quantile axes - #142
Merged
Merged
Conversation
Improve two-point interpolation and compute integral IQR differences before rounding away common offsets. Fix selection-algorithm support in IQR and Freedman-Diaconis bin selection. Retain exact integral quantile boundaries across GC, RC, and custom factories, including maximum integer endpoints. Preserve bin membership when converting integral observations to floating-point coordinates. Add regression tests and document boundary and conversion behavior. AI-Assisted-By: Codex (GPT-6 Astra Medium)
Add opposite-sign operands before halving at the midpoint to avoid rounding subnormal values separately. Strengthen IQR forwarding tests and enforce tighter attributes in separate GC and custom-allocation tests. Skip coordinate correction at compile time when integral-to-floating conversion is always exact. AI-Assisted-By: Codex (GPT-6 Astra Medium)
Select scalar or endpoint/fraction results at compile time within the shared quantile selection routine. Preserve numerical safeguards and integral IQR precision while recovering small-input LDC performance. Add floating-point regressions for subnormals, extreme values, and infinities with @safe pure nothrow @nogc attributes. AI-Assisted-By: Codex (GPT-6 Astra Medium)
Include the observation type in interpolation working precision so narrower result types round only after interpolation. Normalize IQR probability arguments while preserving floating-point precision. Add regression coverage for mixed observation/result precision and integer probability arguments with @safe pure nothrow @nogc attributes. AI-Assisted-By: Codex (GPT-6 Astra Medium)
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Verify type-6 endpoint clamping preserves the exact spread between adjacent integers above 2^53, covering the previously untested path. AI-Assisted-By: Codex (GPT-6 Astra Medium)
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Quantile calculations could lose information by converting observations to floating point too early. This affected selection algorithms on large integers, interpolation between integral endpoints, and interquartile ranges where a small spread was obscured by a large common offset. Quantile-derived histogram axes could likewise collapse distinct integral boundaries.
This PR preserves observation precision through selection and interpolation and carries exact integral boundaries into quantile axes.
Quantiles and interquartile ranges
long.minandlong.maxis-0.5, and adjacent integers above2^53retain their spread.doubleprecision, including when probabilities are supplied asfloat.interquartileRange.Quantile axes and histograms
Implementation and performance
Scalar quantiles and integral interquartile ranges share the selection implementation. Scalar calls return their result directly, while integral IQR calculations retain the selected endpoints and interpolation fractions needed to calculate the spread accurately.
Benchmarks and generated-code comparisons guided this refactoring. The direct scalar path removed the observed small-input LDC overhead from intermediate results. Some DMD floating-point cases remain slower in exchange for the numerical safeguards. The final mixed-precision and IQR compatibility fixes produced unchanged assembly in the tested Windows LDC interpolation and default IQR probes.
Validation