Skip to content

Removes the statistical crate from dependencies.#203

Open
wesc wants to merge 1 commit intoStarlightSearch:mainfrom
filament-dm:remove-statistical-dep
Open

Removes the statistical crate from dependencies.#203
wesc wants to merge 1 commit intoStarlightSearch:mainfrom
filament-dm:remove-statistical-dep

Conversation

@wesc
Copy link
Copy Markdown
Contributor

@wesc wesc commented Apr 1, 2026

The StatisticalChunker implementation requires the statistical package for computing mean, median, and stddev. This PR implements those functions and removes the statistical crate. There are a few reasons to do this:

  1. the statistical crate is ~10 years old and has several dependencies, one of which is rand_os. statistical is likely unmaintained (and in fact, it is almost impossible to Google search when looking for statistics libraries for Rust).
  2. rand_os has a security advisory and has been merged into rand_core: https://rustsec.org/advisories/RUSTSEC-2025-0124
  3. statistical pulls in a few dependencies that aren't necessary
  4. In the wake of the recent high profile supply chain attacks, eliminating unmaintained dependencies is probably a good idea

test_statistical_chunker passes with the new implementations. Note that the old statistical implementations assert when input data doesn't hit a minimum length, and these new versions maintain the same behavior.

Here are the crates that are removed due to cutting out statistical:

  • rand 0.6.5
  • rand_chacha 0.1.1
  • rand_hc 0.1.0
  • rand_isaac 0.1.1
  • rand_jitter 0.1.4
  • rand_os 0.1.3
  • rand_pcg 0.1.2
  • rand_xorshift 0.1.1
  • cloudabi 0.0.3
  • autocfg 0.1.8
  • num 0.2.1
  • num-bigint 0.2.6
  • num-complex 0.2.4
  • num-rational 0.2.4

I generated this PR with the help of Claude Code, and I have reviewed every line of it.

@wesc wesc force-pushed the remove-statistical-dep branch 3 times, most recently from 6011d14 to 4423cbc Compare April 1, 2026 19:58
@wesc wesc force-pushed the remove-statistical-dep branch from 4423cbc to 9d95154 Compare April 1, 2026 20:03
@wesc wesc marked this pull request as ready for review April 1, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant