Skip to content

fix: detect site-packages clobbering of conda packages by PyPI wheels#6344

Merged
baszalmstra merged 2 commits into
prefix-dev:mainfrom
tdejager:fix-site-packages-clobber-detection
Jun 11, 2026
Merged

fix: detect site-packages clobbering of conda packages by PyPI wheels#6344
baszalmstra merged 2 commits into
prefix-dev:mainfrom
tdejager:fix-site-packages-clobber-detection

Conversation

@tdejager

@tdejager tdejager commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Description

While user-testing #6333 I expected an explicit "this conda package is not a PyPI package" mapping to trigger the clobber warning from #6292 (conda boltons + PyPI boltons installed over each other), but nothing showed up. Digging in, it turns out the clobber detection never worked for regular site-packages files — only for PEP 427 .data/<scheme>/ files.

Before this PR, this is what happened:

  1. Conda's paths.json gives us prefix-relative paths, the wheel RECORD gives us site-packages-relative paths:
    paths.json:    lib/python3.12/site-packages/boltons/__init__.py
    wheel RECORD:  boltons/__init__.py
    
  2. To compare the two, we joined the RECORD entry with the wheel's destination directory, which comes from interpreter().virtualenv() and is relative:
    lib/python3.12/site-packages + boltons/__init__.py
    = lib/python3.12/site-packages/boltons/__init__.py
    
  3. We then called .strip_prefix("/proj/.pixi/envs/default") on it. An absolute prefix never matches a relative path, so this always failed and the entry was silently continued — even though the path from step 2 already equals the paths.json key.
  4. .data/<scheme>/ entries joined with an absolute directory instead, so the strip worked there:
    prek-0.4.4.data/scripts/prek -> /proj/.pixi/envs/default/bin/prek
    strip_prefix -> bin/prek   matches
    
    That's why the bin/prek example from feat: Improve PyPI > Conda clobber detection #6292 warned, but site-packages files never did.

The fix (after review discussion, see below): derive all destinations from interpreter().layout().scheme — the exact layout uv's installer writes with — relative-ized against the interpreter's own sys_prefix. Prediction and writes then share one source and cannot drift, and the whole comparison happens in conda's prefix-relative form: no strip_prefix against an externally-supplied root anymore. Entries that escape the prefix (a leading .. after normalization, or an absolute RECORD entry) are skipped; entries that merely escape site-packages (like prek's bin/prek) normalize into the prefix and compare fine.

With the fix this now warns again for the common case:

$ pixi install

 WARN PyPI package files will overwrite files installed by conda packages:
  - PyPI package 'boltons' overwrites conda package 'boltons':
    - lib/python3.12/site-packages/boltons/__init__.py
    - lib/python3.12/site-packages/boltons/cacheutils.py
    - ... 34 other files

Some smaller things in here as well:

  1. get_wheel_info now returns the wheel kind (purelib/platlib) instead of a scheme directory; the destination struct holds prefix-relative paths only, so the absolute/relative mix that hid the bug is gone entirely.
  2. The unit-test fixture used an absolute site-packages path, which is the production-convention mismatch that let the broken code pass its tests. It now uses the relative form, and there is a regression test for the plain pkg/__init__.py case.
  3. The check had three layers of silence (swallowed Err at the call site, two silent continues per wheel); these now log at debug level so a dead check can't hide again.
  4. The prefix-relative form is now a CondaPrefixPath newtype: the registry is keyed by it and the only constructors are "from a conda paths.json entry" and "from a wheel RECORD entry", so the two sides of the comparison can no longer disagree about the path convention without failing to compile.

Reproducer

// mapping.json
{ "boltons": null }
[workspace]
name = "clobber-repro"
channels = ["https://conda.anaconda.org/conda-forge"]
platforms = ["osx-arm64"]
conda-pypi-map = { "https://conda.anaconda.org/conda-forge" = "mapping.json" }

[dependencies]
python = "3.12.*"
boltons = "*"

[pypi-dependencies]
rich = "*"
boltons = "*"

pixi install warns with this PR and is silent on main. Once #6333 lands, boltons = false in an inline mapping is the one-liner version of this.

How Has This Been Tested?

All unit tests in pixi_install_pypi pass, including the new regression test; the two existing prefix-escape tests keep their exact expectations under the corrected fixture. I also ran the reproducer above against a debug build and verified the warning appears with the overlapping site-packages files listed.

@baszalmstra asked whether this respects the python package's python_site_packages_dir. After some back and forth this led to a better design: the check now derives its destinations from interpreter().layout().scheme, which is literally the layout uv installs with (pixi constructs the environment via Interpreter::query + PythonEnvironment::from_interpreter, so the probed sysconfig scheme is what both the installer and the check see). That makes the prediction-matches-writes property an identity instead of a convention, and a relocated site-packages flows through end to end. The record key itself is never consulted — paths.json is already the ground truth of where conda's files are, and the layout is the ground truth of where uv writes; the check is the intersection of the two.

Verified with python-freethreading (the case the key exists for), where the warning correctly reports lib/python3.13t/site-packages/..., alongside the regular boltons and prek reproducers.

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.

Tools: Claude Code with Fable 5

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added sufficient tests to cover my changes.

@tdejager tdejager force-pushed the fix-site-packages-clobber-detection branch 3 times, most recently from 31b622a to b188175 Compare June 11, 2026 07:42
@tdejager tdejager marked this pull request as ready for review June 11, 2026 08:00
@tdejager tdejager requested review from baszalmstra and nichmor June 11, 2026 08:26
@tdejager tdejager force-pushed the fix-site-packages-clobber-detection branch from b188175 to 99b94df Compare June 11, 2026 08:35
The conda-pypi clobber check compared wheel RECORD paths against conda's
paths.json entries by joining regular wheel files onto the wheel's
destination site-packages directory and then stripping the prefix root.
That directory comes from the interpreter's *virtualenv scheme*, which is
relative to the prefix (e.g. lib/python3.12/site-packages), so the
strip_prefix of an absolute prefix root always failed and every regular
wheel file was silently skipped: site-packages clobbering (the common
case) was never detected. Only PEP 427 .data/<scheme> files, which are
joined onto the interpreter's absolute scheme directories, could match —
which is why script clobbering (e.g. bin/prek) worked.

The relative form is exactly the prefix-relative shape conda's paths.json
uses, so accept it directly (rejecting paths that escape the prefix) and
keep the prefix strip for the absolute .data paths.

Also rename the misleading root_scheme field to site_packages with a doc
comment on the relative convention, debug-log the previously silent skip
and error paths, and align the unit-test fixture with the production
(relative) convention; the old fixture used an absolute path, which is
why the tests did not catch this. Reproduced with a conda + pypi boltons
overlap: the warning now lists the overwritten site-packages files.
@tdejager tdejager force-pushed the fix-site-packages-clobber-detection branch from 99b94df to c26fcb6 Compare June 11, 2026 08:36
Instead of joining regular wheel files onto the interpreter's virtualenv
scheme *template* and .data files onto individually-listed absolute
scheme dirs, derive all destinations from interpreter().layout().scheme —
the exact layout uv's installer writes with — relative-ized against the
interpreter's own sys_prefix. Prediction and writes now share one source
and cannot drift; a custom site-packages location (python_site_packages_dir)
flows through because the prediction is the interpreter's real layout,
not the venv template. The check no longer needs the pixi-supplied
prefix root at all: everything is compared in conda's prefix-relative
form, and entries escaping the prefix (leading .. after normalization,
or absolute RECORD entries) are skipped.

get_wheel_info now returns the wheel kind instead of a scheme directory,
and the kind selects between the purelib/platlib destinations.

User-tested: boltons site-packages clobber (python 3.12 and free-threaded
3.13t, which relocates site-packages) and the prek bin/prek scripts
clobber all warn correctly.
@tdejager tdejager force-pushed the fix-site-packages-clobber-detection branch from d2ea2ca to 76ef184 Compare June 11, 2026 11:53
@baszalmstra baszalmstra merged commit 54e0f85 into prefix-dev:main Jun 11, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants