feat: add PyPI dependencies to pixi global environments#6334
Open
wolfv wants to merge 8 commits into
Open
Conversation
Add an install-pypi operation to the CommandDispatcher so that PyPI packages can be installed into a conda prefix through the same handle that already drives conda solves and installs. This makes the PyPI install pipeline reusable outside the workspace install path, e.g. for pixi global. - Extract UvReporter/UvReporterOptions into a new pixi_uv_reporter crate. pixi_reporters depends on pixi_command_dispatcher, so the reporter had to move below the dispatcher to let it depend on pixi_install_pypi without a cycle. pixi_reporters re-exports the types so existing users are unaffected. - Add InstallPypiEnvironmentSpec and CommandDispatcher::install_pypi_environment, which wraps PyPIEnvironmentUpdater and derives the wheel link mode from the dispatcher's configured link options. - Switch the workspace install path in pixi_core to build the spec and go through the dispatcher instead of wiring up the updater configs inline.
Complete the PyPI refactor started with install: resolution now also runs through the CommandDispatcher, so both halves of the PyPI story (solve + install) are reachable outside the workspace code, e.g. for pixi global. - Move the resolve pipeline (resolve_pypi, LazyBuildDispatch, CondaResolverProvider) from pixi_core::lock_file::resolve into pixi_install_pypi::resolve. - Decouple it from the workspace with a new CondaPrefixProvider trait: when uv must build an sdist to get metadata, the provider supplies a conda prefix (python interpreter + activation env vars) on demand. pixi_core implements it as WorkspaceCondaPrefixProvider wrapping the memoized CondaPrefixUpdater and environment activation; both the lock-file update path and the satisfiability metadata checks use it. - Move PypiPackageIdentifier into pixi_install_pypi; the satisfiability-specific satisfies() check stays in pixi_core as an extension trait. - Add SolvePypiEnvironmentSpec and CommandDispatcher::solve_pypi_environment, deriving the link mode from the dispatcher's configured link options. - Drop the dead conda_task plumbing: LazyBuildDispatch::conda_task was never assigned, so resolve_pypi now returns just the locked records and the PypiGroupSolved task keeps forwarding None. - Remove the uv-* dependencies pixi_core no longer needs.
rattler_digest and uv-preview were only used by the PyPI resolve code that moved to pixi_install_pypi.
Global environments can now declare PyPI packages in the manifest:
[envs.jupyter]
channels = ["conda-forge"]
dependencies = { python = "3.13.*" }
pypi-dependencies = { jupyterlab = "*" }
After the conda packages are installed, the declared PyPI packages are
resolved with CommandDispatcher::solve_pypi_environment and installed
into the prefix's site-packages with install_pypi_environment — the
first consumer of the dispatcher PyPI pipeline outside the workspace.
The CondaPrefixProvider implementation simply hands out the freshly
installed prefix when a source distribution must be built during
resolution.
Sync detection treats an environment as out of sync when a declared
PyPI package has no dist-info in site-packages (name presence only;
spec changes for an installed package are reconciled on the next
reinstall), and update operations always re-resolve environments with
PyPI dependencies.
PyPI dependencies of global environments can now be managed from the command line instead of only by editing the manifest: - pixi global install python --pypi httpx --pypi "flask>=2" - pixi global add --environment env --pypi httpx - pixi global remove --environment env httpx (falls back to the PyPI dependency when the name is not a conda dependency) The new Manifest::add_pypi_dependency/remove_pypi_dependency methods edit the [envs.*.pypi-dependencies] table. Removing the last PyPI dependency now also cleans pixi-installed packages out of the prefix's site-packages: leftover detection (dist-info INSTALLER == uv-pixi) marks the environment out of sync and the installer removes the packages on the next sync.
PyPI-installed executables are now treated like conda executables when exposing binaries from a global environment: - Executables are enumerated from each pixi-installed distribution's dist-info RECORD (entries that land in the prefix's binary folders). Declared pypi-dependencies are auto-exposed like direct conda dependencies; executables of transitive distributions are not auto-exposed but are protected from stale-mapping pruning. - pixi global remove unexposes the removed package's scripts (using a snapshot taken before removal) and forces a reinstall: a removed PyPI dependency cannot be detected by the presence-based sync check since transitive distributions are legitimately undeclared, so the installer is invoked explicitly to drop the now-extraneous distribution.
The macOS pytest job of the previous run was cancelled by runner infrastructure.
The PyPI executable enumeration added an extra find_installed_packages call (a full parse of every conda-meta record) to executables_of_all_dependencies and ran another one per expose sync in executables_of_pypi_dependencies, roughly doubling the runtime of the global integration test suite and pushing the macOS pytest job past its 10 minute timeout. Reuse the prefix records that executables_of_all_dependencies already loads, and skip the prefix entirely in executables_of_pypi_dependencies when the environment declares no PyPI dependencies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds support for PyPI dependencies in
pixi globalenvironments, built on top of the command-dispatcher refactor.Manifest support:
PyPI packages are resolved against the environment's python interpreter (via
CommandDispatcher::solve_pypi_environment) and installed into its site-packages (viainstall_pypi_environment) after the conda packages.CLI support:
pixi global install python --pypi httpx --pypi "flask>=2"pixi global add --environment env --pypi httpxpixi global remove --environment env httpx(falls back to PyPI deps when the name is not a conda dependency)Behavior details:
dist-info/RECORD); declared deps auto-expose, transitive ones don't.pixi global removeof a PyPI dep forces a reinstall so the now-extraneous distribution is uninstalled, and unexposes its scripts.pixi global updatealways re-resolves environments with PyPI deps.How Has This Been Tested?
--pypi(script exposed and runnable), add, partial remove (only the removed package is uninstalled), last-dep remove (site-packages cleanup + unexpose), and no-op re-sync.https://claude.ai/code/session_01S4nY8g4frki9JvtuUnZm85
Generated by Claude Code