Restore symlinked Python source traversal#3168
Conversation
Follow symlinked directories when collecting Python package files for wheels and source distributions. This restores support for layouts that share Python code through symlinked package directories instead of duplicating files.
There was a problem hiding this comment.
Pull request overview
This PR restores traversal of symlinked directories when maturin collects Python package sources, so that files inside symlinked folders are included in both wheel and sdist outputs (matching the described historical behavior).
Changes:
- Enable symlink-following in wheel Python source collection (
ModuleWriter::write_python_part). - Enable symlink-following in sdist Python source collection (
add_python_sources). - Add a Unix-only regression test to ensure wheel builds include files inside a symlinked Python directory.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
tests/run/wheel.rs |
Adds a regression test covering inclusion of files inside symlinked Python directories in wheels. |
src/source_distribution/pyproject.rs |
Changes sdist Python-source walking to follow symlinked directories. |
src/module_writer/mod.rs |
Changes wheel Python-source walking to follow symlinked directories. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Skip `ignore` symlink-loop errors while following symlinked Python source directories so cyclic links do not fail wheel or sdist builds. Simplify the wheel walk error handling now that the walker is rooted at each Python package, and share the symlink fixture across wheel and sdist tests.
| // Walk each python package directly (rather than rooting the walker at | ||
| // `project_root` and filtering): this keeps `follow_links(true)` scoped to | ||
| // symlinks inside python packages, instead of also descending into unrelated | ||
| // symlinked trees under the project (e.g. inside `target/` or `.venv/`). | ||
| for package in &python_packages { | ||
| for absolute in WalkBuilder::new(package) | ||
| .hidden(false) | ||
| .parents(false) | ||
| .git_global(false) | ||
| .git_exclude(false) | ||
| .follow_links(true) | ||
| .build() |
| for package in &python_packages { | ||
| for absolute in WalkBuilder::new(package) | ||
| .hidden(false) | ||
| .parents(false) | ||
| .git_global(false) | ||
| .git_exclude(false) | ||
| .follow_links(true) | ||
| .build() |
| for package in python_packages { | ||
| for entry in ignore::Walk::new(package) { | ||
| let source = entry?.into_path(); | ||
| for entry in ignore::WalkBuilder::new(package).follow_links(true).build() { | ||
| let source = match entry { | ||
| Ok(entry) => entry.into_path(), | ||
| Err(err) => { | ||
| if is_symlink_loop_error(&err) { | ||
| warn!( | ||
| "Skipping symlink loop in Python package source tree while building source distribution: {err}" | ||
| ); | ||
| continue; | ||
| } | ||
| return Err(err.into()); | ||
| } |
|
@Bben01 Could you clarify where your shared Python sources live relative to the project checkout? The current PR follows symlinked Python package directories, but there is a safety tradeoff: following symlinks can package files outside the project if a link points somewhere broad or unexpected. One possible rule is to allow symlink targets that stay inside the same git repository/worktree, but reject targets outside it. Would that match your setup, or are your shared Python sources outside the git repository that contains the maturin project? |
|
Yes, the sources are inside a big monorepo, and we build from that multiple packages, so it could work. I currently don't have access to a computer but will do it as soon as possible. If you prefer I could just make it explicit using a config flag in the pyproject.toml |
Summary
Restore traversal of symlinked directories when maturin collects Python package sources for wheels and sdists.
I dont know if it was removed intentional or not. If it was, I could add some bool to the pyproject.toml that could toggle this behavior (and set the current default to false)
Why
In our team we package multiple Python packages that share some common code. We use symlinks for those shared directories and files so the source is maintained in one place instead of duplicated across package trees.
This worked previously (from what i remember - 1.7.0), but recent behavior no longer followed symlinked directories while building a pyo3 Rust + Python layout. Symlinked files were still included, but files inside symlinked directories were omitted from the wheel.
Fix
Use
follow_links(true)when walking Python package sources for both wheel and sdist generation.Validation
linked_dir/nested.pyis missing from the wheel.cargo test --test run -- pyo3_mixed_py_subdir_includes_symlinked_python_files.cargo fmt --allandgit diff --check.AI disclosure
Fix was written by codex 5.5