-
Notifications
You must be signed in to change notification settings - Fork 87
Add packaging guide entry on dependency locking #669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 11 commits
9940eae
cfe2e55
51df0f9
185b382
cd2c12f
a1664bc
88d6b78
71190c2
88f7445
7ad8080
35d8fbf
bd49ceb
b8e862c
7fb2aa9
69d49dd
42d5c8a
68c6446
5e80177
6c5d6fb
1ce130d
dd0e3a7
cab5fd5
cdd4c4a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
|
|
@@ -506,3 +506,187 @@ Why you specify dependencies | |||
| How to specify dependencies | ||||
| When you use different specifiers | ||||
| ::: | ||||
|
|
||||
| ## Dependency Locking | ||||
|
|
||||
| In addition to declaring dependencies in `pyproject.toml`, it is common for | ||||
| packages to lock down exact versions of all their dependencies in a separate | ||||
| lock file. A lock file provides benefits of reproducibility, security, and | ||||
| potentially faster installs, among other things. Pinning the exact dependency | ||||
| versions used in a project eliminates "works on my machine" bugs and gives CI a | ||||
| reproducible baseline. For applications meant to be run rather than imported, | ||||
| lock files also ensure anyone installing the project gets a known-good set of | ||||
| dependencies instead of whatever happens to be latest. | ||||
|
|
||||
| ### `pyproject.toml` vs lock file | ||||
| * `pyproject.toml`: defines all supported environments for users importing | ||||
| your package into their project. | ||||
| * **lock file**: defines a specific environment used for development | ||||
|
|
||||
| :::{admonition} Standardized Lock File | ||||
| :class: note | ||||
| As of March 2025, [PEP 751](https://peps.python.org/pep-0751) defined a standard | ||||
| `pylock.toml` format to unify the various lock file formats in use by other | ||||
| package managers (e.g. `uv.lock`, `poetry.lock`, `pdm.lock`). Most package | ||||
| managers provide ways to generate a PEP 751 compatible file. See [PyPA | ||||
| specification](https://packaging.python.org/en/latest/specifications/pylock-toml/) | ||||
| for up-to-date formatting info on `pylock.toml` | ||||
| ::: | ||||
|
|
||||
| ### How to work with lock files? | ||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think right above this we need a section like "why are lockfiles good" or "why do they exist" to orient people to what they are doing and why before getting into how to work with them. something like "lockfiles create a predictable development environment that helps reduce problems where code runs on one person's machine but crashes on another. together with CI result logs, they are a versioned record of the exact set of code that passed or failed the tests. for applications (as opposed to libraries) that are intended to be used as-is rather than depended on by other packages, they allow someone to install and run it and be confident that it will work, without accidentally installing a more recent version of some dependency that is incompatible"
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added more motivation for lock files in line with your points in the intro paragraph: 88d6b78 |
||||
|
|
||||
| Lock files are not written by hand. Package managers and IDEs provide tools | ||||
| to create, update, and reformat lock files as needed. | ||||
|
|
||||
| 1) **Create** - Package managers often do this automatically though it can be | ||||
| done manually. For example, calling `uv add numpy` will automatically create a | ||||
| `uv.lock` file, setup the environment, and install numpy. | ||||
| 2) **Update** - This is not done automatically by package managers. | ||||
| Maintainers can choose to do this manually or setup their own automated | ||||
| workflow. Updates can be for specific packages or all dependencies. | ||||
| 3) **Reformat** - Package managers currently use native formats (e.g. | ||||
| uv uses `uv.lock`) and provide tools for converting into `pylock.toml` and other | ||||
| formats (e.g. `requirements.txt`) when needed | ||||
|
|
||||
| Below are common package manager CLI workflows for lock files: | ||||
|
|
||||
| ::::{tab-set} | ||||
|
|
||||
| :::{tab-item} uv (recommended) | ||||
| ```sh | ||||
| # Create a uv.lock file based on pyproject.toml | ||||
| > uv lock | ||||
|
|
||||
| # Update uv.lock | ||||
| > uv lock --upgrade | ||||
| > uv lock --upgrade-package pandas | ||||
|
|
||||
| # Install packages into environment based on uv.lock | ||||
| > uv sync | ||||
|
|
||||
| # PEP 751 pylock.toml support | ||||
| > uv export --format pylock.toml -o pylock.toml # export uv.lock -> pylock.toml | ||||
| > uv pip sync pylock.toml # install from pylock.toml | ||||
| ``` | ||||
| See [official docs](https://docs.astral.sh/uv/concepts/projects/sync/) for more details | ||||
| ::: | ||||
|
|
||||
| :::{tab-item} Poetry | ||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not convinced we should be adding a mini-tutorial for every tool. This guide is meant to provide a single happy path for packaging and this seems distracting. We also don't use poetry or pdm anywhere else in the guide (we use hatch, which can use pip or uv underneath, so the uv [pip] section would seem to cover all those cases).
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe adding some data to the comparison table would be a better place
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if we take these out, maybe links to the relevant docs? i think it's useful to see that the same concepts apply across different packaging frontends, but yeah maybe we don't need to write and maintain tutorials for each tool.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Happy to keep things focused. Removed lock file examples but kept links: 6c5d6fb |
||||
| ```sh | ||||
| # Create a poetry.lock file based on pyproject.toml | ||||
| > poetry lock | ||||
|
|
||||
| # Update poetry.lock | ||||
| > poetry update | ||||
| > poetry update pandas numpy | ||||
|
|
||||
| # Install packages into environment based on poetry.lock | ||||
| > poetry sync | ||||
|
|
||||
| ``` | ||||
| PEP 751 pylock.toml not yet supported (track progress on [GitHub](https://github.com/python-poetry/poetry/issues/10356)) | ||||
|
|
||||
| See [official docs](https://python-poetry.org/docs/basic-usage/#installing-dependencies) for more details | ||||
| ::: | ||||
|
|
||||
| :::{tab-item} PDM | ||||
| ```sh | ||||
| # Create a pdm.lock file based on pyproject.toml | ||||
| > pdm lock | ||||
|
|
||||
| # Update pdm.lock | ||||
| > pdm update | ||||
|
|
||||
| # Install packages into environment based on pdm.lock | ||||
| > pdm sync | ||||
|
|
||||
| # PEP 751 pylock.toml support | ||||
| > pdm export -f pylock -o pylock.toml # export pdm.lock -> pylock.toml | ||||
| ``` | ||||
| See [official docs](https://pdm-project.org/latest/usage/lockfile/) for more details | ||||
| ::: | ||||
|
|
||||
| :::: | ||||
|
|
||||
| ### Should I use a lock file? | ||||
|
|
||||
| Most package managers will generate a lock file automatically for you (e.g. uv, | ||||
| Poetry, PDM). The real question is when you version control the lock file as | ||||
| part of your package. | ||||
|
|
||||
| :::{admonition} Rule of Thumb | ||||
| :class: tip | ||||
| If your project is an application others use directly, include a lock file as | ||||
| the recommended environment. | ||||
|
|
||||
| If your project is a library to be used in other projects and it is mature | ||||
| enough to have CI, include a lock file for CI and contributors. | ||||
|
|
||||
| ::: | ||||
|
alexarmstrongvi marked this conversation as resolved.
Outdated
|
||||
|
|
||||
| There is some maintenance cost from lock files. Maintainers should aim to update | ||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we need to address best practices of which lock file to keep in VCS. You mentioned the native and universal formats supported by tools, but at most one should be committed. The question is then which one is our best practice for application developers?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. imo we should recommend pylock bc it's the standard (and using it ubiquitously would definitely be a better situation for ecosystem at large than continuing to use frontend-specific lockfiles)
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed. Added new recommendation box: dd0e3a7 |
||||
| the lock file neither too rarely nor too often. | ||||
| * Too rarely means you risk missing updates with bugfixes, security patches, | ||||
| performance improvements, etc. | ||||
| * Too often means you may introduce bugs or even security vulnerablilites before | ||||
| maintainers of your dependencies catch them. Package managers are starting to | ||||
| support [dependency cooldowns]( | ||||
| https://blog.pypi.org/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/#dependency-cooldowns | ||||
| ) to mitigate this. | ||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. with dep cooldowns, i think only the 'too rare' case is a problem. I would probably frame this as "one risk when developing using lockfiles is that your lockfile will fall too far behind the most recent versions of your dependencies. When people install your package, they typically will not be using your lockfile, and will install the latest versions of the packages supported by their environment. If your dependencies update and break something you rely on, you might not notice it until someone reports it to you."
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. to be more specific, i basically think we should just say "you should use dependency cooldowns" as an unambiguous recommendation, but then on the other end give a bit of discussion to build intuition about why not updating frequently would be bad, which is the more common problem to have.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree "too rare" is the more likely situation. As far as falling behind on package updates, is there anything beyond bugfixes, security patches, and performance improvements that should be mentioned as reasons to update? Regarding people installing your package and ignoring lock files (presumably for the case of a library), I intended the tip at the end about CI testing different environments to address this. A discussion on dependency cooldowns seems best added into sections on building packages and CI as it's something that should be added to tool configs instead of invoked when generating a lock file
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
well the main one w.r.t. keeping lockfiles updated is detecting backwards incompatible API changes in the deps. Different reasons for updating for package consumers at install time vs. updating locked deps at Dev time, and different considerations for libs vs. Apps. But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"
I'll take another full read later to see how reading flow goes, on phone rn
I'm not sure what you mean, dep cooldowns are something that can both be done by the consuming installer (ignoring lockfile) and configured for the lockfile resolver, and both should be done. Since goal here is to give intuition and "best practices" guidance, we should just say "you should configure your package to use dep cooldowns when locking" - I think all the major package managers support this now, I just pulled this into pdm a week or two ago
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Gotcha. Added a new recommendation box: 7fb2aa9
I just meant I don't normally see people call sync with cooldown CLI args so that its not something you have to remember every time you make/update lock files. In either case the recommendation to use cooldowns was added above.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok ya totally, on the same page. Asking ppl to remember a special set of params on every invocation is no good. |
||||
|
|
||||
| When you decide to update a lock file, make sure to test that the resulting | ||||
| environment works before committing. If it fails because of some dependency | ||||
| update, then it may be necessary to update `pyproject.toml` to cap the supported | ||||
| versions of that dependency unless/until the code can be updated to support it. | ||||
|
|
||||
| It can also be good, though not necessary, to double check what changed when | ||||
| updating a lock file. The diff can be noisy so the main changes to focus on are | ||||
| 1) major version updates (e.g. `pandas 2.X.X` -> `pandas 3.X.X`) | ||||
| 2) new transitive dependencies (i.e. not part of your `pyproject.toml`) | ||||
|
|
||||
| :::{tip} | ||||
| A lock file captures one tested environment, not the full compatibility range | ||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ambitiously, yes. More realistically, a lock file can be presumed to be a valid resolution to the set of dependencies that will work with this code. Whether or not testing was done in that environment is not captured by a lock file.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (this might be my fault, i was thinking of this as motivation for lockfile: if you have CI logs by commit, then what you have is a complete versioned record of state of code + dependencies + tests, but that might be too subtle/unnecessary)
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Changed to "A lock file captures one environment for CI testing, ...": cab5fd5 |
||||
| declared in `pyproject.toml`. Projects that use lock files should still have CI | ||||
| test other environments such as | ||||
|
|
||||
| 1) the latest packages consistent with your `pyproject.toml`, subject to | ||||
| dependency cooldowns. This lets you know if a dependency update breaks your | ||||
| package. | ||||
| 2) older supported versions of Python to let you know if a recent change to your | ||||
| package no longer works with an older Python release. | ||||
|
|
||||
| ::: | ||||
|
Comment on lines
+666
to
+677
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have done this, where i have one branch of the tests run with a pip install and the rest run with the lockfile as a sentinel, we might want to add a bit more scaffolding in the form of an example CI action for this. Usually I just to that on linux with latest python.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems good to reference the testing section here and then add an example there. To avoid the combinatoric explosion of maintaining guides on local vs CI testing, for various tools (e.g. nox, hatch, GitHub Actions), with and without lock files, what are your thoughts on making a high level point that testing can setup environments from a lock files instead of resolving at runtime, showing one example replacing
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, good point, agreed. Don't need a full CI example for this, having just one CI example for updating the lockfile is plenty for one page. Just having like a two line thing like As you say is probably plenty. I can imagine that helping generally with "wait when would I do one command vs the other" and "what does pip do/what does uv do" confusion as well.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added a dropdown box giving more detail on cooldowns: 69d49dd I am not familiar with hatch and nox workflows but after a quick search I wasn't seeing how to enable cooldowns or just calling uv instead of pip. The most recent pip releases have a PIP_UPLOADED_PRIOR_TO env var that can be used so perhaps that is the solution to recommend for the Tests section. |
||||
|
|
||||
|
|
||||
| ::::{dropdown} What about `requirements.txt` | ||||
| :icon: info | ||||
| :color: primary | ||||
|
|
||||
| Older approaches to locking used `pip freeze` to generate a `requirements.txt` | ||||
| that got used as a lock file. These are minimal lock files that pin a specific | ||||
| version for the system on which the command was run. They might look like | ||||
| ``` | ||||
| # requirements.txt | ||||
| numpy==2.4.6 | ||||
| plotly==6.7.0 | ||||
| pyzmq==27.1.0 | ||||
| ``` | ||||
|
|
||||
| However, this minimal level of specificity has several downsides making lock | ||||
| files the preferred format: | ||||
| * The versions satisfying `pyproject.toml` may differ between your Windows | ||||
| laptop and the Linux server your CI runs on. A single lock file contains the | ||||
| information needed to build platform specific and Python version specific | ||||
| environments. In contrast, a separate `requirements.txt` files is needed to | ||||
| store this information (e.g. `requirements.ci.txt`, | ||||
| `requirements.py313-macos.txt`) | ||||
| * Packages can get updated without a version update for both legitimate and | ||||
| malicious reasons. Lock files include package hashes to catch this. A hash | ||||
|
sneakers-the-rat marked this conversation as resolved.
Outdated
|
||||
| number is a unique signature computed from the code and any change to the code | ||||
|
sneakers-the-rat marked this conversation as resolved.
Outdated
|
||||
| will cause the release to have a different hash even if is given the same | ||||
| release version number. | ||||
| * Other metadata determined during resolution of `pyproject.toml` (e.g. which | ||||
| dependencies are transitive, where the packages were downloaded from, etc.) that | ||||
| can help speed up future installs is lost. | ||||
|
|
||||
| :::: | ||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like your motivation section, a few notes on the small details, take these as suggestions
The way I see it, pyproject should strive to eliminate false-negatives, denying users the ability to install software because is doesn't fit their environment; this will create some false-positives, where it was installed but won't work. pylock, on the other hand, should reduce false-positives, if it installs the environment is known-good, but this disallows installing into unknown-good environments and restricts deployment possibilities.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 here, this is a good framing.
i am not sure i am getting your point in the second bullet, but maybe one distinction (that i don't think is being made yet) is between application-like things and library-like things, where when i am using an application-like thing i do indeed run from the lockfile in deployment, but not in library-like things (mostly as a byproduct of how python's import machinery/historical reasons/etc. that's mostly a digression). not sure if that would be a useful distinction or TMI as a motivating example of how lockfiles are used/how their use depends on how the package is used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also like the false-positive/false-negative framing. Added a paragraph below the bullet summary to give more nuance. 68c6446
By "supporting" I didn't intend it to mean "validated via testing" so I also clarified that "supporting" is about a commitment from maintainers (e.g. to respond to issue tickets related to something not working that pyproject.toml says should work)
Regarding the second bullet, I wouldn't say a lock file is for deploying and releasing. I see your example as highlighting that a lock file is worked on most during deploying and releasing so that it can be used for development and CI going forward.