Skip to content
Open
Changes from 2 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9940eae
Added packaging guide entry on dependency locking
May 22, 2026
cfe2e55
Merge remote-tracking branch 'origin/main' into docs/491-add-packagin…
May 23, 2026
51df0f9
Clarify pyproject.toml vs lock file
May 23, 2026
185b382
maintained to written
May 23, 2026
cd2c12f
Prose description of lock file operations
May 23, 2026
a1664bc
Clarify checking diff of lock file is not necessary
May 23, 2026
88d6b78
Added more justification for lock files in intro paragraph
May 23, 2026
71190c2
Elaborate on package hash
May 23, 2026
88f7445
Clarified lock files supporting multiple platforms and python version…
May 23, 2026
7ad8080
maintain changed to generate
May 23, 2026
35d8fbf
Typos and other wording fixes
May 23, 2026
bd49ceb
Add pdm update package example
May 24, 2026
b8e862c
Clarified carve out for not including lock file
May 24, 2026
7fb2aa9
Add recommendation for updating lock files
May 24, 2026
69d49dd
Add more details on dependency cooldowns
May 24, 2026
42d5c8a
Add link for hash
May 24, 2026
68c6446
Elaborate on pyproject vs lock file in intro
May 31, 2026
5e80177
Merge remote-tracking branch 'upstream/main' into docs/491-add-packag…
May 31, 2026
6c5d6fb
Remove poetry and pdm lock file examples
May 31, 2026
1ce130d
Change lock to sync in uv example
May 31, 2026
dd0e3a7
Recommend pylock.toml format
May 31, 2026
cab5fd5
Clarified tip about lock file and multiple CI testing environments
May 31, 2026
cdd4c4a
Merge branch 'main' into docs/491-add-packaging-guide-entry-on-depend…
alexarmstrongvi Jun 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 156 additions & 0 deletions package-structure-code/declare-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -506,3 +506,159 @@ Why you specify dependencies
How to specify dependencies
When you use different specifiers
:::

## Dependency Locking

In addition to declaring dependencies in `pyproject.toml`, it is common for
packages to lock down exact versions of all their dependencies in a separate
lock file. A lock file provides benefits of reproducibilty, security, and
potentially faster installs, among other things.

### `pyproject.toml` vs lock file
* `pyproject.toml`: provides the range of requirements for others to
use your package in their project.
Comment thread
sneakers-the-rat marked this conversation as resolved.
Outdated
* **lock file**: records a fully resolved environment for working with or on
your package directly

:::{admonition} Official Lock File
:class: note
As of March 2025, [PEP 751](https://pep.python.org/pep-0751) defined a standard
`pylock.toml` format to unify the various lock file formats in use by other
package managers (e.g. `uv.lock`, `poetry.lock`, `pdm.lock`). Most package
managers provide ways to generate a PEP 751 compatible file. See [PyPA
specification](https://packaging.python.org/en/latest/specifications/pylock-toml/)
for up-to-date formatting info on `pylock.toml`
:::

### How to work with lock files?
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think right above this we need a section like "why are lockfiles good" or "why do they exist" to orient people to what they are doing and why before getting into how to work with them.

something like "lockfiles create a predictable development environment that helps reduce problems where code runs on one person's machine but crashes on another. together with CI result logs, they are a versioned record of the exact set of code that passed or failed the tests. for applications (as opposed to libraries) that are intended to be used as-is rather than depended on by other packages, they allow someone to install and run it and be confident that it will work, without accidentally installing a more recent version of some dependency that is incompatible"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more motivation for lock files in line with your points in the intro paragraph: 88d6b78


Lock files are not maintained manually. Package managers and IDEs provide tools
Comment thread
sneakers-the-rat marked this conversation as resolved.
Outdated
to create, update, and reformat lock files as needed. Below are common package
manager CLI workflows for lock files:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe before the examples, we give a sense of the kinds of operations that need to be done in prose:

  • create the lockfile
  • update the lockfile
  • update a single package in the lockfile
  • creating different forms of the lockfile for different python versions, OSes, etc.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed: cd2c12f
Doesn't seem like PDM supports updating a single package in the lockfile like uv and poetry apart from some workaround using the --constraint arg so I left that out of the examples. I figure people will go to the linked docs if they really are going to use lock files and want to see tool specific workflows.

Copy link
Copy Markdown
Contributor

@sneakers-the-rat sneakers-the-rat May 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does!

https://pdm-project.org/en/latest/reference/cli/#update_2

Positional Arguments:

packages: If packages are given, only update them

It looks like the links in the docs are a little wonky, where the narrative docs link to the wrong place in the CLI docs.

I'll double check this later when at keyboard

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. The docs said "Update package(s) in pyproject.toml" instead of in lock file so I skipped over that but I see now what it means. Added: bd49ceb

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs are def a little messy on pdm, trying to help out a bit with that atm


::::{tab-set}

:::{tab-item} uv (recommended)
```sh
# Create a uv.lock file based on pyproject.toml
> uv lock

# Update uv.lock
> uv lock --upgrade

# Install packages into environment based on uv.lock
> uv sync

# PEP 751 pylock.toml support
> uv export --format pylock.toml -o pylock.toml # export uv.lock -> pylock.toml
> uv pip sync pylock.toml # install from pylock.toml
```
See [official docs](https://docs.astral.sh/uv/concepts/projects/sync/) for more details
:::

:::{tab-item} Poetry
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced we should be adding a mini-tutorial for every tool. This guide is meant to provide a single happy path for packaging and this seems distracting. We also don't use poetry or pdm anywhere else in the guide (we use hatch, which can use pip or uv underneath, so the uv [pip] section would seem to cover all those cases).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe adding some data to the comparison table would be a better place

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we take these out, maybe links to the relevant docs? i think it's useful to see that the same concepts apply across different packaging frontends, but yeah maybe we don't need to write and maintain tutorials for each tool.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to keep things focused. Removed lock file examples but kept links: 6c5d6fb

```sh
# Create a poetry.lock file based on pyproject.toml
> poetry lock

# Update poetry.lock
> poetry update

# Install packages into environment based on poetry.lock
> poetry sync

```
PEP 751 pylock.toml not yet supported (track progress on [GitHub](https://github.com/python-poetry/poetry/issues/10356))

See [official docs](https://python-poetry.org/docs/basic-usage/#installing-dependencies) for more details
:::

:::{tab-item} PDM
```sh
# Create a pdm.lock file based on pyproject.toml
> pdm lock

# Update pdm.lock
> pdm update

# Install packages into environment based on pdm.lock
> pdm sync

# PEP 751 pylock.toml support
> pdm export -f pylock -o pylock.toml # export pdm.lock -> pylock.toml
> pdm lock --lockfile pylock.toml # install from pylock.toml
```
See [official docs](https://pdm-project.org/latest/usage/lockfile/) for more details
:::

::::

### Should I use a lock file?

Most package managers will maintain a lock file automatically for you (e.g. uv,
Comment thread
sneakers-the-rat marked this conversation as resolved.
Outdated
Poetry, PDM). The real question is when you version control the lock file as
part of your package.

:::{admonition} Rule of Thumb
:class: tip
If your project is an application others use directly, include a lock file as
the recommended environment.

If your project is a library to be used in other projects and it is mature
enough to have CI, include a lock file for CI and contributors.

:::
Comment thread
alexarmstrongvi marked this conversation as resolved.
Outdated

There is some maintenance cost from lock files. Maintainers should aim to update
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to address best practices of which lock file to keep in VCS. You mentioned the native and universal formats supported by tools, but at most one should be committed. The question is then which one is our best practice for application developers?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo we should recommend pylock bc it's the standard (and using it ubiquitously would definitely be a better situation for ecosystem at large than continuing to use frontend-specific lockfiles)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Added new recommendation box: dd0e3a7

the lock file neither too rarely nor too often.
* Too rarely means you risk missing updates with bugfixes, security patches,
performance improvements, etc.
* Too often means you may introduce bugs or even security vulnerablilites before
maintainers of your dependencies catch them. Package managers are starting to
support [dependency cooldowns](
https://blog.pypi.org/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/#dependency-cooldowns
) to mitigate this.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with dep cooldowns, i think only the 'too rare' case is a problem. I would probably frame this as "one risk when developing using lockfiles is that your lockfile will fall too far behind the most recent versions of your dependencies. When people install your package, they typically will not be using your lockfile, and will install the latest versions of the packages supported by their environment. If your dependencies update and break something you rely on, you might not notice it until someone reports it to you."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be more specific, i basically think we should just say "you should use dependency cooldowns" as an unambiguous recommendation, but then on the other end give a bit of discussion to build intuition about why not updating frequently would be bad, which is the more common problem to have.

Copy link
Copy Markdown
Author

@alexarmstrongvi alexarmstrongvi May 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree "too rare" is the more likely situation. As far as falling behind on package updates, is there anything beyond bugfixes, security patches, and performance improvements that should be mentioned as reasons to update?

Regarding people installing your package and ignoring lock files (presumably for the case of a library), I intended the tip at the end about CI testing different environments to address this.

A discussion on dependency cooldowns seems best added into sections on building packages and CI as it's something that should be added to tool configs instead of invoked when generating a lock file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reasons to update

well the main one w.r.t. keeping lockfiles updated is detecting backwards incompatible API changes in the deps. Different reasons for updating for package consumers at install time vs. updating locked deps at Dev time, and different considerations for libs vs. Apps. But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

I intended the tip at the end about CI testing different environments to address this.

I'll take another full read later to see how reading flow goes, on phone rn

it's something that should be added to tool configs instead of invoked when generating a lock file

I'm not sure what you mean, dep cooldowns are something that can both be done by the consuming installer (ignoring lockfile) and configured for the lockfile resolver, and both should be done. Since goal here is to give intuition and "best practices" guidance, we should just say "you should configure your package to use dep cooldowns when locking" - I think all the major package managers support this now, I just pulled this into pdm a week or two ago

Copy link
Copy Markdown
Author

@alexarmstrongvi alexarmstrongvi May 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for the sake of this section I think we can give simple guidance like "use dep cooldowns and update & test lockfile frequently"

Gotcha. Added a new recommendation box: 7fb2aa9

I'm not sure what you mean

I just meant I don't normally see people call sync with cooldown CLI args uv sync --exclude-newer "3 days" and instead put it in a user or project config

[tool.uv]
exclude-newer = "3 days"

so that its not something you have to remember every time you make/update lock files. In either case the recommendation to use cooldowns was added above.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok ya totally, on the same page. Asking ppl to remember a special set of params on every invocation is no good.


When you decide to update a lock file, consider what changed before committing
it to the project. Good changes to focus on are
1) major version updates (e.g. `pandas 2.X.X` -> `pandas 3.X.X`)
2) new transitive dependencies (i.e. not part of your `pyproject.toml`)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't read the lockfile when it updates, and so we might not want to give the impression to newbies that it's normal to read lockfile changes since it can be daunting. My pattern is basically "update the lockfile, run the tests." maybe i'll keep a loose eye for transitive dependencies if e.g. something suddenly starts pulling in a huge dep like scipy or something. maybe the thing to communicate is like "what to do if you update and a new version breaks your stuff" and how to handle adding e.g. temporary version caps

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified its not necessary and that testing is more important: a1664bc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the changed language, when others get around to reviewing this I imagine there will be more said on version capping, which is a hot topic for us lol, but this is one case where it makes sense - the version range should reflect the range of versions that work for a given state of the package. Ideally one would just adapt to the changes in the upstream dep (and maybe scooch up the dep version floor if the changes aren't backwards compatible), but cutting a release with a version cap temporarily until you can make the fix so that it actually runs when people install it is a reasonable case for a cap.


:::{tip}
A lock file captures one tested environment, not the full compatibility range
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ambitiously, yes. More realistically, a lock file can be presumed to be a valid resolution to the set of dependencies that will work with this code. Whether or not testing was done in that environment is not captured by a lock file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(this might be my fault, i was thinking of this as motivation for lockfile: if you have CI logs by commit, then what you have is a complete versioned record of state of code + dependencies + tests, but that might be too subtle/unnecessary)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "A lock file captures one environment for CI testing, ...": cab5fd5

declared in `pyproject.toml`. Projects that use lock files should still have CI
test other environments such as

1) the latest packages consistent with your `pyproject.toml`, subject to
dependency cooldowns. This lets you know if a dependency update breaks your
package.
2) older supported versions of Python to let you know if a recent change to your
package no longer works with an older Python release.
:::
Comment on lines +666 to +677
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have done this, where i have one branch of the tests run with a pip install and the rest run with the lockfile as a sentinel, we might want to add a bit more scaffolding in the form of an example CI action for this. Usually I just to that on linux with latest python.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good to reference the testing section here and then add an example there. To avoid the combinatoric explosion of maintaining guides on local vs CI testing, for various tools (e.g. nox, hatch, GitHub Actions), with and without lock files, what are your thoughts on making a high level point that testing can setup environments from a lock files instead of resolving at runtime, showing one example replacing run: python -m pip install ... with run: uv sync --frozen, and the linking to documentation for various tools?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good point, agreed. Don't need a full CI example for this, having just one CI example for updating the lockfile is plenty for one page. Just having like a two line thing like

# install most recent dependencies compatible with [project.dependencies]
python -m pip install .

# install exact versions in lockfile
uv sync --frozen

As you say is probably plenty. I can imagine that helping generally with "wait when would I do one command vs the other" and "what does pip do/what does uv do" confusion as well.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a dropdown box giving more detail on cooldowns: 69d49dd

I am not familiar with hatch and nox workflows but after a quick search I wasn't seeing how to enable cooldowns or just calling uv instead of pip. The most recent pip releases have a PIP_UPLOADED_PRIOR_TO env var that can be used so perhaps that is the solution to recommend for the Tests section.



::::{dropdown} What about `requirements.txt`
:icon: info
:color: primary

Older approaches to locking used `pip freeze` to generate a `requirements.txt`
that got used as a lock file. These are minimal lock files that pin a specific
version for the system on which the command was run. They might look like
```
# requirements.txt
numpy==2.4.6
plotly==6.7.0
pyzmq==27.1.0
```

However, this minimal level of specificity has several downsides making lock
files the preferred format:
* The versions satisfying `pyproject.toml` may differ between your MacOS and the
Linux server your CI runs on. Lock files contain platform-specific resolutions
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

platform and python version deps are the main things that my lockfiles vary by: maybe generalized like "lockfiles can be customized to include all the different versions that are applicable across different operating systems, python versions, and other platform markers, while that information can only be stored in requirements.txt files through file naming conventions."

idk that text isn't very clear either, but something like that

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified: 88f7445

* Packages can get updated without a version update for both legitimate and
malicious reasons. Lock files specify exact hashes to catch this.
Comment thread
sneakers-the-rat marked this conversation as resolved.
Outdated
* Other metadata determined during resolution of `pyproject.toml` (e.g. which
dependencies are transitive, where the packages were downloaded from, etc.) that
can help speed up future installs is lost.

::::