Python Requirements

requirements.in files contain the first-order requirements. It is split into requirements required for production, and those used in development. If you want to change requirements, change this file.

requirements.txt includes the exact versions and package hashes for the first-order requirements as well as the requirements-of-requirements. This is the file used with pip install to install packages in the Docker image. pip-compile, from pip-tools, generates and maintains this file using requirements.in.

requirements-docs.txt has the requirements used on ReadTheDocs to build the Ichnaea documentation on each merge to main. Since this is a non-production environment, we neither pin nor hash the requirements.

Compiling requirements.txt

After making changes, the .in files can be compiled to .txt output files by running:

make update-reqs

This will start a Docker container and run pip-compile with the proper options. Running in the container ensures that the correct dependencies are chosen for the Docker environment, rather than your host environment.

There will be a warnings at the end of the process:

The generated requirements file may be rejected by pip install. See # WARNING lines for details.

This is expected. pip and setuptools are provided by the container, and should not be pinned.

To apply the new requirements, rebuild your Docker image:

make build

Automated Updates

Note

Dependabot is currently broken, see issue #1428

Dependabot opens PRs for updates around the first of the month. It also opens PRs for security updates when they are available. It seems to have some support for pip-tools, but it may be necessary to manually run make update-reqs to correctly regenerate the requirements.

paul-mclendahand is useful for packaging several PRs into a single PR, and avoiding the rebase / rebuild / test cycle when merging one Dependabot PR at a time.

Manually upgrading requirements.txt

To upgrade all the requirements, run make shell to enter the Docker environment, and run

CUSTOM_COMPILE_COMMAND="make update-reqs" pip-compile --generate-hashes --upgrade

To upgrade a single package, run this instead:

CUSTOM_COMPILE_COMMAND="make update-reqs" pip-compile --generate-hashes --upgrade-package <package-name>

You’ll need to exit the Docker environment and run make build to recreate the Docker image with your changes.

Other tools

pipdeptree displays the requirements tree, which can be useful to determine which package required an unknown package.

hashin is useful for generating a list of hashes. We used it exclusively before pip-compile, and it may be handy if you need to manually update a .txt file.