New tools for managing Python dependencies in edx-platform

Development
,
1

Summary

The Arch-BOM team at 2U has added some new GitHub Actions (and Make targets) that can automate some common tasks around Python dependencies in edx-platform, and there’s a new CI check that will help keep Python dependencies consistent.

Documentation is in the repo’s requirements/README.rst.

Background

If you work with the edx-platform repo, you’re almost certainly aware of some difficulties in updating Python dependencies:

  • Adding or updating a single dependency used to require running make upgrade, which changes the versions of unrelated packages and makes the cause of failures harder to isolate.
  • make upgrade itself takes about 5 minutes to run, tempting people to hand-edit the lockfiles.
  • Changing dependencies sometimes results in mysterious changes to unrelated dependencies (sometimes the result of a previous PR where someone hand-edited the lockfiles, sometimes due to more esoteric things.)

New GitHub Workflows

Upgrading one dependency

The ability to run make upgrade on GitHub against any branch has existed for several years in the form of upgrade-python-requirements.yml, and now we’ve added a companion workflow for upgrading just a single package: upgrade-one-python-dependency.yml.

This workflow can be used to safely upgrade the version of any package, and will create a PR for it against any branch you choose. For example, you can use it to create a PR against master if all you are doing is upgrading a dependency, or you can create a PR against your own branch if you’re accompanying the upgrade with code changes. Thanks to some work by Kira Miller, it can even update constraints.txt to a specific pinned version, if requested.

This new workflow is backed by make upgrade-package package=somepkg. You could run that directly, but it still takes a while, and running it outside of a devstack container on a Mac will introduce unwanted changes. So we recommend using the GitHub workflow instead.

Recompiling dependencies with minimal changes

Additionally, there is now a compile-python-requirements.yml so you can push up a change to .in and constraints.txt files and then have GitHub do the work of updating the .txt files. This helps when adding or removing dependencies or changing constraints. Just run the workflow against your branch, and it will make a PR against your branch with the updates.

This is backed by make compile-requirements, which was introduced by Régis Behmo a couple years ago. It recompiles the .in specification files into .txt lockfiles just like make upgrade does, except that it tries to not change the versions already in the lockfiles, where possible. This produces small diffs and minimal changes to production.

Consistency check for PRs

With these new tools in place, we’re also finally able to add some new guardrails: If make compile-requirements produces changes, the PR is given a failing check. This will detect several types of issues, including hand-editing of the .txt lockfiles. But it will also show you what changes need to be made to bring the dependencies back into a consistent state, and will link to helpful documentation for avoiding these problems in the future.

The check is not yet marked as required, but we hope to change that soon.

Let me know if there’s anything we can do to improve these tools or the new documentation!

  • Tim McCormack (for Arch-BOM at 2U)
4 Likes