Breaking apart edx-platform's `./common/lib` folder

Development Architecture
,
1

Shorter

Some edx-platform code has moved around! Rebase your branches and reboot your development stacks. More details below.

Longer

Hey folks! A while back I posted about how Koa changed some edx-platform import paths in order to simplify the platform and remove some nasty tooling hacks. I’m posting again to share & celebrate some more simplifications that will be finished ahead of the Dec 2022 Olive release. All of these changes have already landed in master, except the capa and xblock_discussion changes which I expect will land soon.

(For those interested in the context and reasoning behind the changes, check out BOM-2579.)

Implications

General notes

  • The edx-platform folder ./common/lib will be deleted. Its contents have been moved to various other locations. Some import paths have been changed in order to make this happen. You can see the Detailed Changes in the next section below.

  • There is a new top-level code directory in edx-platform: ./xmodule.

  • Once the last couple changes land… going forward, if you see import <BLAH> in edx-platform, you can assume that either (a) it’s importing from an external library, or (b) <BLAH> represents the full path to the module, relative to the root of edx-platform. This is a pretty basic assumption that most Python projects have, so I’m happy that it’s finally true for the platform :relieved:

Special notes for edx-platform developers who follow master

If you have an open pull request against master, rebase your branch in order to avoid any potential logical merge conflicts and/or false-positives from the test suite.

For Devstack & Tutor Nightly users: When you pull these changes from master, you may run into strange errors about missing modules or unsatisfied requirements. If you see these, I recommend:

  • Bringing down containers
  • Pulling new images
  • Git-pulling the latest edx-platform code
  • Running make clean to delete all __pycache__ directories, which might be caching old compiled code
  • Running git clean -fd common/lib/ to ensure any old code is properly deleted (NOTE: This will delete all untracked files in common/lib. If you have unsaved work there, back it up first!)
  • Starting containers back up
  • Rebuilding static assets
  • For Tutor users who mount a local virtualenv: using make requirements in an LMS shell to uninstall the local projects and re-install the correct requirements.

Special notes for edx-platform operators

  • If you have a fork of edx-platform that changes code in common/lib, then when you eventually rebase onto Olive, ensure that your changes are reflected in the new code locations. Make sure you don’t accidentally recreate the common/lib directory; any code placed there would be ignored.
  • If you maintain code that imports modules from edx-platform/common/lib, you may need to update some of those import paths.
  • The Python dependencies lists in edx-platform/requirements no longer point to any code in the local repository. So, if you maintain a Dockerfile or other sort of edx-platform build script, you now have the option of copying in and installing requirement files before copying in the code directories.

Detailed Changes

symmath

safe_lxml

sandbox-packages (loncapa, verifiers, eia)

xmodule

capa

xblock_discussion

Thanks!

Thank you, all those who have helped with this big refactoring:

Bit by bit, we’re simplifying edx-platform & making it more accessible for operators and developers who want to help shape the future of education :tada: :pray:

14 Likes