Tutor Enhancement Proposal (TEP) for a quicker development workflow

Development
, , ,
1

Hello! :wave: I’m Kyle from tCRIL, and I’ve been working to make it easier to use Tutor as an Open edX development environment.

This post is dual-purpose:

:one: First, I want to share my new Tutor plugin: “quickdev”. It builds upon Tutor development improvements we made for Nutmeg (notably, the --mount functionality), aiming to further improve performance & reduce complexity. If you’re a Tutor power user, feel free to check it out and let me know what you think! That being said, don’t count on the plugin existing in the long-term, which brings me to the second purpose of this post…

:two: I’d like to propose to incorporate some or all of quickdev into Tutor proper. To be more specific, I’d like to make these changes to tutor dev:

  1. Define named volumes for the LMS/CMS venv, egg-info, node_modules, and generated static assets. Declare them in the Dockerfile so that they’re auto-populated when LMS/CMS are first started.

  2. Do the same in the official Tutor plugins that add services: discovery, notes, mfe, etc.

  3. In order to prevent the contents of these named volumes from becoming stale over time, provide a mechanism for deleting them so that they can be freshly populated from the image. We could have an automatic way, a manual way, or both:

    i. Automatic: Delete the named volumes regularly by triggering on a common event An aggressive version of this would be deleting the named volumes whenever the platform is stopped. A less-aggressive idea would be to clear them whenever launch is run, as well as whenever the related Docker image is pulled or built. We would need to add a few new hooks to implement this:

    • IMAGE_BUILT and IMAGE_PULLED actions, providing a way to to trigger the volume deletion; and
    • a COMPOSE_DEV_IMAGE_NAMED_VOLUMES filter in order to map images to their named volumes.

    ii. Manual: Create commands to clear out the named volmes on-demand. In quickdev, those commands are tutor quickdev [pip|npm|static]-restore, but they are not extensible. In Tutor, we could have tutor dev restore <image> <volume>. For example: tutor dev restore openedx venv would delete the named volume holding the edx-platform virtual environment. This command could also use the new COMPOSE_DEV_IMAGE_NAMED_VOLUMES filter in order to map "<image> <volume>" to actual Docker volumes.

  4. Automatically mount suspected edx-platform packages to a standard location. In quickdev, the location is /openedx/mounted-packages, and the recognized prefixes are:

    • xblock-*, a fairly established pattern for XBlock repos,
    • platform-plugin-*, which 2U/edX started using recently and I like, and
    • platform-lib-*, which is not a pattern yet, but I wouldn’t mind encouraging it.

    The same thing could be done in plugins if we could figure out any reasonable naming conventions for their packages.

  5. Provide a command for installing all packages mounted in the standard location mentioned above In quickdev, that’s tutor quickdev pip-install-mounted -m ... -m .... In Tutor, this could be implemented using the upcoming CLI_DO_COMMANDS filter, giving us tutor dev do pip-install-mounted -m ... -m ... -m ...

Proposed benefits of incorporating quickdev:

  • Developers would see better performance when working with bind-mounted repos.
  • The extra setup steps required after bind-mounting a repository would go away.
  • The need for image rebuilds or virtual environment bind-mounting when working on platform packages would go away.

Possible drawbacks:

  • More state would be persisted between platform restarts, which increases the number of things that could be “wrong”, invisible to the developer.
  • More implicitness: tutor dev run would now share a virtual environment with other containers, even though the command doesn’t make it clear.
  • Increased difference between tutor local and tutor dev, since local mode wouldn’t use these named volumes.

Unknowns:

  • I think this system could ease frontend development (using tutor-mfe) as well, but I’m not certain yet. I’ll need to do more research here, but I wanted to get this TEP out there first.

@tutor-maintainers @regis , let me know what you think!

10 Likes
2

@kmccormick I really like the fact this post came right when I was about to work on the basic features (named volumes, default mount location, and a volume population mechanism).
I definitely would love to see any dev life saver contribution become part of Tutor.

I’ll be sure to test the plugin and come back with well-constructed feedback.

2 Likes
3

Thanks for this detailed TEP @kmccormick! I’ll outline my thoughts below.

It’s really that you found a new method to make Open edX development more convenient and faster. Persisting the python virtual environment and other folders makes it much easier to install custom requirements and hack on external dependencies.

That being said, I’m worried by a couple issues raised by your TEP.

dev/local/k8s disparity

In this TEP named volumes are introduced in dev mode only. I understand this rationale: people should be careful with what they install in their production platforms. Having stateful volumes makes it harder to troubleshoot things in production. Also, this concept does not apply to k8s deployments.

But I expect that users will customize their environments in dev, and they will want to apply the same changes to production. In many companies, it’s the same people who hack on Open edX (dev) and have to handle deployments (devops). They will have trouble understanding why they “can’t just run tutor local run lms pip install their-custom-xblock” and see their xblocks appear in the studio.

We can certainly make it very explicit in the docs that this is a dev-only feature. Still, I expect that it will create some frustration.

Stateful containers

The lack of state in containers is one of the things that make Tutor more intuitive and less risky. Consider the following points:

  • To uninstall an Open edX platform, it is sufficient to delete the ~/.local/share/tutor folder.
  • Same for backups: migrating from one server to the next requires just one rsync command.

(These items are less important in dev mode than in local. This means that we can’t really resolve the disparity issue mentioned in the previous section.)

The fact that the containers are stateful will mean that we need an easy way to clear their states (i.e: the named volumes). Which brings us to the following item:

Volume freshness

Of course it would be great if we could automatically detect when volumes should be cleared. But it’s difficult to clear volumes as infrequently as possible and in a way that is not-too-surprising for the end-user.

It would be great if we could clear volumes on image building. Unfortunately, the IMAGE_BUILT action hook would not be relevant, because we are unable to detect when an image build is triggered in some circumstances. For instance, when we run docker compose run --build ....

Thus, I think that we would mostly have to rely on manual volume deletion. This means that users will have to remember to run this command once in a while.

Opinion: are we applying a band aid on a wooden leg?

I don’t know how properly resolve the issues listed above. Which leads me to question whether we are doing things the “right” way.

The following is a personal opinion, and an open question: isn’t the fact that it’s so difficult to troubleshoot Open edX by bind-mounting directories from the host an indication that there is a deeper upstream issue? (i.e: in edx-platform) Aren’t we attempting to get around design issues in edx-platform that should not exist in the first place?

I don’t mean to deflect responsibility to you Kyle. Of course I know that you’ve already worked a lot on making edx-platform more compatible with Tutor, for instance by moving dependencies out of the common/lib trunk. I want to ask if more can be done in edx-platform to bring the repo more in line with modern development practices. For instance:

  • Can we move node_modules to a different folder, such that it’s not overwritten when we bind-mount edx-platform? That way we would not have to re-run npm install.
  • Is there anything else we can do to avoid collecting static assets in dev?
2 Likes
4

Thanks for the review @regis . I too want to make sure we are solving the right problems rather than applying workarounds to things that are broken on a deeper level. Let me break down the problems I see with tutor dev as a it stands.

node_modules

I’m no npm expert, but from my research it seems that npm really wants node_modules to be stored in the root of the repository itself. The ways around this I can think of are:

  1. Install all packages in “global” mode (npm -g) so that they are installed somewhere in the container outside of the repository. In a prototype I was able to hackily achieve this by shadowing the npm binary with a script that forwarded all arguments to npm -g. This would also require further unraveling of the paver-based assets pipeline, which is a good thing anyway, and is something I think we could get 2U’s help on.
  2. In the dev Docker image, move /openedx/edx-platform/node_modules to /openedx/node_modules and replace the former with a link to the latter. Then, do the same thing for mounted repositories using an ENTRYPOINT script. I had this idea working in an earlier version of quickdev, but dropped it because it seemed too hacky.
  3. For micro-frontends, stop trying to use Docker containers in development mode. For backend services, just accept that we will have this problem until all legacy frontends are removed and our backends are purely JSON API servers.

python virtualenv

Thankfully, since the venv is stored at /openedx/venv, which is outside of edx-platform, running pip install -r requirements/dev.txt isn’t required after mounting a repository. However, people often want to modify Python requirements and/or install local versions of packages without too much of a hassle.

Outside of turning the venv into a named volume, I think the best improvement we could make would be to speed up the dev image build as much as possible. That way, it is much faster for developers to update edx-platform’s requirements lists and re-build the image. A few ideas I have:

  • Convert GitHub dependencies to PyPI requirements wherever possible. This is already in progress, and I could push to expedite if we decide against quickdev.
  • Hunt down and remove extraneous edx-platform requirements. Easier said than done, but it’s something the project ought to do at some point anyway.
  • Convince Docker to use the downloaded & cached version of the openedx image when re-building openedx-dev. That is: for me, at least, it seems like Docker wants to re-build the image from scratch when I run tutor dev dc build lms, even if I just ran tutor images pull openedx… there’s got do be something we can do to re-use those downloaded stages from the production image.

static assets

I’m not very knowledgable here but I’m sure the situation could be improved upstream. There’s discussion about the asset pipeline on this github issue. Even without overhauling the whole edx-platform asset pipeline, I have to hope that’s there’s something we could do to move the generated assets out of edx-platform…

jobs

This one’s already in progress but it’s worth mentioning anyway: having the pluggable jobs framework merged should help us condense any remaining “mounting repository preparation” commands down into single command, e.g. tutor dev do prepare-mounted-platform --mount=edx-platform.

egg-info

This one’s fairly minor, but as far as I can tell, edx-platform will always need Open_edX.egg-info file to be present in the repository. If we don’t use a named volume, this means that pip install -e . is non-optional. I think the jobs framework, mentioned above, could handle this nicely.

… in conclusion, I think there several things we could do to address the “wooden leg” itself, and I’m willing to embark on that, especially since most of them would benefit non-Tutor users too. I just think it will take a lot longer :sweat_smile: Let me know what you think.

1 Like
5

(Drive-by contribution to the thread)

To clarify, that investigation is focused primarily on a faster build tool for MFEs, and preliminary results aren’t amazing (at least while testing a drop-in solution that keeps webpack).

1 Like
6

Today I tried quickdev and it made it a million times easier and less confusing to try out a modified version of edx-platform master. Thanks so much!

1 Like
7

@regis @kmccormick I’ve read the replies and I would like to express my opinion about this:

TL;DR:

quickdev is a life changer that needs to look less @kmccormick’ly opinionated and more community-like, through a thorough discussion about features and implementation.

The longer version

Stateful containers?

Developers should be aware of mounted volumes by providing the proper feedback messages (info/warning) in the best places (saving configs, starting/restarting containers, etc.)

A band-aid on a wooden leg is indeed meaningless

I agree that issues ought to be solved upstream instead of providing fixes that would contribute to further delaying radical, in-depth solutions.

You can’t cast your wooden leg, but even a healthy person would love the comfort of a motorized wheelchair

I don’t think the tutor dev community should be deprived of quality-of-life improvements for this reason. The Tutor project has an advantage over Open edX in the way it has less strict barriers to embracing positive change. And whenever our wooden leg gets upgraded to become a modern prosthetic, we’ll just make sure to do the proper feature deprecations.

dev/local/k8s disparity ?

if we are relentless in integrating quickdev features in the Tutor core, why not instead work towards adopting quickdev into the official tutor plugin family with a proper adoptive name (e.g. tutor-devtools)?

We can then elaborate on which features to include, and how they would fit in the full picture (i.e. interoperability with the existing plugins). Then, in the end, add to the Tutor core any actions/hooks/defaults required by the plugin to serve its purpose (which is to streamline the Open edX development processes).

3 Likes
8

What you propose @ARMBouhali all makes a lot of sense. I agree with everything you said.

1 Like
9

In that case, maybe we should move the discussion to the next steps.
You can count me in as developer/tester/early adopter if we agree to go down that path.

Can we have a more fine-grained approach in dealing with @kmccormick 's proposition ? Maybe we can model it as atomic features and for each feature we can discuss:

  • implementation preferences: where it should be ?(core/a specific plugin/nowhere). How it should work.
  • Required tutor core changes for the feature to work.
  • Interfacing with existing plugins: how they can be changed to benefit from the feature.
  • Implementation concerns and how to deal with them: identify the impact on the user experience and express the precautions taken to address that (automated processes, additional feedback, etc)

When we get to that level of detail and have smaller decisions, the dev work can be split

If you agree to go with that. We may do it with a series of GitHub issues (which repo?) or a shared Google doc, or any means of collaboration.

Someone has to point out the direction. @regisb

1 Like
10

@ARMBouhali , glad to hear we have you as an early adopter. I agree with your perspective and I would also like to choose next steps. I think we can continue discussion here until we get to a point where we need to break it down into smaller tasks.

To speak directly to one open question:

implementation preferences: where it should be ?(core/a specific plugin/nowhere).

I still prefer core, primarily because I believe it’d make Tutor’s documentation for developers more cohesive. That is: when features are added via plugins, I assume that we cannot include them in the official Tutor docs.

So, if quickdev remained a plugin, the Tutor docs would need to describe one developer workflow (the current one) and then we’d have a separate set of documentation just for quickdev. When helping out new Tutor users, we’d have to navigate questions about both the built-in dev workflow as well as the quickdev workflow. For end users who don’t know or care about all the context behind quickdev, this would just be extra complexity.

That said, if the consensus is to keep the changes in a plugin, that is OK with me, and I will happily help make any changes necessary in order to get to a proper “1.0” release of the quickdev/devtools/<insert_name_here>.

Finally, if the consensus is “forget quickdev and just make upstream fixes instead”, I am also OK with that, but I would like to get going on those changes ASAP… they will take time, and I am very eager to get to a point where Tutor is great for Open edX development.

Now, all my cards are on the table :slightly_smiling_face: I would like to step back and hear from others about how they’d like to move forward.

11

Also a bit of a drive-by, but from my perspective Automatically mount suspected edx-platform packages to a standard location and Provide a command for installing all packages mounted in the standard location mentioned above are two particular things I have been missing in my personal development. I make a lot of use of the devstack convention of putting things in src. I think if necessary this could be separated out, although I also have to plug quickdev as a whole as something that made development significantly faster on my machine.

1 Like
12

Let me summarise my position in a nutshell: I know for a fact that quickdev will be hugely beneficial to developers. But I’m afraid that it will cause all sorts of trouble down the road (see above). I do not know yet whether this fear is justified or not. Thus, I suggest that we go with a plugin for now. And if these fears turn out not to be well-founded, then we’ll pull quickdev into the core.

I’m also curious what other people have to say.

1 Like
13

I’ve compiled the three paths I see forward (core, plugin, and upstream improvements) into an issue with a bunch of nested sub-tasks:

Please check it out if you’re interested in moving this forward. At the moment I’m poking at the “upstream improvements” path.

1 Like