Tutor Enhancement Proposal (TEP) for a quicker development workflow

Development
, , ,
4

Thanks for the review @regis . I too want to make sure we are solving the right problems rather than applying workarounds to things that are broken on a deeper level. Let me break down the problems I see with tutor dev as a it stands.

node_modules

I’m no npm expert, but from my research it seems that npm really wants node_modules to be stored in the root of the repository itself. The ways around this I can think of are:

  1. Install all packages in “global” mode (npm -g) so that they are installed somewhere in the container outside of the repository. In a prototype I was able to hackily achieve this by shadowing the npm binary with a script that forwarded all arguments to npm -g. This would also require further unraveling of the paver-based assets pipeline, which is a good thing anyway, and is something I think we could get 2U’s help on.
  2. In the dev Docker image, move /openedx/edx-platform/node_modules to /openedx/node_modules and replace the former with a link to the latter. Then, do the same thing for mounted repositories using an ENTRYPOINT script. I had this idea working in an earlier version of quickdev, but dropped it because it seemed too hacky.
  3. For micro-frontends, stop trying to use Docker containers in development mode. For backend services, just accept that we will have this problem until all legacy frontends are removed and our backends are purely JSON API servers.

python virtualenv

Thankfully, since the venv is stored at /openedx/venv, which is outside of edx-platform, running pip install -r requirements/dev.txt isn’t required after mounting a repository. However, people often want to modify Python requirements and/or install local versions of packages without too much of a hassle.

Outside of turning the venv into a named volume, I think the best improvement we could make would be to speed up the dev image build as much as possible. That way, it is much faster for developers to update edx-platform’s requirements lists and re-build the image. A few ideas I have:

  • Convert GitHub dependencies to PyPI requirements wherever possible. This is already in progress, and I could push to expedite if we decide against quickdev.
  • Hunt down and remove extraneous edx-platform requirements. Easier said than done, but it’s something the project ought to do at some point anyway.
  • Convince Docker to use the downloaded & cached version of the openedx image when re-building openedx-dev. That is: for me, at least, it seems like Docker wants to re-build the image from scratch when I run tutor dev dc build lms, even if I just ran tutor images pull openedx… there’s got do be something we can do to re-use those downloaded stages from the production image.

static assets

I’m not very knowledgable here but I’m sure the situation could be improved upstream. There’s discussion about the asset pipeline on this github issue. Even without overhauling the whole edx-platform asset pipeline, I have to hope that’s there’s something we could do to move the generated assets out of edx-platform…

jobs

This one’s already in progress but it’s worth mentioning anyway: having the pluggable jobs framework merged should help us condense any remaining “mounting repository preparation” commands down into single command, e.g. tutor dev do prepare-mounted-platform --mount=edx-platform.

egg-info

This one’s fairly minor, but as far as I can tell, edx-platform will always need Open_edX.egg-info file to be present in the repository. If we don’t use a named volume, this means that pip install -e . is non-optional. I think the jobs framework, mentioned above, could handle this nicely.

… in conclusion, I think there several things we could do to address the “wooden leg” itself, and I’m willing to embark on that, especially since most of them would benefit non-Tutor users too. I just think it will take a lot longer :sweat_smile: Let me know what you think.

1 Like