Post OEP-58, site operators will be expected to pull updated translations as part of the deployment process.
Deprecation
The pre OEP-58 translations strategy (having translation files checked into repos) has been deprecated.
This will be finalized by removing the (outdated) translation files from their respective repos. The progress of this is being tracked here.
Preparing for removal
To pull updated translations as part of your deployment process, you will need to set the OPENEDX_ATLAS_PULL enviornment variable and run make pull_translations.
This should be a one-line addition to your deployment process:
make OPENEDX_ATLAS_PULL=true pull_translations
You can find more detailed documentation on the process for pulling updated translations here.
After removal
After the pull request removing the checked-in translations from a repository is merged, you will no longer need to set the OPENEDX_ATLAS_PULL environment variable.
Timeline
We plan to begin merging PRs removing checked-in translations on Monday, March 4th, 2024.
You will need to update your deployment process to ensure your deployment still has translations after the PRs are merged.
Concerns?
If you encounter any issues integrating make pull_translations into your deployment process, please let me @brian.smith know as soon as possible.
If I do not hear anything, I will assume the integration for that repo has gone smoothly.
Thank you!
I’m extremely excited to see this project reach completion! Thank you so much for helping us move on to the final steps!
We’re having a surprisingly difficult time getting this working in our build/deployment pipeline – I don’t think we’ll be able to get it done in all repos by March 4th.
So far this is what we have for edx-platform, where we need to download Atlas, get a Python environment set up, and also get a Node environment set up:
mkdir -p bin
curl -sS -L https://github.com/openedx/openedx-atlas/releases/latest/download/atlas -o ./bin/atlas
chmod +x ./bin/atlas
source {{ edxapp_venv_dir }}/bin/activate
# Use production-like environment and minimal config to avoid needing dev dependencies or full config.
PATH="./bin/:$PATH" DJANGO_SETTINGS_MODULE=lms.envs.production \
LMS_CFG=lms/envs/minimal.yml STUDIO_CFG=lms/envs/minimal.yml \
OPENEDX_ATLAS_PULL=true make pull_translations
rm ./bin/atlas
…but we’re getting NPM dependency conflicts at the moment (in the compilejs step, after the atlas-specific part.)
I don’t think we’ll be able to get it done in all repos by March 4th.
That’s OK. This isn’t something that needs to happen all at once. If we need to hold off on a couple repos we can still get most of the PRs removing the need for the OPENEDX_ATLAS_PULL merged.
we’re getting NPM dependency conflicts at the moment
Could you elaborate on these? Also, @omar is this something you’ve encountered when working with atlas on edx-platform before?
Not really. We’ve tried to keep atlas dependency free – except for Bash itself.
There’s more than a way to install atlas but for the edX Platform we’re installing it via pip install openedx-atlas not manually but in the base.in file of course.
In npm packages, we’re adding it as an npm dependency npm i @edx/openedx-atlas in package.json files.
But overall, all what these packages do is to help make the atlas binar/script available in the bin directory, so we didn’t have to install it via curl.
This certainly work, but I’d say it’s usually not needed and I recommend using the pip or npm package. It has nicer versioning as well.
@Tim_McCormack We’d be able to help more if we have a pull request to look at to understand the full context. I’m assuming you’re updating the configuration repo, which is something we’re familiar with.
Meanwhile, I recommend you taking a look at the Tutor builds which can be a good example to look at:
Ah, thank you for the tip! From the installation instructions I had gotten the impression that I had to fetch it either as a shell script or as an NPM dependency – I didn’t realize there was a Python package. That definitely simplifies some things.
And thanks for the pointers to the other integrations. I’ll take a look at those.
I’m creating a pull request to avoid using paver which would fix the issue for production deployments.
I had gotten the impression that I had to fetch it either as a shell script or as an NPM dependency – I didn’t realize there was a Python package. That definitely simplifies some things.
You’re right @Tim_McCormack The installation instructions is outdated, I’m going to add an pip installation section.
This will cause git status to get into a dirty state. Usually Ansible breaks. This isn’t a bug, it’s part of how atlas pull works.
We can and should fix it in two ways:
Perform a git reset before pulling during Ansible deployment (I’ll need your help with that, as we’re not using Ansible and lack the testing infrastructure for it)
Given the slow development cycle for these pipelines (I currently have to wait 25 minutes to find out the next error!) it would be great if we could get a 2 week extension. I really hope it won’t take that long, and probably things will go a lot faster after edx-platform is dealt with, but… I don’t feel like I can promise anything sooner than that. (Diana is also working on the MFEs in parallel.)
I’ve started an inventory at 2U Atlas translations cutover inventory [PUBLIC] - Google Sheets but I’m unsure how the translations for some of these repos work. The IDAs and MFEs I understand, to a first approximation. But what about the xblocks, apps, and libraries? Can I assume that the pull_translations for each IDA and MFE is set up to somehow gather all of the translations for the dependencies as well? And if so, does that mean that pull_translations always has to be run in the production environment, complete with production configs and private dependencies?
Yes. This allows each IDA and MFE to pass options into atlas (the most common use case being which languages to deploy) and have those options applied when pulling the dependencies as well.
OK, I think the key thing here is that eventually we get to for _xblock_tag, xblock_class in XBlock.load_classes(): – the set of translations to pull is dynamically determined by what dependencies have been installed and activated. So pull_translations definitely needs to be called from inside the production-like environment (with any private dependencies installed, etc.)
2U is all set with the ecommerce repo (since we’re using a different branch) and all of the XBlocks. edx-platform should be all set, but I’d like to let it sit in production for a day or two before calling that one done. Still working on other things.
EDIT: OK, I think we’re good to go on those – I’ll test in stage after the removals take effect.
(Diana is also working on the MFEs in parallel.)
