Pulling translations for non-tutor deployments


Before OEP-58, translations lived in their respective repositories.

Those checked-in translations are no longer being updated.

Updated translations now live in the openedx-translations repository.

Post OEP-58, site operators will be expected to pull updated translations as part of the deployment process.


The pre OEP-58 translations strategy (having translation files checked into repos) has been deprecated.

This will be finalized by removing the (outdated) translation files from their respective repos. The progress of this is being tracked here.

Preparing for removal

To pull updated translations as part of your deployment process, you will need to set the OPENEDX_ATLAS_PULL enviornment variable and run make pull_translations.

This should be a one-line addition to your deployment process:

make OPENEDX_ATLAS_PULL=true pull_translations

You can find more detailed documentation on the process for pulling updated translations here.

After removal

After the pull request removing the checked-in translations from a repository is merged, you will no longer need to set the OPENEDX_ATLAS_PULL environment variable.


We plan to begin merging PRs removing checked-in translations on Monday, March 4th, 2024.

You will need to update your deployment process to ensure your deployment still has translations after the PRs are merged.


If you encounter any issues integrating make pull_translations into your deployment process, please let me @brian.smith know as soon as possible.

If I do not hear anything, I will assume the integration for that repo has gone smoothly.

Thank you!

I’m extremely excited to see this project reach completion! Thank you so much for helping us move on to the final steps!


Currently that Make target requires the edx-i18n-tools Python package, which is only pulled in at the testing.in requirements layer. That means it’s not usually available in our deployment pipeline, which only installs the base.in requirements (and a few other things). I think it would make sense to move this dependency up to the base layer so that it’s always available: build: Install i18n-tools in base dependencies for pull_translations by timmc-edx · Pull Request #34285 · openedx/edx-platform · GitHub


We’re having a surprisingly difficult time getting this working in our build/deployment pipeline – I don’t think we’ll be able to get it done in all repos by March 4th.

So far this is what we have for edx-platform, where we need to download Atlas, get a Python environment set up, and also get a Node environment set up:

mkdir -p bin
curl -sS -L https://github.com/openedx/openedx-atlas/releases/latest/download/atlas -o ./bin/atlas
chmod +x ./bin/atlas
source {{ edxapp_venv_dir }}/bin/activate
# Use production-like environment and minimal config to avoid needing dev dependencies or full config.
PATH="./bin/:$PATH" DJANGO_SETTINGS_MODULE=lms.envs.production \
  LMS_CFG=lms/envs/minimal.yml STUDIO_CFG=lms/envs/minimal.yml \
  OPENEDX_ATLAS_PULL=true make pull_translations
rm ./bin/atlas

…but we’re getting NPM dependency conflicts at the moment (in the compilejs step, after the atlas-specific part.)

I don’t think we’ll be able to get it done in all repos by March 4th.

That’s OK. This isn’t something that needs to happen all at once. If we need to hold off on a couple repos we can still get most of the PRs removing the need for the OPENEDX_ATLAS_PULL merged.

we’re getting NPM dependency conflicts at the moment

Could you elaborate on these? Also, @omar is this something you’ve encountered when working with atlas on edx-platform before?

Not really. We’ve tried to keep atlas dependency free – except for Bash itself.

There’s more than a way to install atlas but for the edX Platform we’re installing it via pip install openedx-atlas not manually but in the base.in file of course.

In npm packages, we’re adding it as an npm dependency npm i @edx/openedx-atlas in package.json files.

But overall, all what these packages do is to help make the atlas binar/script available in the bin directory, so we didn’t have to install it via curl.

This certainly work, but I’d say it’s usually not needed and I recommend using the pip or npm package. It has nicer versioning as well.

@Tim_McCormack We’d be able to help more if we have a pull request to look at to understand the full context. I’m assuming you’re updating the configuration repo, which is something we’re familiar with.

Meanwhile, I recommend you taking a look at the Tutor builds which can be a good example to look at:

Because we’ve gone through a ton of repositories, here’s all the work that has been done so far to make atlas supported:

@omar you mentioned

However, if I’m understanding correctly, the issue isn’t with atlas dependencies, but instead with dependencies in the compilejs step

So it seems there’s something about the environment in which the deployment is running that is causing

to fail.

Ah, thank you for the tip! From the installation instructions I had gotten the impression that I had to fetch it either as a shell script or as an NPM dependency – I didn’t realize there was a Python package. That definitely simplifies some things.

And thanks for the pointers to the other integrations. I’ll take a look at those.

Thanks for the tip @brian.smith, I wasn’t aware of the nature of the failure.

This is a legacy development/ci step we kept. Now I’m realizing it’s not suitable with production due to the fact it tries to install requirements:

I’m creating a pull request to avoid using paver which would fix the issue for production deployments.

I had gotten the impression that I had to fetch it either as a shell script or as an NPM dependency – I didn’t realize there was a Python package. That definitely simplifies some things.

You’re right @Tim_McCormack The installation instructions is outdated, I’m going to add an pip installation section.

@Tim_McCormack @brian.smith here are the PRs that should address both issues:

1 Like

@Tim_McCormack Just in case it’s not clear.

This will cause git status to get into a dirty state. Usually Ansible breaks. This isn’t a bug, it’s part of how atlas pull works.

We can and should fix it in two ways:

Wonderful, thank you! So… this means that npm should no longer be required at all for this make recipe, right?

Given the slow development cycle for these pipelines (I currently have to wait 25 minutes to find out the next error!) it would be great if we could get a 2 week extension. I really hope it won’t take that long, and probably things will go a lot faster after edx-platform is dealt with, but… I don’t feel like I can promise anything sooner than that. :sweat_smile: (Diana is also working on the MFEs in parallel.)

That seems reasonable. Would you be able to compile a list of the repos that you need more time for?

I’ve started an inventory at 2U Atlas translations cutover inventory [PUBLIC] - Google Sheets but I’m unsure how the translations for some of these repos work. The IDAs and MFEs I understand, to a first approximation. But what about the xblocks, apps, and libraries? Can I assume that the pull_translations for each IDA and MFE is set up to somehow gather all of the translations for the dependencies as well? And if so, does that mean that pull_translations always has to be run in the production environment, complete with production configs and private dependencies?

Yes. This allows each IDA and MFE to pass options into atlas (the most common use case being which languages to deploy) and have those options applied when pulling the dependencies as well.

edx-platform/docs/decisions/0019-oep-58-atlas-translations-design.rst at master · openedx/edx-platform · GitHub should help clarify and provide some context around that decision.

To add a specific example, we can look at XBlocks in edx-platform

Hopefully that helps! If you have any other questions please let me and/or @omar know!

OK, I think the key thing here is that eventually we get to for _xblock_tag, xblock_class in XBlock.load_classes(): – the set of translations to pull is dynamically determined by what dependencies have been installed and activated. So pull_translations definitely needs to be called from inside the production-like environment (with any private dependencies installed, etc.) :+1:

1 Like

2U is all set with the ecommerce repo (since we’re using a different branch) and all of the XBlocks. edx-platform should be all set, but I’d like to let it sit in production for a day or two before calling that one done. Still working on other things.

EDIT: OK, I think we’re good to go on those – I’ll test in stage after the removals take effect.

1 Like

I’m happy to see it deployed in 2U :slight_smile:

That literally happened overnight, after a year for work :sweat_smile:

We’ll go ahead and triage the deprecation PRs!

1 Like

Yes, npm isn’t required for the edx-platform atlas pull because it’s already installed in the python virtual env.

I just saw this post and I’m unsure if I replied to it or not, and I hope we’re already past this and no longer need npm.

Yep, all sorted out at this point – I was able to move the pull_translations call above the Ansible tasks that dealt with nodejs.