Please update your git URLs for edx-platform and several other repos

Summary

If you develop or deploy Open edX, you may need to update some URLs to several repositories due to redirects that have been lost.

What happened

  • Several years ago, when 2U and Axim split, the Open edX git repositories were moved from the edx GitHub org to the openedx GitHub org. This left redirects in place; references to the repositories under the edx org continued to work.
  • Today (2/26), 2U moved to consolidate some GitHub orgs, moving archived repos from the edx-unsupported org into edx. What we did not anticipate is that among those were some old edx.org-specific forks of six Open edX repos (listed below). These had the same names as their origin repos.
  • The result was that instead of being redirected to the correct repos in the openedx org, requests for the repos in the edx org received these 3 year old forks. This broke deployments at 2U and caused git errors for developers.
  • The redirects cannot be restored readily. 2U is replacing these unexpected repos on the edx side with “stub” copies that have a README pointing to this post and are otherwise empty. The remaining work is to update references in code, configuration, and documentation. At a high priority this includes deployment and dependency configuration; at a low priority this also includes PR and issue links in documentation. (These are all changes that needed to be made eventually, but due to today’s name collisions, the need to change them has become urgent.)

Affected repos:

If you have any of the following checked out, you will need to update them according to the instructions below:

  • edx-platform
  • cs_comments_service
  • XBlock
  • xblock-sdk
  • devstack
  • django-require

Action items

Developers

  • Go to your local copy of each repo and run git remote -v. Find the remote that lists a URL pointing to either the openedx or edx version. (For most people this will be a remote called “origin” or “upstream”, and may be the only one.)
  • If the URL uses the openedx org, you’re all set. You don’t need to do anything.
  • If the URL instead uses the edx org, you’ll need to update that remote, e.g. git remote set-url origin ...

Maintainers and deployers

We’ll need to ensure that any remaining pointers to the edx URL for these repos are updated to the openedx URL, in both Open edX repositories and in private deployments.

Here’s a GitHub search query that will find git and http URLs that probably need to be updated:

org:openedx NOT is:archived /(?i)github\.com[^A-Za-z]+edx\/(edx-platform|cs_comments_service|XBlock|xblock-sdk|devstack|django-require)\b/

You can also add NOT language:reStructuredText NOT language:Markdown to exclude instances in documentation, since those are lower priority to update.

(The Arch-BOM team at 2U has updated all of the critical locations that we could find in public repos, but private repos, documentation, and non-git-controlled text have not been updated.)

2U-side remediation

For any repo that was moved from edx to openedx and that was replaced on the edx side by accident this week, the process is:

  1. Move the edx-org repo to a new name, preserving its PRs, issues, and other history.
  2. Create a new repository in its place, breaking the redirect that is unavoidably created in step 1.
  3. In the new repository, create a README instructing people to update the URL that brought them there and pointing to this page.

I think that’s the most we can do on our side to smooth the transition. My apologies to anyone who is experiencing disruption from this.

4 Likes

Thanks for the detailed write-up Tim! This issue will also affect all users running Tutor<13.1.3 (Maple release). These users stuck on older releases should add a build argument to their build commands:

tutor images build -a EDX_PLATFORM_REPOSITORY=https://github.com/openedx/edx-platform.git openedx
1 Like

Ooof, right, named releases – yes, both edx-platform and cs_comments_service would be affected.

Theoretically this could also affect any git-dependency on the other 4 repos, like if someone is pinning a dependency to a specific commit.

One option for a temporary fix on older releases could be to make a patch in openedx-dockerfile-git-patches-default with a git config change, something like

git config url."https://github.com/openedx/django-require.git".insteadOf "https://github.com/edx/django-require.git"

Avoids having to cherrypick a fix into edx-platform

2 Likes

This is as implemented by my teammate at WGU, Carlos Marquez:

name: django-require-edx-org-fix
version: 0.2.0
patches:
  openedx-dockerfile-minimal: |
    #----------------------------------------------------DJANGO-REQUIRE-EDX-ORG-FIX----------------------------------------------------#
    RUN git config --global url."https://github.com/openedx/django-require.git".insteadOf "https://github.com/edx/django-require.git"
    #--------------------------------------------------END DJANGO-REQUIRE-EDX-ORG-FIX--------------------------------------------------#
3 Likes

Thanks for letting me know about this @bryan.w!. I ran into this issue trying to build a new Docker openedx image with Tutor for the nutmeg release because of this dependency on a repo that’s now not there due to this article’s update mentioned above.

edx-platform/requirements/edx/github.in at open-release/nutmeg.master · openedx/edx-platform

# original repo is not maintained any more.
git+https://github.com/edx/django-require.git@0c54adb167142383b26ea6b3edecc3211822a776#egg=django-require==1.0.12

Error Received

I don’t see a tag for version 1.0.12 that nutmeg used to reference, so does this .insteadOf use the latest release tag?

cc @Tim_McCormack @regis

@bryan.w Figured it out. The openedx-dockerfile-minimal patch comes before git is actually installed. I moved this django-require command to patch openedx-dockerfile-git-patches-default like you mentioned above and the tutor image build openedx works great.

Location of openedx-dockerfile-minimal in relation to git install

@bryan.w I was afraid that even with that .insteadOf git command above that it would not know how to grab version 1.0.12 for openedx-django-require because it doesn’t exist.

Here is what I’m seeing in the virtual environment within the LMS container.

I think that I’m going to need to install latest version of openedx/django-require to ensure that it can be used by the edx-platform apps.

Moving this to the openedx-dockerfile-post-python-requirements patch as well because it comes after the ./requirements/edx/base.txt call.

hooks.Filters.ENV_PATCHES.add_items([
   (
       "openedx-dockerfile-git-patches-default",
       """
# Fixing this `django-require` package to be from `openedx` org rather than `edx` org.
RUN git config url."https://github.com/openedx/django-require.git".insteadOf "https://github.com/edx/django-require.git"
"""
   ),
   (
       "openedx-dockerfile-post-python-requirements",
       """
# Make sure to install latest version of `openedx/django-require` for the platform to use. Uninstall existing version 1.0.12 first, then reinstall v2.0.0.
RUN pip uninstall -y django-require
RUN pip install -e git+https://github.com/openedx/django-require.git@v2.0.0#egg=openedx-django-require==2.0.0
    ),
    ...
])

Seems like v2.0.0 of openedx/django-require will work with the nutmeg Django 3.2.16 version because of this django-require/setup.py at v2.0.0 · openedx/django-require (github.com).

And version 2.1.0 of openedx/django-require will work with version Django 3.2 and 4.2.
django-require/setup.py at 2.1.0 · openedx/django-require (github.com)

The only way that I was able to perform a new Docker image build for openedx on nutmeg was to perform this edx-platform update which is what I’m trying to avoid if possible with the tutor fix that @bryan.w mentions above.

fix: Applied django-require package updates needed. · CUCWD/edx-platform@c9a27ed (github.com)

The build error appears to indicate that the git fetch for the particular commit 0c54adb167142383b26ea6b3edecc3211822a776 is not found in the openedx/django-require repo. I performed this manually though and didn’t receive an error on the fetch, so I don’t understand why the tutor patches above wouldn’t work.

@bryan.w fix worked like a charm. I’m not sure why it didn’t work @Zachary_Trabookis , you don’t need to link the whole pip-like URL, just link the primary git URL as mentioned by Bryan. Maybe you forgot to run tutor config save, not sure

edit: @Zachary_Trabookis you’re applying the link in openedx-dockerfile-git-patches-default which will affect only code stage. Please apply it on openedx-dockerfile-minimal to have it on all stages