FAQ: Running an edx-platform Fork with Tutor

Community Show & Tell
,
2

Mini Course Docs: Tutor Fork Quickstart

Quickstart

Latest update: May 19, 2022 9:52 PM (Please change the versions accordingly if this post is too old.)

Table of Contents

0. Prerequisites

1. Set up Environment

Collect the versions for the software you will be using.

For example, my current setup for Maple uses the following.

Tutor: 13.2.2

Python: 3.8.12

edx-platform: https://github.com/overhangio/tutor/blob/v13.2.2/tutor/templates/config/defaults.yml#L51

(Choose according to your release)

1.1. Create Virtual Python Environment

Create a folder to store your Tutor environment.

mkdir tutor-maple
cd tutor-maple

The following examples assume you are running the code from the ~/tutor-maple directory. I will also refer to this directory as $(pwd). You should replace $(pwd) with the result of running the following code in your working directory.

pwd

The following code can be run to create a Python environment with latest versions.

pyenv virtualenv 3.8.12 tutor-maple

You can then activate the environment.

pyenv local tutor-maple

1.2. Install Tutor

The easiest way to install Tutor for a development environment is by using the pip package. You can run the following code to install Tutor.

pip install "tutor[full]==v13.2.2"

1.3. Environment Variables

You should add the following variables, so that your Tutor environment will be placed in an easy to reach location. You can also change these variables to place more than one Tutor environment in your development machine.

export TUTOR_ROOT=$(pwd)/tutor
export TUTOR_PLUGINS_ROOT=$(pwd)/tutor-plugins

It is extremely important that

these values do not change during the set up, otherwise you will end up with two broken tutor environments. The environmental variables will change if you use another terminal during the set up or close your terminal. Simply run the export commands shown above again if your terminal changed for any reason. You should follow section 1.3.1 below if you do not want to worry about the environmental variables.

1.3.1. (Optional) Auto-export Environment Variables

You will need to run the code above every time you want to work on this specific instance of Tutor. You can automate this process by installing https://direnv.net/.

You simply need to create an .envrc file in your working directory with the export commands above and they will be automatically run.

1.4. Save Configuration

You should run the save configuration command now to generate the Tutor environment at your working directory. These prompt should be familiar as they are the same the tutor local quickstart, however, we do not want to start Tutor yet, so we will only create the configuration files.

tutor config save --interactive

You should answer first question as no since you will be doing development.

1.4.1. (Optional) Initialize Git Repository

You can initialize a git repository inside your working directory at this point to store your Tutor configuration. You should be careful about uploading this git repository as it will include all your database passwords and other secretes. This repository should be a private repo if you have to upload it.

git init

1.4.2. .gitignore

You might want to create a .gitignore file, so that you do not include the Tutor data files.

tutor/volumes
tutor/data*

1.5. Additional Tutor Configuration

You should add the following values to the config.yml located inside your Tutor environment created in 1.3. Environment Variables . (You can find it by running echo $(pwd)/tutor if you lost it.)

LOCAL_PROJECT_NAME: tutor_maple
DEV_PROJECT_NAME: tutor_maple_dev

Your Tutor installation will be designated the name tutor in your Docker environment by default, if you do not change it here, you will overwrite that Docker environment.

Do not forget to save your config.yml file by running tutor config save.

2. Build Development Image

You are now ready to create the openedx and openedx Development images.

2.1. openedx Image

You can now run the familiar quickstart command.

tutor local quickstart

Once you confirm that your site is up and running, stop it.

tutor local stop

You have now downloaded the openedx image and are ready to build the development image.

2.2. openedx Development Image

The development image is created by simply running the following code.

tutor dev dc build lms

After the image is build, you can start your development instance in the detached mode.

tutor dev start -d

Once you confirmed that your development site is up and running, stop it.

tutor dev stop

2.3. (Optional) Tag Your Images

You should now tag your images to upload them to your image repository if you will be using different machines for development and production.

DOCKER_IMAGE_OPENEDX: <IMAGE REPO>/<IMAGE NAME>:<IMAGE TAG>
DOCKER_IMAGE_OPENEDX_DEV: <IMAGE REPO>/<DEV IMAGE NAME>:<IMAGE TAG>

You should now tag your development image.

2.3.1. (Optional) Create Development Image Again to Tag it

(You can also find the image ID from Docker Desktop and use the Docker CLI to this step.)

tutor dev dc build lms

2.3.2 (Optional) Upload Your Image

You can run the following commands to upload your images.

tutor images push openedx
docker push $(tutor config printvalue DOCKER_IMAGE_OPENEDX_DEV)

3. Creating Development Environment

You are now ready to start working on your own edx-platform fork.

3.1. Create Your Fork

Use GitHub to create a fork of the edx-platform.

https://github.com/openedx/edx-platform

3.2. Download Your Forked Repo to Your Development Environment

You are now ready to use your forked repo for development.

First receive the files from the Tutor installation.

mkdir -p $(tutor config printroot)/volumes/edx-platform
tutor dev copyfrom lms /openedx/edx-platform $(tutor config printroot)/volumes/

This way you will be able to use the development tools that are installed inside the Development image. It will take some time to copy over the files.

Then change the remote to point to your fork.

cd $(pwd)/tutor/volumes/edx-platform
git remote set-url origin <YOUR_REPO_URL>

Sync your repo.

git fetch

Finally change to the branch you will be using.

git checkout -b <USERNAME>-dev

3.3. Point Tutor Towards Your Forked Repo

You should create an override file docker-compose.override.ymlat this location $(pwd)/tutor/env/dev/docker-compose.override.yml with the following content.

You should replace the variables with your working directory.

version: "3.7"
services:
  lms:
    volumes:
      - "<PWD>/tutor/volumes/edx-platform:/openedx/edx-platform"
  cms:
    volumes:
      - "<PWD>/tutor/volumes/edx-platform:/openedx/edx-platform"
  lms-worker:
    volumes:
      - "<PWD>/tutor/volumes/edx-platform:/openedx/edx-platform"
  cms-worker:
    volumes:
      - "<PWD>/tutor/volumes/edx-platform:/openedx/edx-platform"

3.4. Start Development

Your Tutor installation will now use your fork.

tutor dev start

Congratulations, you now have a development environment to start working on an edx-platform fork.

4. Build your Custom Image

You have done your changes and are now ready to build your custom image.

4.1. Apply the Patches

You need to make sure you have all the patches applied to your codebase because we will remove them to prevent problems.

You can find the current patches from the Dockerfile at the $(pwd)/tutor/env/build/openedx/Dockerfile directory.

The patches section will start with the following code.

# Identify tutor user to cherry-pick commits
RUN git config --global user.email "tutor@overhang.io" \
  && git config --global user.name "Tutor"
# Patch edx-platform

You can directly run the codes after the RUN command in your edx-platform fork root directory (which is $(pwd)/tutor/volumes/edx-platform if you use the one suggested in this guide).

For example, Tutor 13.2.2 uses the following patches. 
# Fix forum notification for questions
# https://github.com/openedx/edx-platform/pull/29611
RUN git fetch --depth=2 https://github.com/open-craft/edx-platform/ 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2
# SAML security fix
# https://github.com/overhangio/edx-platform/tree/overhangio/sec-fix-saml-vulnerability
RUN git fetch --depth=2 https://github.com/overhangio/edx-platform/ 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7
# Rate limiting security fix
# https://github.com/overhangio/edx-platform/tree/overhangio/sec-rate-limiting
RUN git fetch --depth=2 https://github.com/overhangio/edx-platform/ b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f
# Fix studio-frontend by pinning the installed version
# https://github.com/openedx/edx-platform/pull/30309
RUN git fetch --depth=2 https://github.com/uetuluk/edx-platform/ 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458

You simply need to run the following commands in your edx-platform root directory.

git fetch --depth=2 https://github.com/open-craft/edx-platform/ 03731f19459e558f188c06aac5cc9ca1bbc675c2 && git cherry-pick 03731f19459e558f188c06aac5cc9ca1bbc675c2
git fetch --depth=2 https://github.com/overhangio/edx-platform/ 3b985f207853e88090d68a81acd52866b71f5af7 && git cherry-pick 3b985f207853e88090d68a81acd52866b71f5af7
git fetch --depth=2 https://github.com/overhangio/edx-platform/ b5723e416e628cac4fa84392ca13e1b72817674f && git cherry-pick b5723e416e628cac4fa84392ca13e1b72817674f
git fetch --depth=2 https://github.com/uetuluk/edx-platform/ 53ea60eee86e094f35815ac1c4114d6811f4d458 && git cherry-pick 53ea60eee86e094f35815ac1c4114d6811f4d458

4.2. Remove the Patches

Now that you applied the patches, you do not need them anymore, use the following v1 plugin to remove them.

"""
NO Patch plugin to prevent applying the patch again.

name: nopatch
version: 0.1.0
patches:
  openedx-dockerfile-git-patches-default: "#"
"""

from tutor import hooks

hooks.Filters.ENV_PATCHES.add_item(
  (
  "openedx-dockerfile-git-patches-default",
  "#"
  )
)

4.2.1. Creating and enabling the Plugin

If you do not know what a Tutor plugin is, you can follow this quick guide.

You need to create a file called nopatch.py inside the plugins directory declared in 1.3. Environment Variables. ($(pwd)/tutor-plugins for this guide)

Then you need to enable the plugin by running

tutor plugins enable nopatch

And save your configuration.

tutor config save

If you are using git, you will notice that the Dockerfile has changed to remove the patches.

4.2.2. Consequences

This plugin will cause your images to never apply the patches inserted by Tutor, therefore you will have to manually check if there are any new patches every time you upgrade your Tutor version.

4.3. Update your Repo

You should now commit and push your changes to your fork repo since the Tutor build commands use the remote repo rather than your local one.

cd $(pwd)/tutor/volumes/edx-platform
git add .
git commit -m "Your Commit Message"

4.4. Build your Image

You can now build your image. Since you are using a custom edx-platform repository, you need to let Tutor know where to look.

Use the following command with added arguments to build your custom openedx image.

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="https://github.com/<USERNAME>/edx-platform" --build-arg EDX_PLATFORM_VERSION="<USERNAME>-dev"

4.4.1. Tagging the Image

Simply follow 2.3. (Optional) Tag Your Images to tag your images.

4.4.2. Private edx-platform Fork

If you are using a private edx-platform fork, you can create a Personal access token (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) ****and add it to the EDX_PLATFORM_REPOSITORY argument.

tutor images build openedx --build-arg EDX_PLATFORM_REPOSITORY="https://<PAT>@github.com/<USERNAME>/edx-platform" --build-arg EDX_PLATFORM_VERSION="<USERNAME>-dev"

4.5. Use your Image in Production

Simply follow 2.3. (Optional) Tag Your Images to change the image Tutor will use when you run tutor local quickstart.

3 Likes