It appears that the Github workflow for unit-tests on the edx-platform repo use a Github action self-hosted configuration.
Does edX or the community have recommendations for this configuration on AWS that they can share?
Github’s Documentation on Self-Hosted Runner Configuration
The self-hosted Github workflow runner doesn’t have to be on AWS. We’re just looking for a way to run these unit-tests workflows that the edx-platform runs on a PR. Looking for hosting configuration examples if possible.
Thanks @sarina for including others here. I guess the questions is what additional configuration outside of install requirements for edx-platform needs to occur. What’s a typical system requirements for this CI server? Does edX use Docker containers to run these unit-tests workflow actions in to avoid security issues and conflicts with other developers PR actions running simultaneously?
I noticed that this unit-tests workflow requires the following
Any other prerequisites?
It would be helpful to have instructions for this additional self-hosted Github action configuration if possible.
@jmbowman would know more, but I believe the docker file being used for those self hosted runners is here: https://github.com/openedx/edx-platform/blob/master/scripts/ci-runner.Dockerfile
@feanil @jmbowman
It looks like this Github workflow builds and pushes the openedx/edx-platform-ci-runner image to Dockerhub.
Does edX have instructions for running that Docker image above on a Kubernetes setup or AWS, etc?
The main edx-platform repo uses self-hosted runners because the volume of commits and test shards is large enough to overwhelm the GitHub plan’s allotment of concurrent workers. We’re also using faster machines than the default hosted runners, giving us shorter test run durations.
The original thinking was that we should be able to configure this for forks to just fall back to using the GitHub hosted runners at no cost, just needing to wait longer for tests to complete. What we haven’t figured out yet is how to allow this without needing to customize the unit-tests.yml workflow definition in the fork; if we add hosted runners as a fallback, the main repo will sometimes start using slower hosted workers before new self-hosted workers have time to finish spinning up.
For now, I’d recommend customizing your fork’s worfklow to run in “ubuntu-20.04” instead of “edx-platform-runner”. You could set up a Kubernetes cluster for self-hosted runners like we have, but it’s complex enough that I don’t think it’s worth the effort for most forks. And I welcome suggestions on how to make the default configuration work better for forks; the best idea I have so far is two copies of the workflow with different if conditions, one to run only on the main repo and the other to run only in forks (but I suspect we’d often forget to keep them synchronized).
@jmbowman I see your point in using Github hosted workers to avoid having to setup Kubernetes cluster to run self-hosted runners. Do you have recommendations on that Kubernetes configuration in case we’d like to look into that further?
Jeremy: “… test shards is large enough to overwhelm the GitHub plan’s allotment of concurrent workers. We’re also using faster machines than the default hosted runners, giving us shorter test run durations.”
Will this work for Github hosted runs-on: ubuntu-20.04? It seems like it could take longer but will all the shards run?
Also when @feanil indicated that there was this Docker image Docker that use use for this unit-tests.yml workflow it made me think what additional software do we need to run these tests prior to the edx-platform installing requirements.
Looking at the workflow steps it appears MongoDB would need to start.
It looks like MongoDB is setup in the Docker image here.
I like your idea of using if conditions to factor in using self-hosted runners versus Github hosted runners may work but there doesn’t appear to be an else statement. Wonder the if expression should be a branch name instead of organization/repo.
if: github.repository == ‘openedx/edx-platform’
runs-on: [ edx-platform-runner ]
I’m also concerned about your comment. Will this work with Github hosted runners?
Jeremy: “test shards is large enough to overwhelm the GitHub plan’s allotment of concurrent workers”
It did what I thought it would do immediately and fail due to
Run sudo /etc/init.d/mongodb start
sudo: /etc/init.d/mongodb: command not found
Error: Process completed with exit code 1.
I made another change and it appears to be running now on Github hosted runners for the unit-tests.yml file.
I mimicked my code against this workflow.
@jmbowman Received the additional error when running GitHub hosted runner. Any idea how to resolve this?
Maybe you can look at my modifications to the unit-tests.yml configuration and give recommendations on how to fix.
@jmbowman I wonder if this would work.
name: unit-tests
on:
pull_request:
push:
branches:
- master
- open-release/*.master
jobs:
run-tests-master:
if: ${{ github.ref == 'refs/heads/master' }}
runs-on: [ edx-platform-runner ]
strategy:
matrix:
python-version: ['3.8']
django-version: ["3.2"]
shard_name: [
"lms-1",
"lms-2",
"lms-3",
"lms-4",
"lms-5",
"lms-6",
"openedx-1",
"openedx-2",
"cms-1",
"cms-2",
"common-1",
"common-2",
]
name: python-${{ matrix.python-version }},django-${{ matrix.django-version }},${{ matrix.shard_name }}
steps:
- name: sync directory owner
run: sudo chown runner:runner -R .*
- uses: actions/checkout@v2
- name: start mongodb service
run: |
sudo /etc/init.d/mongodb start
…
run-tests-open-release-master:
if: ${{ github.ref == 'refs/heads/open-release/*.master’ }}
runs-on: [ ubuntu-20.04 ]
strategy:
matrix:
python-version: ['3.8']
django-version: ["3.2"]
shard_name: [
"lms-1",
"lms-2",
"lms-3",
"lms-4",
"lms-5",
"lms-6",
"openedx-1",
"openedx-2",
"cms-1",
"cms-2",
"common-1",
"common-2",
]
name: python-${{ matrix.python-version }},django-${{ matrix.django-version }},${{ matrix.shard_name }}
steps:
- name: sync directory owner
run: sudo chown runner:runner -R .*
- uses: actions/checkout@v2
…
Looking at the documentation for action workflow syntax it appears that you can write that jobs.<job_id>.if before the runs-on portion.
Essentially I’ve created two separate jobs that are conditionally run based on branch name.
@jmbowman Whenever you get a second please just look at this latest PR. I setup two separate jobs in the workflow based on the branch reference name. This should allow the upstream Open edX job to use runs-on: [ self-hosted ] and the fork repo to use runs-on: [ubuntu-20.04]. Let me know what you think.
I’m getting errors on install requirements step within the run-tests-fork job, however, it’s probably how the pip cache directory works. If you have any suggestions on that please let me know.
@jmbowman
Any way that you can look over my workflow update and give recommendations on how to run this successfully on Github platform runner for the unit-tests.yml workflow?
How can I get past this install requirements error?
ERROR: Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
WARNING: You are using pip version 20.3.4; however, version 22.0.3 is available.
You should consider upgrading via the ‘/usr/bin/python3 -m pip install --upgrade pip’ command.
Error: Process completed with exit code 1.
As noted in another article it’s related to MongoDB.