Is this a platform that you created from scratch in Olive, or did it have some data dating back from Nutmeg? (I’m wondering whether we should first set the mongodb compatibility version to 4.2, before upgrading to 4.4.)
The system has been continuously upgraded in place since Koa or so.
How would I do that?
Thank you.
@regis when we were still running under Native, it was important to make sure to upgrade MongoDB properly, especially if moving from one version to the other required to make use of the compatibility version.
Our own platform was transferred to Tutor beginning with Koa. We are currently running Nutmeg in Production and Olive in a pre-production system. I am now wondering if I would encounter the same issues as @ABC when we try upgrading to Palm…
That explains everything… For some reason we skipped the Mongodb feature compatibility check (from 4.0 to 4.2) in Olive. And now, in Palm, we are trying to upgrade your 4.0 system straight to 4.4. The fix will be easy, but it’s good that we detected this issue early!
Please re-install tutor from the palm branch:
pip install -U git+https://github.com/overhangio/tutor@palm#egg=tutor
And run upgrade again:
tutor local upgrade --from=olive
This should upgrade your mongodb instance first to 4.2, then to 4.4.
EDIT: and yes @sambapete, of course it’s crucial to be able to upgrade a platform from one version to the next, so we are extremely careful about such upgrades.
Just doing this as-is didn’t work, as it seemed to still be trying to go to mongodb 4.4. However, when I reverted the VM and started again, I did see it transition through 4.2, and it did complete the upgrade. (Don’t forget when you’re writing up upgrade instructions that you’re going to have to tell people to manually upgrade their Docker to versions not necessarily supported by their OS package manager.)
Now there’s a new error on launch:
File "/openedx/edx-platform/common/djangoapps/course_modes/models.py", line 186, in CourseMode
settings.COURSE_MODE_DEFAULTS['android_sku'],
KeyError: 'android_sku'
Full error:
[+] Running 1/1
✔ Container tutor_local-permissions-1 Started 0.3s
[+] Building 0.0s (0/0)
2023/06/07 17:14:27 Ready: tcp://mysql:3306.
2023/06/07 17:14:27 Ready: tcp://mongodb:27017.
Loading settings lms.envs.tutor.production
Traceback (most recent call last):
File "./manage.py", line 103, in <module>
startup.run()
File "/openedx/edx-platform/lms/startup.py", line 20, in run
django.setup()
File "/openedx/venv/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/openedx/venv/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/openedx/venv/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
self.models_module = import_module(models_module_name)
File "/opt/pyenv/versions/3.8.15/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/openedx/edx-platform/common/djangoapps/student/models/__init__.py", line 4, in <module>
from .course_enrollment import *
File "/openedx/edx-platform/common/djangoapps/student/models/course_enrollment.py", line 36, in <module>
from common.djangoapps.course_modes.models import CourseMode, get_cosmetic_verified_display_price
File "/openedx/edx-platform/common/djangoapps/course_modes/models.py", line 44, in <module>
class CourseMode(models.Model):
File "/openedx/edx-platform/common/djangoapps/course_modes/models.py", line 186, in CourseMode
settings.COURSE_MODE_DEFAULTS['android_sku'],
KeyError: 'android_sku'
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm lms-job sh -e -c dockerize -wait tcp://mysql:3306 -timeout 20s
dockerize -wait tcp://mongodb:27017 -timeout 20s
echo "Loading settings $DJANGO_SETTINGS_MODULE"
./manage.py lms migrate
# Create oauth2 apps for CMS SSO
# https://github.com/openedx/edx-platform/blob/master/docs/guides/studio_oauth.rst
./manage.py lms manage_user cms cms@openedx --unusable-password
./manage.py lms create_dot_application \
--grant-type authorization-code \
--redirect-uris "https://studio.test.mysite.com/complete/edx-oauth2/" \
--client-id cms-sso \
--client-secret zcJcEOetznLHyjT2nBxIifPU \
--scopes user_id \
--skip-authorization \
--update cms-sso cms
./manage.py lms create_dot_application \
--grant-type authorization-code \
--redirect-uris "http://studio.test.mysite.com:8001/complete/edx-oauth2/" \
--client-id cms-sso-dev \
--client-secret zcJcEOetznLHyjT2nBxIifPU \
--scopes user_id \
--skip-authorization \
--update cms-sso-dev cms
# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
if [ -n "$(ls -A /openedx/data/uploads/)" ]; then
echo "Migrating LMS uploaded files to shared directory"
mv /openedx/data/uploads/* /openedx/media/
rm -rf /openedx/data/uploads/
fi
fi
# Create waffle switches to enable some features, if they have not been explicitly defined before
# Completion tracking: add green ticks to every completed unit
(./manage.py lms waffle_switch --list | grep completion.enable_completion_tracking) || ./manage.py lms waffle_switch --create completion.enable_completion_tracking on
(I don’t have the android plugin enabled FWIW.)
Speaking of plugins, tutor plugins update still doesn’t get me v16 versions of plugins, they’re all at 15
ubuntu@ip-172-31-18-14:~$ tutor plugins update
Error applying action: func=<function discover_package.<locals>.load at 0x7fd52011ab90> contexts=['plugins']'
⚠️ Failed to enable plugin 'forum': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7fd52011b760> contexts=['plugins']'
⚠️ Failed to enable plugin 'indigo_mine': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7fd52011ad40> contexts=['plugins']'
⚠️ Failed to enable plugin 'license': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7fd52011aef0> contexts=['plugins']'
⚠️ Failed to enable plugin 'mfe': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Fetching index https://overhang.io/tutor/main/palm/plugins.yml...
Failed to update index. Request error: HTTP Error 404: Not Found
Plugin index local cache: /home/ubuntu/.local/share/tutor/env/plugins/index/cache.yml
1 Like
@regis any thoughts on the above errors? (Palm release is coming up quick, and I’ve been trying to upgrade for 2 weeks ;))
Is any of your custom plugins overriding the COURSE_MODE_DEFAULTS setting? You should try to disable all plugins except for the MFE plugin:
tutor plugins disable all
tutor plugins enable mfe
tutor plugins update does not upgrade plugins. It only fetches the new versions from the index. To upgrade plugins, you should run tutor plugins upgrade. BUT this command is not going to work either because the plugin index has not been updated for Palm yet. It will be once Palm is out. In the meantime, you should upgrade plugins manually by pip-installing them.
@regis Yes, there was a single plugin still enabled:
name: set_default_enrollment
version: 0.1.0
patches:
lms-env: |
"COURSE_MODE_DEFAULTS": {
"name": "Honor",
"slug": "honor",
"bulk_sku": null,
"currency": "usd",
"description": null,
"expiration_datetime": null,
"min_price": 0,
"sku": null,
"suggested_prices": "",
"certificate_type": "honor"
}
That we picked up from here: How to convert all the enrollments from audit to professional - #7 by jaqusa, without which Certificates don’t work.
I did
ubuntu@ip-172-31-18-14:~$ git clone https://github.com/overhangio/tutor-mfe.git
Cloning into 'tutor-mfe'...
remote: Enumerating objects: 1045, done.
remote: Counting objects: 100% (365/365), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 1045 (delta 255), reused 293 (delta 226), pack-reused 680
Receiving objects: 100% (1045/1045), 1.43 MiB | 19.51 MiB/s, done.
Resolving deltas: 100% (507/507), done.
ubuntu@ip-172-31-18-14:~$ cd tutor-mfe/
ubuntu@ip-172-31-18-14:~/tutor-mfe$ git checkout palm
Branch 'palm' set up to track remote branch 'palm' from 'origin'.
Switched to a new branch 'palm'
ubuntu@ip-172-31-18-14:~/tutor-mfe$ pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/ubuntu/tutor-mfe
Preparing metadata (setup.py) ... done
Requirement already satisfied: tutor<17.0.0,>=16.0.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor-mfe==16.0.0) (16.0.0)
Requirement already satisfied: appdirs in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (1.4.4)
Requirement already satisfied: click>=8.0 in /usr/lib/python3/dist-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (8.0.3)
Requirement already satisfied: jinja2>=2.10 in /usr/lib/python3/dist-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (3.0.3)
Requirement already satisfied: kubernetes in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (25.3.0)
Requirement already satisfied: mypy in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (0.991)
Requirement already satisfied: pycryptodome in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (3.16.0)
Requirement already satisfied: pyyaml>=6.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (6.0)
Requirement already satisfied: typing-extensions>=4.4.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (4.4.0)
Requirement already satisfied: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (2020.6.20)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (1.16.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (2.8.2)
Requirement already satisfied: setuptools>=21.0.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (67.8.0)
Requirement already satisfied: google-auth>=1.0.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (2.15.0)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (1.2.3)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (2.25.1)
Requirement already satisfied: requests-oauthlib in /home/ubuntu/.local/lib/python3.10/site-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (1.3.1)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3/dist-packages (from kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (1.26.5)
Requirement already satisfied: mypy-extensions>=0.4.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from mypy->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (0.4.3)
Requirement already satisfied: tomli>=1.1.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from mypy->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (2.0.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (5.2.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth>=1.0.1->kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (0.2.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/ubuntu/.local/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (4.9)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib->kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (3.2.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4->google-auth>=1.0.1->kubernetes->tutor<17.0.0,>=16.0.0->tutor-mfe==16.0.0) (0.4.8)
Installing collected packages: tutor-mfe
Attempting uninstall: tutor-mfe
Found existing installation: tutor-mfe 15.0.6
Uninstalling tutor-mfe-15.0.6:
Successfully uninstalled tutor-mfe-15.0.6
Running setup.py develop for tutor-mfe
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tutor-ecommerce 15.0.2 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-ecommerce 15.0.2 requires tutor-mfe<16.0.0,>=15.0.0, but you have tutor-mfe 16.0.0 which is incompatible.
Successfully installed tutor-mfe
ubuntu@ip-172-31-18-14:~$ tutor plugins enable mfe
Plugin mfe enabled
Configuration saved to /home/ubuntu/.local/share/tutor/config.yml
You should now re-generate your environment with `tutor config save`.
ubuntu@ip-172-31-18-14:~$ tutor config save
Configuration saved to /home/ubuntu/.local/share/tutor/config.yml
Environment generated in /home/ubuntu/.local/share/tutor/env
ubuntu@ip-172-31-18-14:~$ tutor local launch
And I got the following error:
Configuration saved to /home/ubuntu/.local/share/tutor/config.yml
Environment generated in /home/ubuntu/.local/share/tutor/env
======================================
Building Docker images
======================================
No image to build
==============================================
Stopping any existing platform
==============================================
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local stop
[+] Stopping 11/0
✔ Container tutor_local-lms-worker-1 Stopped 0.0s
✔ Container tutor_local-cms-worker-1 Stopped 0.0s
✔ Container tutor_local-caddy-1 Stopped 0.0s
✔ Container tutor_local-cms-1 Stopped 0.0s
✔ Container tutor_local-lms-1 Stopped 0.0s
✔ Container tutor_local-elasticsearch-1 Stopped 0.0s
✔ Container tutor_local-mongodb-1 Stopped 0.0s
✔ Container tutor_local-smtp-1 Stopped 0.0s
✔ Container tutor_local-redis-1 Stopped 0.0s
✔ Container tutor_local-mysql-1 Stopped 0.0s
✔ Container tutor_local-permissions-1 Stopped 0.0s
======================================================
Starting the platform in detached mode
======================================================
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/dev/docker-compose.yml --project-name tutor_dev stop
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local up --remove-orphans -d
[+] Running 1/0
✘ mfe Error 0.1s
Error response from daemon: manifest for overhangio/openedx-mfe:16.0.0 not found: manifest unknown: manifest unknown
Error: Command failed with status 18: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local up --remove-orphans -d
Almost there:
tutor images build all
tutor local launch
Thanks @regis, I didn’t realize it would need a rebuild again.
It looks like it’s finally launched and working with palm! \o/
Time to see what’s broken! Starting with, why am I not able to use that previous plugin which is required to make certificates work by default for everyone?
name: set_default_enrollment
version: 0.1.0
patches:
lms-env: |
"COURSE_MODE_DEFAULTS": {
"name": "Honor",
"slug": "honor",
"bulk_sku": null,
"currency": "usd",
"description": null,
"expiration_datetime": null,
"min_price": 0,
"sku": null,
"suggested_prices": "",
"certificate_type": "honor"
}
This error implies that you need to add a key named “android_sku” to your configuration. But what should the value be? I don’t have a clue.
It looks like there are ‘android_sku’ and ‘ios_sku’ values now required (according to this code: edx-platform/models.py at master · openedx/edx-platform · GitHub)
Adding them as null seemed to get it back to working… but then when I checked an hour later it wasn’t working again, so I’m now debugging again…
For the record, as of right now, here’s my notional theory of how to upgrade from Olive to Palm before Palm is released, on a stock Ubuntu 22.04 system (which I’ll need to re-confirm again after reverting the VM snapshot):
1. Upgrade docker per https://docs.docker.com/engine/install/ubuntu/
1. sudo apt-get update
2. sudo apt-get install ca-certificates curl gnupg
3. sudo install -m 0755 -d /etc/apt/keyrings
4. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
5. sudo chmod a+r /etc/apt/keyrings/docker.gpg
6. echo \
7. "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
8. "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
9. sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
10. sudo apt-get update
11. sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
12. sudo reboot
13. sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
14. sudo docker run hello-world
2. Update tutor & plugins
1. pip install -U git+https://github.com/overhangio/tutor@palm#egg=tutor
2. tutor config save
3. git clone https://github.com/overhangio/tutor-mfe.git
4. cd tutor-mfe
5. git checkout palm
6. pip install -e .
7. git clone https://github.com/overhangio/tutor-forum.git
8. cd tutor-forum
9. git checkout palm
10. pip install -e .
11. tutor plugins enable mfe
12. tutor config save
13. tutor images build all
14. tutor local upgrade --from=olive
15. tutor local launch
3 Likes
@regis how can I upgrade tutor-indigo for palm? It doesn’t seem to have a palm branch like the other tutor-* repos.
The Indigo palm branch was actually created from a forked repo: upgrade to palm by gondaljutt · Pull Request #43 · overhangio/tutor-indigo · GitHub
pip install --user --upgrade --editable git+https://github.com/gondaljutt/tutor-indigo.git@jalal/palm#egg=tutor-indigo
@regis did you change tutor-forum so that there’s not a palm branch anymore (like with indigo)?
user@uVM:~$ git clone https://github.com/overhangio/tutor-forum.git
Cloning into 'tutor-forum'...
remote: Enumerating objects: 256, done.
remote: Counting objects: 100% (180/180), done.
remote: Compressing objects: 100% (101/101), done.
remote: Total 256 (delta 94), reused 139 (delta 68), pack-reused 76
Receiving objects: 100% (256/256), 46.07 KiB | 476.00 KiB/s, done.
Resolving deltas: 100% (105/105), done.
user@uVM:~$ cd tutor-forum/
user@uVM:~/tutor-forum$ git checkout palm
error: pathspec 'palm' did not match any file(s) known to git
This because it’s now released, i.e you should be install them through pip with version 16 refs tutor-forum · PyPI and tutor · PyPI
1 Like
I tried a bunch of things mentioned here, but still do not see indigo showing up in list:
~/.local/share/tutor-plugins/src/tutor-indigo$ tutor plugins list
⚠️ Failed to enable plugin 'indigo': plugin 'indigo' is not installed.
NAME STATUS VERSION
android installed 16.0.0
discovery installed 16.0.0
ecommerce installed 16.0.0
forum installed 16.0.0
license installed 16.0.0
mfe ✅ enabled 16.0.0
notes ✅ enabled 16.0.0
webui installed 16.0.0
xqueue installed 16.0.0
~/.local/share/tutor-plugins/src/tutor-indigo$ docker version
Client:
Version: 20.10.21
API version: 1.41
Go version: go1.18.1
Git commit: 20.10.21-0ubuntu1~20.04.2
Built: Thu Apr 27 05:56:19 2023
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go1.18.1
Git commit: 20.10.21-0ubuntu1~20.04.2
Built: Thu Apr 27 05:37:01 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.9-0ubuntu1~20.04.6
GitCommit:
runc:
Version: 1.1.4-0ubuntu1~20.04.3
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
:~/.local/share/tutor-plugins/src/tutor-indigo$ docker compose version
Docker Compose version v2.18.1
I am having the same issue in local launch and getting the following error
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local stop
unknown shorthand flag: 'f' in -f
See 'docker --help'.
It seems docker compose should be replaced by docker-compose in the new docker env.
I had the same issue as you until I realised that I had an older version of docker. New version uses a compose plugin so “docker-compose” is not the correct program to use anymore. Make sure you install latest version of docker as posted in @ABC’s solution above.
1 Like