Updating tutor Lilac to Palm (now that Palm's released) fails

Site Operators Tutor Help
1

I previously documented my experiments trying to test upgrading to Palm before it was released, so that I could test things out and see what’s still broken. I had hoped that I would be able to use slightly modified instructions from there after Palm’s release to do the upgrade on the same VM, but for real this time. (In reality I don’t want to upgrade until that issue is fixed, but I’m being forced to, otherwise I’ll lose google analytics visibility. :-/)

Here are the steps I was trying, and how far I got.

Upgrade Docker (the same as before, works):

sudo apt-get update
sudo apt-get upgrade
 - This just installs any missing OS updates
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
 - This seems to be failing on the first try and needs to be run twice
sudo docker run hello-world
sudo usermod -a -G docker $USER
sudo reboot

docker run hello-world
 - To confirm permissions are correct

Upgrade Tutor (different, now that it’s released, fails):

tutor plugins list
 - then disable all my plugins
tutor config save
tutor local quickstart
 - reconfigure for test hostname, and confirm it's working before upgrade.
tutor --version
 = tutor, version 12.0.4 (OLD)
pip install --upgrade "tutor[full]"
tutor --version
 =  tutor, version 16.0.2 (NEW)
tutor plugins enable mfe
tutor plugins enable forum
tutor plugins list
 - confirm they're enabled
tutor config save
tutor local upgrade --from=lilac

This upgrade step fails with the same “KeyError: (‘blackboard’, ‘blackboardenterprisecustomerconfiguration’)” error reported here in a seemingly different context.

(hit post length limit, continuing in next post.)

2

While waiting for help, I also tried tutor images build openedx. That succeeded, but didn’t change the fact I was still getting the above error. So then instead I tried tutor images build all and that fails as per below (too long, had to pull out):

the crux of which is:

------
 > [discussions-common 3/5] RUN --mount=type=cache,target=/root/.npm,sharing=shared npm clean-install --no-audit --no-fund --registry=https://registry.npmjs.org/:
#0 108.0 npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
#0 111.8 npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
#0 137.8 npm ERR! code ECONNRESET
#0 137.8 npm ERR! network aborted
#0 137.8 npm ERR! network This is a problem related to network connectivity.
#0 137.8 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#0 137.8 npm ERR! network 
#0 137.8 npm ERR! network If you are behind a proxy, please make sure that the
#0 137.8 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#0 179.5 
#0 179.5 npm ERR! A complete log of this run can be found in:
#0 179.5 npm ERR!     /root/.npm/_logs/2023-06-29T14_39_17_177Z-debug-0.log
------
------
 > [learning-common 3/5] RUN --mount=type=cache,target=/root/.npm,sharing=shared npm clean-install --no-audit --no-fund --registry=https://registry.npmjs.org/:
#0 184.1 npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
#0 184.3 npm ERR! code ECONNRESET
#0 184.3 npm ERR! network aborted
#0 184.3 npm ERR! network This is a problem related to network connectivity.
#0 184.3 npm ERR! network In most cases you are behind a proxy or have bad network settings.
#0 184.3 npm ERR! network 
#0 184.3 npm ERR! network If you are behind a proxy, please make sure that the
#0 184.3 npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
#0 184.9 
#0 184.9 npm ERR! A complete log of this run can be found in:
#0 184.9 npm ERR!     /root/.npm/_logs/2023-06-29T14_39_17_049Z-debug-0.log
------
Dockerfile:341
--------------------
 339 |     ENV CPPFLAGS=-DPNG_ARM_NEON_OPT=0
 340 |     ENV PACT_SKIP_BINARY_INSTALL=true
 341 | >>> RUN --mount=type=cache,target=/root/.npm,sharing=shared npm clean-install --no-audit --no-fund --registry=$NPM_REGISTRY
 342 |     
 343 |     COPY --from=learning-src / /openedx/app
--------------------
ERROR: failed to solve: process "/bin/sh -c npm clean-install --no-audit --no-fund --registry=$NPM_REGISTRY" did not complete successfully: exit code: 1
Error: Command failed with status 1: docker buildx build --tag=docker.io/overhangio/openedx-mfe:16.0.0 --output=type=docker --cache-from=type=registry,ref=docker.io/overhangio/openedx-mfe:16.0.0-cache /home/ubuntu/.local/share/tutor/env/plugins/mfe/build/mfe

I just wanted to have this documented here for now in case others run into the same problem.

3

This is probaby related the issue, where the new buildkit/Docker would try install X MFEs concurrently,please see this post Npm ERR! code ERR_SOCKET_TIMEOUT when building mfe image for more context Document how to use BuildKit options to reduce resources consumption · Issue #125 · overhangio/tutor-mfe · GitHub

4

Thanks @ghassan, I tried those commands and confirmed a build of just mfe succeeded. (On one system, and froze on the other, still not sure why.) However, when I then went back to trying to complete a build all (assuming the mfe build would be cached), I get the following error:

docker buildx build --tag=docker.io/overhangio/openedx-mfe:16.0.0 --output=type=docker --cache-from=type=registry,ref=docker.io/overhangio/openedx-mfe:16.0.0-cache /home/ubuntu/.local/share/tutor/env/plugins/mfe/build/mfe
[+] Building 31.9s (9/141)                                                                                                                                                                                                                                  
 => [internal] load .dockerignore                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                   0.1s
 => => transferring dockerfile: 21.36kB                                                                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/caddy:2.6.4                                                                                                                                                                                         0.3s
 => [internal] load metadata for docker.io/library/node:18.16.0-bullseye-slim                                                                                                                                                                          0.1s
 => importing cache manifest from docker.io/overhangio/openedx-mfe:16.0.0-cache                                                                                                                                                                        0.2s
 => CACHED [base 1/4] FROM docker.io/library/node:18.16.0-bullseye-slim@sha256:2b05866efb50b6bbc642e6d1cef908372351ad981453b8a747b1d0c9b5777cd8                                                                                                        0.0s
 => => resolve docker.io/library/node:18.16.0-bullseye-slim@sha256:2b05866efb50b6bbc642e6d1cef908372351ad981453b8a747b1d0c9b5777cd8                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                                                      0.0s
 => => transferring context: 1.56kB                                                                                                                                                                                                                    0.0s
 => [production  1/11] FROM docker.io/library/caddy:2.6.4@sha256:f03baefceddeb48279d4cb77fbb65a555e5f72e4d5a0e631a39728c7fb20e94c                                                                                                                      0.0s
 => => resolve docker.io/library/caddy:2.6.4@sha256:f03baefceddeb48279d4cb77fbb65a555e5f72e4d5a0e631a39728c7fb20e94c                                                                                                                                   0.0s
 => ERROR [base 2/4] RUN apt update   && apt install -y git     gcc libgl1 libxi6 make     autoconf libtool pkg-config zlib1g-dev     python g++     libpng-dev     libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev                 31.1s
------                                                                                                                                                                                                                                                      
 > [base 2/4] RUN apt update   && apt install -y git     gcc libgl1 libxi6 make     autoconf libtool pkg-config zlib1g-dev     python g++     libpng-dev     libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev:                             
#0 0.291                                                                                                                                                                                                                                                    
#0 0.291 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                                                                                    
#0 0.291                                                                                                                                                                                                                                                    
#0 30.69 Err:1 http://deb.debian.org/debian bullseye InRelease                                                                                                                                                                                              
#0 30.69   Could not connect to deb.debian.org:80 (146.75.34.132), connection timed out
#0 30.71 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
#0 30.72 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
#0 30.91 Get:4 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [246 kB]
#0 31.00 Get:5 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.8 kB]
#0 31.01 Fetched 353 kB in 31s (11.5 kB/s)
#0 31.01 Reading package lists...
#0 31.04 Building dependency tree...
#0 31.05 Reading state information...
#0 31.05 All packages are up to date.
#0 31.05 W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease  Could not connect to deb.debian.org:80 (146.75.34.132), connection timed out
#0 31.05 W: Some index files failed to download. They have been ignored, or old ones used instead.
#0 31.05 
#0 31.05 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#0 31.05 
#0 31.06 Reading package lists...
#0 31.09 Building dependency tree...
#0 31.09 Reading state information...
#0 31.10 Package libgl1 is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libxi6 is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libjpeg-dev is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libpng-dev is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package pkg-config is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package make is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package gcc is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libgif-dev is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package librsvg2-dev is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package autoconf is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libtool is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 Package libcairo2-dev is not available, but is referred to by another package.
#0 31.10 This may mean that the package is missing, has been obsoleted, or
#0 31.10 is only available from another source
#0 31.10 
#0 31.10 E: Package 'gcc' has no installation candidate
#0 31.10 E: Package 'libgl1' has no installation candidate
#0 31.10 E: Package 'libxi6' has no installation candidate
#0 31.10 E: Package 'make' has no installation candidate
#0 31.10 E: Package 'autoconf' has no installation candidate
#0 31.10 E: Package 'libtool' has no installation candidate
#0 31.10 E: Package 'pkg-config' has no installation candidate
#0 31.10 E: Unable to locate package python
#0 31.10 E: Unable to locate package g+
#0 31.10 E: Package 'libpng-dev' has no installation candidate
#0 31.10 E: Package 'libcairo2-dev' has no installation candidate
#0 31.10 E: Unable to locate package libpango1.0-dev
#0 31.10 E: Couldn't find any package by glob 'libpango1.0-dev'
#0 31.10 E: Package 'libjpeg-dev' has no installation candidate
#0 31.10 E: Package 'libgif-dev' has no installation candidate
#0 31.10 E: Package 'librsvg2-dev' has no installation candidate
------
Dockerfile:4
--------------------
   3 |     
   4 | >>> RUN apt update \
   5 | >>>   && apt install -y git \
   6 | >>>     # required for cwebp-bin
   7 | >>>     gcc libgl1 libxi6 make \
   8 | >>>     # required for gifsicle, mozjpeg, and optipng (on arm)
   9 | >>>     autoconf libtool pkg-config zlib1g-dev \
  10 | >>>     # required for node-sass (on arm)
  11 | >>>     python g++ \
  12 | >>>     # required for image-webpack-loader (on arm)
  13 | >>>     libpng-dev \
  14 | >>>     # required for building node-canvas (on arm, for course-authoring)
  15 | >>>     # https://www.npmjs.com/package/canvas
  16 | >>>     libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev
  17 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt update   && apt install -y git     gcc libgl1 libxi6 make     autoconf libtool pkg-config zlib1g-dev     python g++     libpng-dev     libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev" did not complete successfully: exit code: 100
Error: Command failed with status 1: docker buildx build --tag=docker.io/overhangio/openedx-mfe:16.0.0 --output=type=docker --cache-from=type=registry,ref=docker.io/overhangio/openedx-mfe:16.0.0-cache /home/ubuntu/.local/share/tutor/env/plugins/mfe/build/mfe

Any other magic commands to get around that?

5

What’s the output of the following three commands?

$ tutor config printvalue EDX_PLATFORM_REPOSITORY 
$ tutor config printvalue EDX_PLATFORM_VERSION 
$ tutor mounts list
6

@regis here you go:

ubuntu@ip-172-31-83-146:~$ tutor config printvalue EDX_PLATFORM_REPOSITORY 
https://github.com/openedx/edx-platform.git
ubuntu@ip-172-31-83-146:~$ tutor config printvalue EDX_PLATFORM_VERSION 
open-release/palm.1
ubuntu@ip-172-31-83-146:~$ tutor mounts list
[]

Thank you

7

@ghassan any thoughts on this core upgrade blocker? Thank you.

8

Can you paste here the stacktrace of the error you’re observing? You mentioned it’s the same as another one that occurs with the devstack but we still need the full log.

9

Here you go @regis

ubuntu@ip-172-31-83-146:~$ tutor local upgrade --from=lilac
⚠️  This command only performs a partial upgrade of your Open edX platform. To perform a full upgrade, you should run `tutor local launch` (or `tutor dev launch` in development).
====================================
        Upgrading from Maple
====================================
Environment generated in /home/ubuntu/.local/share/tutor/env
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms sh -e -c './manage.py lms migrate user_tours'
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
[+] Creating 7/7
 ✔ Container tutor_local-smtp-1           Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-permissions-1    Recreated                                                                                                                                                                                                    0.1s 
 ✔ Container tutor_local-mysql-1          Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-redis-1          Recreated                                                                                                                                                                                                    6.1s 
 ✔ Container tutor_local-elasticsearch-1  Recreated                                                                                                                                                                                                    1.4s 
 ✔ Container tutor_local-mongodb-1        Recreated                                                                                                                                                                                                    0.1s 
 ✔ Container tutor_local-forum-1          Recreated                                                                                                                                                                                                   10.3s 
[+] Running 5/5
 ✔ Container tutor_local-permissions-1    Started                                                                                                                                                                                                      0.3s 
 ✔ Container tutor_local-redis-1          Started                                                                                                                                                                                                      0.4s 
 ✔ Container tutor_local-elasticsearch-1  Started                                                                                                                                                                                                      0.5s 
 ✔ Container tutor_local-mongodb-1        Started                                                                                                                                                                                                      0.6s 
 ✔ Container tutor_local-forum-1          Started                                                                                                                                                                                                      0.3s 
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
Operations to perform:
  Apply all migrations: user_tours
Running migrations:
  No migrations to apply.
Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 224, in handle
    executor.loader.project_state(),
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 335, in project_state
    return self.graph.make_state(nodes=nodes, at_end=at_end, real_apps=list(self.unmigrated_apps))
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 315, in make_state
    project_state = self.nodes[node].mutate_state(project_state, preserve=False)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 89, in mutate_state
    operation.state_forwards(self.app_label, new_state)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 222, in state_forwards
    model_state = state.models[app_label, self.model_name_lower]
KeyError: ('blackboard', 'blackboardenterprisecustomerconfiguration')
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms sh -e -c ./manage.py lms migrate user_tours

Thank you

10

I also found this post and it appears that this can be reproduced with just the tutor local run lms ./manage.py lms migrate command listed there, in the server’s current state:

ubuntu@ip-172-31-83-146:~$ tutor local run lms ./manage.py lms migrate
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms ./manage.py lms migrate
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
[+] Creating 6/0
 ✔ Container tutor_local-smtp-1           Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-mysql-1          Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-permissions-1    Created                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-redis-1          Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-elasticsearch-1  Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-forum-1          Running                                                                                                                                                                                                      0.0s 
[+] Running 2/1
 ✔ Container tutor_local-permissions-1  Started                                                                                                                                                                                                        0.3s 
 ✔ Container tutor_local-mongodb-1      Started                                                                                                                                                                                                        0.0s 
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
Operations to perform:
  Apply all migrations: admin, agreements, announcements, api_admin, assessment, auth, badges, blackboard, block_structure, bookmarks, branding, bulk_email, bulk_grades, bundles, calendar_sync, canvas, catalog, celery_utils, certificates, commerce, completion, consent, content_libraries, content_type_gating, contentserver, contenttypes, cornerstone, cors_csrf, course_action_state, course_apps, course_date_signals, course_duration_limits, course_goals, course_groups, course_home_api, course_live, course_modes, course_overviews, courseware, crawlers, credentials, credit, dark_lang, database_fixups, degreed, degreed2, demographics, discounts, discussions, django_celery_results, django_comment_common, django_notify, edx_name_affirmation, edx_proctoring, edx_when, edxval, email_marketing, embargo, enterprise, entitlements, experiments, external_user_ids, grades, instructor_task, integrated_channel, learner_pathway_progress, learning_sequences, lms_xblock, lti1p3_tool_config, lti_consumer, milestones, mobile_api, moodle, oauth2_provider, oauth_dispatch, organizations, outcome_surveys, program_enrollments, programs, redirects, rss_proxy, sap_success_factors, save_for_later, schedules, sessions, site_configuration, sites, social_django, splash, split_modulestore_django, staffgrader, static_replace, status, student, submissions, super_csv, support, survey, survey_report, system_wide_roles, teams, theming, third_party_auth, thumbnail, track, user_api, user_authn, user_tours, util, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_django
Running migrations:
  Applying agreements.0001_initial... OK
  Applying assessment.0007_staff_workflow_blank... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying badges.0004_badgeclass_badgr_server_slug... OK
  Applying integrated_channel.0003_contentmetadataitemtransmission_content_last_changed... OK
  Applying integrated_channel.0004_contentmetadataitemtransmission_enterprise_customer_catalog_uuid... OK
  Applying integrated_channel.0005_auto_20211005_1052... OK
  Applying integrated_channel.0006_contentmetadataitemtransmission_deleted_at... OK
  Applying integrated_channel.0007_delete_learnerdatatransmissionaudit... OK
  Applying integrated_channel.0008_genericlearnerdatatransmissionaudit... OK
  Applying integrated_channel.0009_auto_20220325_1757... OK
  Applying enterprise.0127_enterprisecatalogquery_uuid... OK
  Applying enterprise.0128_enterprisecatalogquery_generate_uuids... OK
  Applying enterprise.0129_enterprisecatalogquery_uuid_unique... OK
  Applying enterprise.0130_lms_customer_lp_search_help_text... OK
  Applying enterprise.0131_auto_20210517_0924... OK
  Applying enterprise.0132_auto_20210608_1921... OK
  Applying enterprise.0133_auto_20210608_1931... OK
  Applying enterprise.0134_enterprisecustomerreportingconfiguration_enable_compression... OK
  Applying enterprise.0135_adminnotification_adminnotificationfilter_adminnotificationread... OK
  Applying enterprise.0136_auto_20210629_2129... OK
  Applying enterprise.0137_enrollment_email_update... OK
  Applying enterprise.0138_bulkcatalogqueryupdatecommandconfiguration... OK
  Applying enterprise.0139_auto_20210803_1854... OK
  Applying enterprise.0140_update_enrollment_sources... OK
  Applying enterprise.0141_make_enterprisecatalogquery_title_unique... OK
  Applying enterprise.0142_auto_20210907_2059... OK
  Applying enterprise.0143_auto_20210908_0559... OK
  Applying enterprise.0144_auto_20211011_1030... OK
  Applying enterprise.0145_auto_20211013_1018... OK
  Applying enterprise.0146_enterprise_customer_invite_key... OK
  Applying enterprise.0147_auto_20211129_1949... OK
  Applying enterprise.0148_auto_20211129_2114... OK
  Applying enterprise.0149_invite_key_required_default_expiry_backfill... OK
  Applying enterprise.0150_invite_key_expiry_required... OK
  Applying enterprise.0151_add_is_active_to_invite_key... OK
  Applying enterprise.0152_add_should_inactivate_other_customers... OK
  Applying enterprise.0153_add_enable_browse_and_request... OK
  Applying integrated_channel.0010_genericenterprisecustomerpluginconfiguration... OK
  Applying integrated_channel.0011_contentmetadataitemtransmission_plugin_configuration_id... OK
  Applying integrated_channel.0012_alter_contentmetadataitemtransmission_unique_together... OK
  Applying integrated_channel.0013_auto_20220405_2311... OK
  Applying integrated_channel.0014_genericenterprisecustomerpluginconfiguration_dry_run_mode_enabled... OK
  Applying integrated_channel.0015_auto_20220718_2113... OK
  Applying integrated_channel.0016_contentmetadataitemtransmission_content_title... OK
  Applying integrated_channel.0017_contentmetadataitemtransmission_friendly_status_message... OK
  Applying integrated_channel.0018_genericlearnerdatatransmissionaudit_friendly_status_message... OK
  Applying integrated_channel.0016_contentmetadataitemtransmission_marked_for... OK
  Applying integrated_channel.0019_merge_20220928_1842... OK
  Applying integrated_channel.0020_auto_20220929_1712... OK
  Applying integrated_channel.0021_remove_contentmetadataitemtransmission_api_response_body... OK
  Applying blackboard.0002_auto_20220302_2231...Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 116, in apply
    operation.state_forwards(self.app_label, project_state)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 222, in state_forwards
    model_state = state.models[app_label, self.model_name_lower]
KeyError: ('blackboard', 'blackboardenterprisecustomerconfiguration')
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms ./manage.py lms migrate
ubuntu@ip-172-31-83-146:~$

Also FWIW tutor local run lms ./manage.py lms migrate backpopulate_user_tours doesn’t work, if that was needed for some reason.

ubuntu@ip-172-31-83-146:~$ tutor local run lms ./manage.py lms migrate backpopulate_user_tours
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms ./manage.py lms migrate backpopulate_user_tours
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
[+] Creating 6/0
 ✔ Container tutor_local-smtp-1           Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-mysql-1          Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-permissions-1    Created                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-redis-1          Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-elasticsearch-1  Running                                                                                                                                                                                                      0.0s 
 ✔ Container tutor_local-forum-1          Running                                                                                                                                                                                                      0.0s 
[+] Running 2/1
 ✔ Container tutor_local-permissions-1  Started                                                                                                                                                                                                        0.3s 
 ✔ Container tutor_local-mongodb-1      Started                                                                                                                                                                                                        0.0s 
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                     
CommandError: No installed app with label 'backpopulate_user_tours'.
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms ./manage.py lms migrate backpopulate_user_tours
11

You should be able to upgrade straight from Lilac to Palm, but the issue you are facing seems to indicate this is not the case. Can you please try to upgrade first to Nutmeg (pip install tutor==14.2.4), run tutor local upgrade --from=maple with that version, then upgrade to v16?

12

@regis I appear to be getting the same “pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64a32d4f96a67dd0d3cbcd7d, topology_type: Single, servers: [<ServerDescription (‘mongodb’, 27017) server_type: Unknown, rtt: None, error=AutoReconnect(‘mongodb:27017: [Errno -3] Temporary failure in name resolution’)>]>” error as mentioned before on the other thread.

{cut}
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/mongodb_proxy.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in find_courselike_blocks_by_id
    docs = [
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in <listcomp>
    docs = [
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
    if len(self.__data) or self._refresh():
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2027, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1976, in __start_session
    server_session = self._get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2013, in _get_server_session
    return self._topology.get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 525, in get_server_session
    session_timeout = self._check_session_support()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 507, in _check_session_support
    self._select_servers_loop(
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64a32d4f96a67dd0d3cbcd7d, topology_type: Single, servers: [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongodb:27017: [Errno -3] Temporary failure in name resolution')>]>
2023-07-03 20:32:03,065 ERROR 7 [mongodb_proxy] [user None] [ip None] mongodb_proxy.py:57 - Attempt 4
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/mongodb_proxy.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 996, in get_course_summaries
    for entry, structure_info in self._get_courselike_blocks_for_branch(branch, **kwargs):
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 880, in _get_courselike_blocks_for_branch
    for entry in self.find_courselike_blocks_by_id(version_guids, block_type):
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 602, in find_courselike_blocks_by_id
    return self.db_connection.find_courselike_blocks_by_id(list(ids), block_type)
  File "/openedx/venv/lib/python3.8/site-packages/mongodb_proxy.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in find_courselike_blocks_by_id
    docs = [
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in <listcomp>
    docs = [
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
    if len(self.__data) or self._refresh():
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2027, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1976, in __start_session
    server_session = self._get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2013, in _get_server_session
    return self._topology.get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 525, in get_server_session
    session_timeout = self._check_session_support()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 507, in _check_session_support
    self._select_servers_loop(
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64a32d4f96a67dd0d3cbcd7d, topology_type: Single, servers: [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongodb:27017: [Errno -3] Temporary failure in name resolution')>]>
Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 76, in handle
    self.enqueue_all_shuffled_tasks(options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 83, in enqueue_all_shuffled_tasks
    for seq_id, kwargs in enumerate(self._shuffled_task_kwargs(options)):
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 102, in _shuffled_task_kwargs
    for course_key in self._get_course_keys(options):
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 126, in _get_course_keys
    course_keys = [course.id for course in modulestore().get_course_summaries()]
  File "/openedx/edx-platform/xmodule/modulestore/mixed.py", line 84, in inner
    retval = func(field_decorator=strip_key_collection, *args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/mixed.py", line 292, in get_course_summaries
    for course_summary in store.get_course_summaries(**kwargs):
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split_draft.py", line 90, in get_course_summaries
    return super().get_course_summaries(
  File "/openedx/venv/lib/python3.8/site-packages/mongodb_proxy.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 996, in get_course_summaries
    for entry, structure_info in self._get_courselike_blocks_for_branch(branch, **kwargs):
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 880, in _get_courselike_blocks_for_branch
    for entry in self.find_courselike_blocks_by_id(version_guids, block_type):
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/split.py", line 602, in find_courselike_blocks_by_id
    return self.db_connection.find_courselike_blocks_by_id(list(ids), block_type)
  File "/openedx/venv/lib/python3.8/site-packages/mongodb_proxy.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in find_courselike_blocks_by_id
    docs = [
  File "/openedx/edx-platform/xmodule/modulestore/split_mongo/mongo_connection.py", line 353, in <listcomp>
    docs = [
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
    if len(self.__data) or self._refresh():
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2027, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1976, in __start_session
    server_session = self._get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2013, in _get_server_session
    return self._topology.get_server_session()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 525, in get_server_session
    session_timeout = self._check_session_support()
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 507, in _check_session_support
    self._select_servers_loop(
  File "/openedx/venv/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongodb:27017: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64a32d4f96a67dd0d3cbcd7d, topology_type: Single, servers: [<ServerDescription ('mongodb', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('mongodb:27017: [Errno -3] Temporary failure in name resolution')>]>
Error: Command failed with status 1: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm lms-job sh -e -c ./manage.py lms compute_grades -v1 --all_courses

Logs:

tutor local logs --tail=100 mongodb
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local logs --tail 100 mongodb
{cut}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:57:29.328+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:57:29.328+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:57:29.328+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.623+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.625+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.626+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.626+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.626+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.626+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.626+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:29.627+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:30.299+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:30.299+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:30.299+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:58:30.299+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.594+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.597+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.597+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.597+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.597+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.597+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.598+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:30.598+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:31.270+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:31.270+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:31.270+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T20:59:31.270+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.574+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.576+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.576+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.576+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.576+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.576+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.577+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:31.577+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:32.249+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:32.249+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:32.249+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:00:32.249+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.550+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.551+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:32.552+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:33.224+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:33.224+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:33.224+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:01:33.224+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.549+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.552+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.552+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.552+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.552+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.552+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.553+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:33.553+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:34.225+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:34.225+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:34.225+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:02:34.225+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.532+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.535+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.535+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"4c5e6ffca36b"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.535+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.22","gitVersion":"fc832685b99221cffb1f5bb5a4ff5ad3e1c416b2","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.535+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.535+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"storage":{"engine":"wiredTiger","journal":{"enabled":false}}}}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.536+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:34.536+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],,log=(enabled=false),"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:35.208+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:35.208+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:35.208+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
tutor_local-mongodb-1  | {"t":{"$date":"2023-07-03T21:03:35.208+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
13

@regis have you had a chance to review those logs, or have any other threads come up that would suggest the issue has been fixed and I should try again? (The inability to upgrade has meant I’ve been losing all Google Analytics data since July 1…)

14

The tutor local upgrade --from=maple command probably fails because you already upgraded your Mongodb instance to the more recent version. Do you have the possibility to start again from scratch?

15

@regis Yes, I am just reverting back to earlier snapshots of a clone of my production Amazon VM to test things out. I reverted and tried upgrading to nutmeg first per your suggestion (that’s what the latest logs were for showing it still failed), and I’d be happy to try a new suggestion.

16

@regis do you have something specifically you want me to be trying differently on the next try?

17

I’m stumped. At this point I’m not sure what’s the exact state of your installation, what error you are facing and what is the command that is causing it. So it’s difficult for me to provide further instructions… The best I can do is suggest you start from a “green” (i.e: functional) state and then perform a release-by-release upgrade.

19

Hey @ABC I am facing the same error when trying to upgrade from maple to palm. Did you solved your issue? If so, how? By upgrading state by state as @regis suggested?

20

I have not solved the error yet :frowning:

21

We run into the same problem while upgrading from Lilac (native) to Palm (tutor): “KeyError: (‘blackboard’, ‘blackboardenterprisecustomerconfiguration’)”. The only workaround I’ve found is to gradually upgrade from Lilac to Nutmeg, then to Palm. This way it works.

1 Like