Please help with Ubuntu 20!

I can see the same behaviour on a new 20.04 server I just created. The logs/install-2020*.log file does not contain any useful information:

$ tail logs/install-20201103-141714.log 
skipping: [localhost] => (item=yum check-update --security) 
skipping: [localhost] => (item=yum update --security -y) 

TASK [demo : create demo app and data dirs] ************************************
changed: [localhost]

TASK [demo : check out the demo course] ****************************************
changed: [localhost]

TASK [demo : import demo course] ***********************************************

How can I view useful logs?

Could you run this in that failed instance?

source /edx/app/edxapp/edxapp_env
source /edx/app/edxapp/venvs/edxapp/bin/activate
python /edx/app/edxapp/edx-platform/manage.py cms  import  /edx/var/edxapp/data  /edx/app/demo/edx-demo-course

Maybe that could provide more information about the issue

Thanks @morenol! I’m getting stuck at updating the course overview:

 2020-11-03 15:51:43,789 INFO 110801 [openedx.core.djangoapps.content.course_overviews.models] [user None] [ip None] models.py:252 - Attempting to load CourseOverview for course course-v1:edX+DemoX+Demo_Course from modulestore.
2020-11-03 15:51:43,791 INFO 110801 [openedx.core.djangoapps.content.course_overviews.models] [user None] [ip None] models.py:172 - Updating course overview for course-v1:edX+DemoX+Demo_Course.

In the /edx/var/log/cms/edx.log file, the Demo Course starts importing with this message:

Nov  3 15:28:32 ip-10-0-0-254 [service_variant=cms][xmodule.modulestore.mixed][env:sandbox] INFO [ip-10-0-0-254  77021] [user None] [ip None] [mixed.py:663] - Creating course run course-v1:edX+DemoX+Demo_Course...

In the /edx/var/log/lms/edx/log file, I start having these error messages around the same time:

Nov  3 15:28:49 ip-10-0-0-254 [service_variant=lms][celery.worker.consumer.consumer][env:sandbox] ERROR [ip-10-0-0-254  55251] [user None] [ip None] [consumer.py:428] - consumer: Cannot connect to redis://:**@localhost:6379//: invalid password.
Trying again in 32.00 seconds... (16/100)

Coincidence?

Debugging with pdb the openedx/core/djangoapps/content/course_overview/models.py file (where the warning comes from) shows that the commands gets stuck at propagating a signal:

> /edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/dispatch/dispatcher.py(207)send_robust()
-> response = receiver(signal=self, sender=sender, **named)
(Pdb) pp(self)
SwitchedSignal('course_published')
(Pdb) pp(receiver)
<function update_block_structure_on_course_publish at 0x7f14ebf701f0>

I think @pierre’s intuition is correct: I’m reaching the following piece of code:

/edx/app/edxapp/edx-platform/openedx/core/djangoapps/content/block_structure/signals.py(31)update_block_structure_on_course_publish()
-> update_course_in_cache_v2.apply_async(...)

This async piece of code should not be blocking – unless the broker backend (redis) is unreachable. So we need to figure out why the Redis password is incorrect.

Redis is configured with a complex REDIS_PASSWORD. This password can be found in my-passwords.yml. But this password is not forwarded to the /edx/etc/studio.yml where the celery broker is configured without a password.

(I will not have time to investigate this further today.)

I will need to reinstall in order to confirm, but after changing the value of EDXAPP_CELERY_PASSWORD with the value assigned to REDIS_PASSWORD in /edx/etc/lms.yml and /edx/etc/studio.yml, I was able to restart the installation successfully. I reached the rbenv part of the installation thus far.

I may need to try this by reinstalling everything and making the change directly in my-passwords.yml before launching the native.sh script.

P.S. I meant CELERY_BROKER_PASSWORD in lms.yml and studio.yml.

1 Like

I have now reached one of the problems I had before using the branches suggested by @nedbat. I didn’t have time to investigate yet.

Elasticsearch and Mongo do not want to collaborate when we install the forums.

TASK [forum : initialize elasticsearch] ****************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/edx/app/forum/cs_comments_service/bin/rake", "search:initialize"], "delta": "0:00:02.473913", "end": "2020-11-03 17:05:47.410882", "msg": "non-zero return code", "rc": 1, "start": "2020-11-03 17:05:44.936969", "stderr": "/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT\n/edx/app/forum/cs_comments_service/lib/tasks/deep_search.rake:7: warning: previous definition of ROOT was here\n/edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: already initialized constant ROOT\n/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: previous definition of ROOT was here\n/edx/app/forum/cs_comments_service/models/constants.rb:2: warning: already initialized constant COURSE_ID\n/edx/app/forum/cs_comments_service/lib/tasks/db.rake:28: warning: previous definition of COURSE_ID was here\nrake aborted!\nElasticsearch::Transport::Transport::Errors::InternalServerError: [500] {\"error\":\"ClassCastException[java.lang.String cannot be cast to java.util.Map]\",\"status\":500}\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:346:in `perform_request'\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/client.rb:176:in `perform_request'\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'\n/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/actions/indices/create.rb:48:in `create'\n/edx/app/forum/cs_comments_service/lib/task_helpers.rb:92:in `block in create_indices'\n/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `each'\n/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `create_indices'\n/edx/app/forum/cs_comments_service/lib/task_helpers.rb:198:in `initialize_indices'\n/edx/app/forum/cs_comments_service/lib/tasks/search.rake:30:in `block (2 levels) in <top (required)>'\n/edx/app/forum/.gem/ruby/2.5.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'\nTasks: TOP => search:initialize\n(See full trace by running task with --trace)", "stderr_lines": ["/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT", "/edx/app/forum/cs_comments_service/lib/tasks/deep_search.rake:7: warning: previous definition of ROOT was here", "/edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: already initialized constant ROOT", "/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: previous definition of ROOT was here", "/edx/app/forum/cs_comments_service/models/constants.rb:2: warning: already initialized constant COURSE_ID", "/edx/app/forum/cs_comments_service/lib/tasks/db.rake:28: warning: previous definition of COURSE_ID was here", "rake aborted!", "Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {\"error\":\"ClassCastException[java.lang.String cannot be cast to java.util.Map]\",\"status\":500}", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:346:in `perform_request'", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/client.rb:176:in `perform_request'", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'", "/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/actions/indices/create.rb:48:in `create'", "/edx/app/forum/cs_comments_service/lib/task_helpers.rb:92:in `block in create_indices'", "/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `each'", "/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `create_indices'", "/edx/app/forum/cs_comments_service/lib/task_helpers.rb:198:in `initialize_indices'", "/edx/app/forum/cs_comments_service/lib/tasks/search.rake:30:in `block (2 levels) in <top (required)>'", "/edx/app/forum/.gem/ruby/2.5.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'", "Tasks: TOP => search:initialize", "(See full trace by running task with --trace)"], "stdout": "W, [2020-11-03T17:05:47.280139 #154006]  WARN -- : Overwriting existing field _id in class User.\nW, [2020-11-03T17:05:47.309354 #154006]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.\nW, [2020-11-03T17:05:47.309399 #154006]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.\nW, [2020-11-03T17:05:47.309417 #154006]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.", "stdout_lines": ["W, [2020-11-03T17:05:47.280139 #154006]  WARN -- : Overwriting existing field _id in class User.", "W, [2020-11-03T17:05:47.309354 #154006]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.", "W, [2020-11-03T17:05:47.309399 #154006]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.", "W, [2020-11-03T17:05:47.309417 #154006]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored."]}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost                  : ok=498  changed=188  unreachable=0    failed=1    skipped=1084 rescued=0    ignored=2   

 
============================================================
Ansible failed!
------------------------------------------------------------
 
Decoded error:
== cmd ===========================
['/edx/app/forum/cs_comments_service/bin/rake', 'search:initialize']
== msg ===========================
non-zero return code
== stderr ===========================
/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT
/edx/app/forum/cs_comments_service/lib/tasks/deep_search.rake:7: warning: previous definition of ROOT was here
/edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: already initialized constant ROOT
/edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: previous definition of ROOT was here
/edx/app/forum/cs_comments_service/models/constants.rb:2: warning: already initialized constant COURSE_ID
/edx/app/forum/cs_comments_service/lib/tasks/db.rake:28: warning: previous definition of COURSE_ID was here
rake aborted!
Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"ClassCastException[java.lang.String cannot be cast to java.util.Map]","status":500}
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/base.rb:346:in `perform_request'
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-transport-7.8.0/lib/elasticsearch/transport/client.rb:176:in `perform_request'
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'
/edx/app/forum/.gem/ruby/2.5.0/gems/elasticsearch-api-7.8.0/lib/elasticsearch/api/actions/indices/create.rb:48:in `create'
/edx/app/forum/cs_comments_service/lib/task_helpers.rb:92:in `block in create_indices'
/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `each'
/edx/app/forum/cs_comments_service/lib/task_helpers.rb:89:in `create_indices'
/edx/app/forum/cs_comments_service/lib/task_helpers.rb:198:in `initialize_indices'
/edx/app/forum/cs_comments_service/lib/tasks/search.rake:30:in `block (2 levels) in <top (required)>'
/edx/app/forum/.gem/ruby/2.5.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => search:initialize
(See full trace by running task with --trace)
== stdout ===========================
W, [2020-11-03T17:05:47.280139 #154006]  WARN -- : Overwriting existing field _id in class User.
W, [2020-11-03T17:05:47.309354 #154006]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
W, [2020-11-03T17:05:47.309399 #154006]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
W, [2020-11-03T17:05:47.309417 #154006]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.

Replacing the value of EDXAPP_CELERY_PASSWORD by the value of REDIS_PASSWORD before launching the native.sh script on a new server allowed me to install the demo course and reach the same point I am having trouble with right now.

TASK [forum : initialize elasticsearch]

There seems to be references to a previous version of ROOT being already initialized or previously defined in a few places under /edx/app/forum/cs_comments_service/lib/tasks/{deep_search.rake, flags.rake, kpis.rake) and /edx/app/forum/cs_comments_service/models/constants.rb

I am guessing this is an issue with multiple SINATRA_ENV being defined somewhere…

There are 2 instances of SINATRA_ENV in the configuration repo.

docker/build/forum/ansible_overrides.yml: FORUM_ **SINATRA_ENV** : "staging"
playbooks/roles/forum/defaults/main.yml: FORUM_ **SINATRA_ENV** : "development"

I will try again by modifying

FORUM_RACK_ENV: "development"
FORUM_SINATRA_ENV: "development"
devstack: "false"

in docker/build/forum/ansible_overrides.yml

UPDATE: That did not change a thing… There are also warning about COURSE_ID that I did not notice before.

UPDATE 2: I am wondering if the WARNING we got early in the native.sh process could be a part of the problem? I am no Mongo or forum expert.

[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/edxapp/defaults/main.yml, line 16, column 1, found a
duplicate dict key (EDXAPP_ENABLE_SPECIAL_EXAMS). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (RACK_ENV). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (SINATRA_ENV). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (SEARCH_SERVER). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (SEARCH_SERVER_ES7). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (MONGOHQ_URL). Using last defined value only.
[WARNING]: While constructing a mapping from
/var/tmp/configuration/playbooks/roles/forum/defaults/main.yml, line 84, column 3, found a
duplicate dict key (MONGOID_AUTH_MECH). Using last defined value only.

As of late Tuesday evening, I encountered problems with the EDXAPP_CELERY_PASSWORD and EDXAPP_CELERY_PASSWORD, and with the elasticsearch tasks (“initialize elasticsearch” and “rebuild elasticsearch indexes”) in the forum role. Not running these two tasks allowed me to complete the installation.

I haven’t checked if anything works yet, but I do have a running system.

BTW, one of the services still marked as “not upgraded” to Ubuntu 20 on the wiki page is XQueue. Maybe there’s not an issue to be solved, we need someone to look at whether it works or not.

One of the way I will be able to test XQueue will be with our own fork when we have a first alpha for Koa. Since we still use PDF certificates, and we plug ourself into the ID verification process instead of calling Software Secure, I know XQueue and certs are needed in order to process the generation of a PDF certificate. It’s already part of my manual tests for certs and auto-generated certs with each new release.

@sambapete FYI I am facing the same error when migrating the forums to Koa in Tutor:

Elasticsearch::Transport::Transport::Errors::InternalServerError: [500] {"error":"ClassCastException[java.lang.String cannot be cast to java.util.Map]","status":500}

I fear that this may be caused by the Elasticsearch downgrade from 7.8 (note the ```
elasticsearch-transport-7.8.0 module). Are the forums working on your platform?

Yes, you probably didn’t see that you need to tag a specific version of the forums for it to work with Ubuntu 20.04. It was mentionned by Joe Mulloy in one of the threads in Slack. I had suggested they should update the instructions at the top of this thread in Discourse, but I guess nobody had time to do it.

export OPENEDX_RELEASE=nedbat/tk.1028a
export CONFIGURATION_VERSION=nedbat/ub20
export FORUM_VERSION="609eef0fa2f4c69dba92ce2ef98c557b876e77b9"

From what @adzuci told us:

I don’t believe the commits beyond that commit aren’t causing issues with 20.04, but the newer commits don’t work with elasticsearch 1.5 which is what’s available via the native install.Your point about the remaining Ubuntu work is a good one. We want the Koa branch to be held back to before the elasticsearch 7 upgrade work but also want it to have any Ubuntu 20.04 changes. One way to do that is to cut the koa branch at the commit we’re pegging to and then cherry pick the Ubuntu 20.04 PRs onto the koa branch. @ned is there a cleaner way / am I over complicating things?

2 Likes

We have a good installation on Ubuntu 20 now, using the “nedbat/tk.1106c” branches, which is a trial run of what the Koa master branches will be.

2 Likes

Hello,

I tried it a fresh install on 20.04, rabbitmq fails

TASK [rabbitmq : add rabbit’s packagecloud repository] *************************
fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “E:The repository ‘https://packagecloud.io/rabbitmq/rabbitmq-server/ubuntu focal Release’ does not have a Release file.”}

Following a tutorial like


solves it, but requires tweeking

RABBITMQ_VERSION: 3.8.2-0ubuntu1.1
rabbitmq_repo: “deb https://dl.bintray.com/rabbitmq-erlang/debian focal erlang-22.x”
rabbitmq_repo_key: “https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
(should be adapted)

Quickly searched how to install 3.6.9 on Ubuntu20.04 and whether it would work but did not find anything.

Could be related to this.

We fixed the Rabbit/Redis confusion today, and the Koa master branches are now available.

4 Likes

same problem in Maple native installation also (faced same in koa also)
( I know Open edX doesn’t support native installation for maple still… Im running Maple native :wink:)

A quick hack is

/edx/bin/supervisorctl restart lms cms