Celery issues when running edX Juniper

Gonzalo_Gomez · July 29, 2020, 4:17pm

Hi,

We are trying to run edX Juniper.2 using Python 3.7.8 but we found out issues trying to run the celery workers

Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/djcelery/management/commands/celery.py", line 23, in run_from_argv
    ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 194, in run
    pool_cls = (concurrency.get_implementation(pool_cls) or
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/__init__.py", line 29, in get_implementation
    return symbol_by_name(cls, ALIASES)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module>
    from celery.concurrency.base import BasePool
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module>
    from celery.utils import timer2
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/utils/timer2.py", line 19
    from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
                   ^
SyntaxError: invalid syntax

We checked the base.txt requirements file and the celery version listed is 3.1.26.post2, which doesn’t support Python 3.7 according to its release information. Also, async is a reserved keyword in Python 3.7.

We tried to update celery to a newer version supporting this Python version, but it requires changing other dependencies. Do you know of any workaround to run celery? Are there any plans to upgrade celery in the short term?

Regards,
Gonzalo

Juan_M_Mendez · July 30, 2020, 1:59pm

Not sure if this is the issue here.
Are you sure that the system is using the right Python version?
In my case, a Debian GNU/Linux system, I had to update-alternatives to ensure we are using python3 and pip3, instead of python2.
In MacOS, I set aliases for python and pip pointing to python3 and pip3.

Gonzalo_Gomez · July 31, 2020, 11:06am

Hi @Juan_M_Mendez,

Thanks for checking. We are running the celery workers from a virtualenv with Python3 installed. And it seems to be working fine

(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python3
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python3
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ python3 -c "import os; print('Hello')"
Hello
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ python3 --version
Python 3.7.8

Also python is a symbolic link to python3 in that virtualenv

(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ ls -lart /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python
lrwxrwxrwx 1 admin admin 7 Jul 29 15:07 /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python -> python3
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python2
/usr/bin/python2

When trying to run the Celery worker, we keep getting the same error message pasted previously.

python3 ./manage.py lms --settings=bitnami celery worker --loglevel=info --queues=edx.lms.core.default --hostn
ame=edx.lms.core.default.%h --concurrency=1
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's do
cumentation for alternative uses
  import imp
2020-07-31 10:34:54,313 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/newrelic/console.py:7
1: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  args[1:], varargs, keywords, defaults)

2020-07-31 10:34:55,422 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/sorl/thumbnail/conf/_
_init__.py:16: RemovedInDjango30Warning: The DEFAULT_CONTENT_TYPE setting is deprecated.
  setattr(self, attr, getattr(obj, attr))

2020-07-31 10:34:55,423 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/sorl/thumbnail/conf/_
_init__.py:16: RemovedInDjango31Warning: The FILE_CHARSET setting is deprecated. Starting with Django 3.1, all files read from disk must be UTF-8 encoded.
  setattr(self, attr, getattr(obj, attr))

2020-07-31 10:34:56,013 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py:28
4: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  S = getargspec(self.run)

Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/djcelery/management/commands/celery.py", line 23, in run_from_argv
    ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 194, in run
    pool_cls = (concurrency.get_implementation(pool_cls) or
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/__init__.py", line 29, in get_implementation
    return symbol_by_name(cls, ALIASES)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module>
    from celery.concurrency.base import BasePool
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module>
    from celery.utils import timer2
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/utils/timer2.py", line 19
    from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
                   ^
SyntaxError: invalid syntax

In case it is not related to the Celery version as mentioned in my previous post, could you help us debug this? Can you tell us which Python 3.x version are you using?

Juan_M_Mendez · July 31, 2020, 11:21am

I have python 3.8.5 on a GNU/Linux box where I installed devstack (docker based)
For a native install, in an Ubuntu 16.04 VM, it is python 3.5.

Not sure if that could be the cause for the python and celery version mismatch.

Gonzalo_Gomez · August 10, 2020, 3:23pm

Hi @Juan_M_Mendez,

Thanks for the info. Can you check the Python version in use into the LMS container? We checked the devstack version for “juniper.2” and althought the host machine is running Python 3.7, the container image is running Python 3.5 and the Celery version we mentioned in a previous post that doesn’t support Python 3.7 but seems to support Python 3.5 (not officially though according to the version description in pypi.org).

Can you check if you get the same results? It is true that other services run on top of Python 3.8, but unfortunately not all of them. For example, the credentials container runs Python 3.8

Do you know if there is any plan to update those versions in all containers?

Juan_M_Mendez · August 10, 2020, 3:47pm

Hi @Gonzalo_Gomez:

 make lms-shell
docker-compose exec lms env TERM=xterm-256color /edx/app/edxapp/devstack.sh open

root@lms:/edx/app/edxapp/edx-platform# ps aux |grep -i python
root           1  1.8  0.0  19712  3260 pts/0    Ss+  15:39   0:00 bash /edx/app/edxapp/devstack.sh bash -c source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done
root           6  0.0  0.0  19712  3264 pts/0    S+   15:39   0:00 bash -c source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done
root           7 30.4  0.8 960048 263396 pts/0   S+   15:39   0:08 python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker
root          37 69.8  0.9 1070236 317620 pts/0  Sl+  15:39   0:11 /edx/app/edxapp/venvs/edxapp/bin/python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker
root          95  0.0  0.0  12948   932 pts/1    S+   15:39   0:00 grep --color=auto -i python
root@lms:/edx/app/edxapp/edx-platform# /edx/app/edxapp/venvs/edxapp/bin/python --version
Python 3.5.2
root@lms:/edx/app/edxapp/edx-platform# pip3 list|grep -i celery
celery                        3.1.26.post2 
django-celery                 3.3.1        
edx-celeryutils               0.5.1        
pylint-celery                 0.3

~/d/e/o/devstack (master|✔) $ make credentials-shell
docker-compose exec credentials env TERM=xterm-256color bash -c 'source /edx/app/credentials/credentials_env && cd /edx/app/credentials/credentials && /bin/bash'
root@credentials:/edx/app/credentials/credentials# python3 --version
Python 3.8.4
root@credentials:/edx/app/credentials/credentials# /edx/app/credentials/venvs/credentials/bin/python --version
Python 3.8.4

I have not worked on openedx development yet, so I don’t know.

Are you having this issue just be doing a make dev.provision, or make dev.up? Without having carved into your issue, sometimes for me, just restarting everything, solved some issues: make dev.reset It will take time to download all the docker images again though.

Gonzalo_Gomez · August 11, 2020, 9:56am

Hi @Juan_M_Mendez,

Thanks for checking this. According to the official Python downloads website, the Python 3.5 branch is under security maintenance until the next 13th of September

Do you know if we can ask anyone else in this forum about this or if there is any roadmap on updating this Python version and the required dependencies?

For our use case (http://bitnami.com/stack/edx), we provide our users with all-in-one products containing all the required tools to run Open edX, but we can’t provide multiple Python versions. That’s why we would like to know if supporting a newer Python version for all services in on your plans.

Regards,
Gonzalo

Juan_M_Mendez · August 11, 2020, 11:34am

Sorry, I don’t know. I just found this, that seems to be a roadmap to migrate to Python 3.8
https://openedx.atlassian.net/wiki/spaces/COMM/pages/1594000004/BD-06+Python+3.8+Upgrade

And since this is a free software project, people can join as contributors, I guess that with time we can get familiar with their JIRA to see the current tasks.
https://openedx.atlassian.net/secure/Dashboard.jspa

Gonzalo_Gomez · October 30, 2020, 6:24pm

Hello,

I checked the GitHub repo again and I see only the juniper.master branch. It is also using the celery==3.1.26.post2 package in the requirements/edx/base.txt file, which has the issue mentioned above with Python >= 3.6.x.

Does anyone have any update regarding this?

Gonzalo_Gomez · November 11, 2020, 8:27am

Just a quick note to mention that Python 3.5 reached its end-of-life.

I see some pull requests in GitHub regarding the celery version, but it looks like Python 3.5 will still be used. Not sure if there is any plan to update it as well

Topic		Replies	Views
JUNIPER.MASTER - installation error Site Operations Help juniper	1	1070	April 13, 2021
Juniper upgrade, getting error while restoring the data Site Operations Help juniper	2	554	October 27, 2020
ConnectionFailure in mongodb Site Operations Help how-to	2	355	January 19, 2021
Python, Django, and Juniper Releases juniper	13	2100	March 12, 2020
Create Superuser Issue - Open edX Juniper on AWS EC2 R5.Large Site Operations Help juniper	10	3418	August 10, 2021

Celery issues when running edX Juniper

Related topics