Celery issues when running edX Juniper

Hi,

We are trying to run edX Juniper.2 using Python 3.7.8 but we found out issues trying to run the celery workers

Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/djcelery/management/commands/celery.py", line 23, in run_from_argv
    ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 194, in run
    pool_cls = (concurrency.get_implementation(pool_cls) or
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/__init__.py", line 29, in get_implementation
    return symbol_by_name(cls, ALIASES)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module>
    from celery.concurrency.base import BasePool
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module>
    from celery.utils import timer2
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/utils/timer2.py", line 19
    from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
                   ^
SyntaxError: invalid syntax

We checked the base.txt requirements file and the celery version listed is 3.1.26.post2, which doesn’t support Python 3.7 according to its release information. Also, async is a reserved keyword in Python 3.7.

We tried to update celery to a newer version supporting this Python version, but it requires changing other dependencies. Do you know of any workaround to run celery? Are there any plans to upgrade celery in the short term?

Regards,
Gonzalo

Not sure if this is the issue here.
Are you sure that the system is using the right Python version?
In my case, a Debian GNU/Linux system, I had to update-alternatives to ensure we are using python3 and pip3, instead of python2.
In MacOS, I set aliases for python and pip pointing to python3 and pip3.

Hi @Juan_M_Mendez,

Thanks for checking. We are running the celery workers from a virtualenv with Python3 installed. And it seems to be working fine

(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python3
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python3
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ python3 -c "import os; print('Hello')"
Hello
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ python3 --version
Python 3.7.8

Also python is a symbolic link to python3 in that virtualenv

(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ ls -lart /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python
lrwxrwxrwx 1 admin admin 7 Jul 29 15:07 /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/bin/python -> python3
(edxapp) admin@ip-172-31-0-44:~/edx-juniper.2-0/apps/edx/edx-platform$ which python2
/usr/bin/python2

When trying to run the Celery worker, we keep getting the same error message pasted previously.

python3 ./manage.py lms --settings=bitnami celery worker --loglevel=info --queues=edx.lms.core.default --hostn
ame=edx.lms.core.default.%h --concurrency=1
/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's do
cumentation for alternative uses
  import imp
2020-07-31 10:34:54,313 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/newrelic/console.py:7
1: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
  args[1:], varargs, keywords, defaults)

2020-07-31 10:34:55,422 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/sorl/thumbnail/conf/_
_init__.py:16: RemovedInDjango30Warning: The DEFAULT_CONTENT_TYPE setting is deprecated.
  setattr(self, attr, getattr(obj, attr))

2020-07-31 10:34:55,423 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/sorl/thumbnail/conf/_
_init__.py:16: RemovedInDjango31Warning: The FILE_CHARSET setting is deprecated. Starting with Django 3.1, all files read from disk must be UTF-8 encoded.
  setattr(self, attr, getattr(obj, attr))

2020-07-31 10:34:56,013 WARNING 11850 [py.warnings] [user None] warnings.py:110 - /home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py:28
4: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  S = getargspec(self.run)

Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/djcelery/management/commands/celery.py", line 23, in run_from_argv
    ['{0[0]} {0[1]}'.format(argv)] + argv[2:],
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/bin/worker.py", line 194, in run
    pool_cls = (concurrency.get_implementation(pool_cls) or
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/__init__.py", line 29, in get_implementation
    return symbol_by_name(cls, ALIASES)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module>
    from celery.concurrency.base import BasePool
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module>
    from celery.utils import timer2
  File "/home/admin/edx-juniper.2-0/apps/edx/venvs/edxapp/lib/python3.7/site-packages/celery/utils/timer2.py", line 19
    from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
                   ^
SyntaxError: invalid syntax

In case it is not related to the Celery version as mentioned in my previous post, could you help us debug this? Can you tell us which Python 3.x version are you using?

1 Like

I have python 3.8.5 on a GNU/Linux box where I installed devstack (docker based)
For a native install, in an Ubuntu 16.04 VM, it is python 3.5.

Not sure if that could be the cause for the python and celery version mismatch.