Upgrading from Nutmeg to Olive Redis Error

Development
, ,
1

I am currently on the process of upgrading my Open Edx instance from Koa to Olive and successfully migrated native to tutor.

I followed @sambapete’s instructions on
this discussion here.

On the part when I’ve reached nutmeg version, I am trying to upgrade it to olive using the following commands:

tutor local stop
sudo curl -L "https://github.com/overhangio/tutor/releases/download/v15.3.3/tutor-$(uname -s)_$(uname -m)" -o /usr/local/bin/tutor
sudo chmod 0755 /usr/local/bin/tutor
tutor config save

then after that, I’ve used the command:
tutor local upgrade --from=nutmeg

but then I encountered this error:


Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
    return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/openedx/venv/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 645, in _connect
    for res in socket.getaddrinfo(
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
    yield
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 949, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 927, in create_channel
    channel = self.Channel(connection)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 737, in __init__
    self.client.ping()
  File "/openedx/venv/lib/python3.8/site-packages/redis/commands/core.py", line 1132, in ping
    return self.execute_command("PING", **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 1235, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error -3 connecting to redis:6379. Temporary failure in name resolution.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 76, in handle
    self.enqueue_all_shuffled_tasks(options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 85, in enqueue_all_shuffled_tasks
    result = tasks.compute_grades_for_course_v2.apply_async(kwargs=kwargs, **task_options)
  File "/openedx/venv/lib/python3.8/site-packages/celery_utils/logged_task.py", line 24, in apply_async
    result = super().apply_async(args=args, kwargs=kwargs, **options)
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
    return app.send_task(
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
    return _publish(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
    return fun(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
    channel = self.channel
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
    value = self.__value__ = self.__contract__()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 225, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
    self._ensure_connection(**conn_opts)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: Error -3 connecting to redis:6379. Temporary failure in name resolution.

How can I resolve this issue?

2

I have never encountered that error before.

Have you tried installing with pip instead of the binaries?

pip install "tutor[full]==v15.3.3"

We haven’t upgraded our production server to Olive yet, but we haven’t encountered any issues on our test server.

I am concerned about the following error message you shared;
socket.gaierror: [Errno -3] Temporary failure in name resolution

Have you tried restarting redis manually? It may simply be down.

3

I already tried it. Also, I can’t use the pip install method on my end since my previous native installed open edx was built from the binary release.

4

We need a bigger stacktrace what is going on here. At what step of the process is this error occurring? What are the logs surrounding this error? Are all containers up?

5

I’m facing the same problem.

After the command tutor local upgrade --from=nutmeg

I see:

docker-compose -f /home/user/.local/share/tutor/env/local/docker-compose.yml -f /home/user/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/user/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/user/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/user/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm lms-job sh -e -c './manage.py lms compute_grades -v1 --all_courses'

Starting tutor_local_mongodb-permissions_1 ... done
Starting tutor_local_mysql_1               ... done
Starting tutor_local_mongodb_1             ... done
Creating tutor_local_lms-job_run           ... done
Traceback (most recent call last):

After that comes the traceback

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
    return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/openedx/venv/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 645, in _connect
    for res in socket.getaddrinfo(
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
    yield
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 949, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 927, in create_channel
    channel = self.Channel(connection)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/transport/redis.py", line 737, in __init__
    self.client.ping()
  File "/openedx/venv/lib/python3.8/site-packages/redis/commands/core.py", line 1132, in ping
    return self.execute_command("PING", **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/redis/client.py", line 1235, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/openedx/venv/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error -3 connecting to redis:6379. Temporary failure in name resolution.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 76, in handle
    self.enqueue_all_shuffled_tasks(options)
  File "/openedx/edx-platform/lms/djangoapps/grades/management/commands/compute_grades.py", line 85, in enqueue_all_shuffled_tasks
    result = tasks.compute_grades_for_course_v2.apply_async(kwargs=kwargs, **task_options)
  File "/openedx/venv/lib/python3.8/site-packages/celery_utils/logged_task.py", line 24, in apply_async
    result = super().apply_async(args=args, kwargs=kwargs, **options)
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
    return app.send_task(
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/openedx/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
    return _publish(
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
    return fun(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
    channel = self.channel
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
    value = self.__value__ = self.__contract__()
  File "/openedx/venv/lib/python3.8/site-packages/kombu/messaging.py", line 225, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
    self._ensure_connection(**conn_opts)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/openedx/venv/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: Error -3 connecting to redis:6379. Temporary failure in name resolution.
Error: Command failed with status 1: docker-compose -f /home/user/.local/share/tutor/env/local/docker-compose.yml -f /home/user/.local/share/tutor/env/local/docker-compose.prod.yml -f /home/user/.local/share/tutor/env/local/docker-compose.tmp.yml --project-name tutor_local -f /home/user/.local/share/tutor/env/local/docker-compose.jobs.yml -f /home/user/.local/share/tutor/env/local/docker-compose.jobs.tmp.yml run --rm lms-job sh -e -c ./manage.py lms compute_grades -v1 --all_courses

As a note, I tried both of the following download methods, and both end up failing the same way:

  1. pip install "tutor[full]==v15.3.3"
  2. sudo curl -L "https://github.com/overhangio/tutor/releases/download/v15.3.3/tutor-$(uname -s)_$(uname -m)" -o /usr/local/bin/tutor sudo chmod 0755 /usr/local/bin/tutor
6

What’s the output of docker ps | grep redis?

7

Here’s the output:

36331721af04   redis:6.2.6     "docker-entrypoint.s…"   3 days ago   Up 3 days   6379/tcp     tutor_local_redis_1
8

Are you running a fork of edx-platform? What’s the output of:

tutor config printvalue EDX_PLATFORM_REPOSITORY
tutor config printvalue EDX_PLATFORM_VERSION
tutor local run lms git log -1
9

It’s not a fork, it’s vanilla except for one custom plugin which just sets some configuration.

Here’s the output of those commands:

tutor config printvalue EDX_PLATFORM_REPOSITORY: https://github.com/openedx/edx-platform.git

tutor config printvalue EDX_PLATFORM_VERSION: 
open-release/olive.3

tutor local run lms git log -1: 
commit a8cc8d08430b665b448e32362e5bbf74cfe8175b (HEAD)
Author: Maria Grimaldi <maria.grimaldi@edunext.co>
Date:   Tue Dec 6 11:30:06 2022 -0400
10

What’s the output of tutor local logs --tail=500 redis?

11

It seems to be this kind of thing repeated many times:

redis_1                      | 871:C 17 May 2023 14:57:26.121 * DB saved on disk
redis_1                      | 871:C 17 May 2023 14:57:26.154 * RDB: 1 MB of memory used by copy-on-write
redis_1                      | 1:M 17 May 2023 14:57:26.282 * Background saving terminated with success
redis_1                      | 1:M 17 May 2023 15:02:27.044 * 10 changes in 300 seconds. Saving...
redis_1                      | 1:M 17 May 2023 15:02:27.061 * Background saving started by pid 872
redis_1                      | 872:C 17 May 2023 15:02:44.856 * DB saved on disk
redis_1                      | 872:C 17 May 2023 15:02:44.898 * RDB: 1 MB of memory used by copy-on-write
redis_1                      | 1:M 17 May 2023 15:02:45.083 * Background saving terminated with success
redis_1                      | 1:M 17 May 2023 15:07:46.088 * 10 changes in 300 seconds. Saving...
redis_1                      | 1:M 17 May 2023 15:07:46.120 * Background saving started by pid 873
redis_1                      | 873:C 17 May 2023 15:08:07.060 * DB saved on disk
redis_1                      | 873:C 17 May 2023 15:08:07.095 * RDB: 1 MB of memory used by copy-on-write
redis_1                      | 1:M 17 May 2023 15:08:07.188 * Background saving terminated with success
12

Can you try to bring down the network and containers and then launch again?

tutor local dc down
tutor local launch -I
1 Like
13

Thanks. I’ve run those commands. The site appears to be fully functional. Should I also run tutor local upgrade --from=nutmeg?

14

Running this upgrade command does not hurt, so sure you can go ahead.

15

It ran successfully this time!

Thank you for taking the time to help.

16

Just to confirm I ran into this same issue attempting to upgrade from Nutmeg to Olive, and these commands worked as a workaround to allow the process to complete.

@Engr_James_Lusuegro can you mark this as the solution?

17

Hello @kit and @regis ,

i got an below error when ran the
tutor dev launch -I command

[+] Running 3/3
 ✘ discussions Error                                                                                                                                                                                   3.3s 
 ✘ course-authoring Error                                                                                                                                                                              3.3s 
 ✘ authn Error                                                                                                                                                                                         3.3s 
Error response from daemon: pull access denied for course-authoring, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

and when I ran
tutor dev upgrade --from=nutmeg this command then got an below error

Usage: tutor dev [OPTIONS] COMMAND [ARGS]...
Try 'tutor dev -h' for help.

Error: No such command 'upgrade'.

Please suggest some solutions or stpes as i want to upgrade from nutmeg to olive
and i used pip to install olive.

Thanks in advance.

18

Hello. I cannot mark this as a solution yet because it worked for kit, but not for me. It still doesn’t work.