Failed to get program UUIDs from the cache

Site Operators Site Operations Help
1

Hi everyone!

I’ve made an experiment. I’ve created an AWS Elasticache memcached, got it’s endpoint and updated the lms.env.json and cms.env.json. My idea was to segragate the service. I’ve changed everything that had localhost:11211 to :11211. But now, I get this error in my /edx/app/edxapp/lms.env.json:

“[service_variant=lms][openedx.core.djangoapps.catalog.utils][env:sandbox] WARNING [ip-172-35-35-35 1852] [utils.py:66] - Failed to get program UUIDs from the cache.”

Well, ok, I think I’ve messed up something, but I wold be really cool to have memcached runnig in a specific service… Does anyone knows what am i missing here?

2

Ok, made a few tests and I’ve discovered that this problem was going on even before the memcached change I’ve made. Does anyone knows what causes this?

3

@Scarback we use shared memcached services provisioned through AWS ElastiCache for Open edX servers, so this is definitely possible.

There are a few things that might be going wrong.

We use the 1.4.34 compatible version for memcached. If this is incorrect, there might be an issue.
Also, did you change localhost:11211 to the full address seen in the “Configuration Endpoint” on your ElastiCache dashboard in the AWS console?

Are there any other errors before that line in your logs?

Also, if you have changed the configuration, the new cache may not be populated. The program cache is generated by a management command, called cache_programs. You may try running this command in a shell:

/edx/bin/{python,manage}.edxapp lms --settings=aws cache_programs
4

Alternatively to re-cache the program IDs, you can visit a URL like the following, with the correct domain for your instance: http://localhost:18000/catalog/management/cache_programs/.

This will trigger the same management command I mentioned above.

5

@joshm, thank you very much for the tips! :smiley:

I’ve tryed to execute the command /edx/bin/{python,manage}.edxapp lms --settings=aws cache programs and I get this response:

root@ip-172-20-2-81:/home/ubuntu# /edx/bin/{python,manage}.edxapp lms --settings=aws cache_programs
2019-10-07 16:12:23,023 INFO 13318 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
2019-10-07 16:12:29,380 INFO 13318 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:32 - populate-multitenant-programs switch is ON
2019-10-07 16:12:39,433 ERROR 13318 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:41 - Failed to create API client. Service user lms_catalog_service_user does not exist.
Traceback (most recent call last):
** File “/edx/bin/manage.edxapp”, line 121, in **
** execute_from_command_line([sys.argv[0]] + django_args)**
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line**
** utility.execute()**
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/init.py”, line 356, in execute**
** self.fetch_command(subcommand).run_from_argv(self.argv)**
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py”, line 283, in run_from_argv**
** self.execute(args, cmd_options)
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py”, line 330, in execute*
** output = self.handle(args, options)
** File “/edx/app/edxapp/edx-platform/openedx/core/djangoapps/catalog/management/commands/cache_programs.py”, line 38, in handle*
** user = User.objects.get(username=username)**
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method**
** return getattr(self.get_queryset(), name)(args, kwargs)
** File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/db/models/query.py”, line 380, in get*
** self.model._meta.object_name**
django.contrib.auth.models.DoesNotExist: User matching query does not exist.

Unfortunately, when i tail -f /edx/var/log/lms/edx.log, I still get this error :pensive: :

Oct 7 19:16:45 ip-172-20-2-81 [service_variant=lms][openedx.core.djangoapps.catalog.utils][env:sandbox] WARNING [ip-172-20-2-81 15549] [utils.py:66] - Failed to get program UUIDs from the cache.
Oct 7 19:17:35 ip-172-20-2-81 [service_variant=lms][py.warnings][env:sandbox] WARNING [ip-172-20-2-81 15547] [init.py:1555] - /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/init.py:1145: DeprecationWarning: Use of … or absolute path in a resource path is not allowed and will raise exceptions in a future release.
** self, resource_name**
Oct 7 19:17:37 ip-172-20-2-81 [service_variant=lms][openedx.core.djangoapps.catalog.utils][env:sandbox] WARNING [ip-172-20-2-81 15546] [utils.py:66] - Failed to get program UUIDs from the cache.
Oct 7 19:17:38 ip-172-20-2-81 [service_variant=lms][openedx.core.djangoapps.catalog.utils][env:sandbox] WARNING [ip-172-20-2-81 15559] [utils.py:66] - Failed to get program UUIDs from the cache.
Oct 7 19:18:05 ip-172-20-2-81 [service_variant=lms][openedx.core.djangoapps.catalog.utils][env:sandbox] WARNING [ip-172-20-2-81 15559] [utils.py:66] - Failed to get program UUIDs from the cache.

6

This is getting my attention: Failed to create API client. Service user lms_catalog_service_user does not exist.

I had a problem before with the celery user, beacause the AWS AMI don’t copy this user when the AMI is created. Could be the same problem with this lms_catalog_service_user?

7

@joshm!

I was able to run this command by creating the user lms_catalog_service_user on my /admin.

Now, I get this output:

root@ip-172-20-2-81:/home/ubuntu# /edx/bin/{python,manage}.edxapp lms --settings =aws cache_programs
2019-10-07 21:34:16,447 INFO 10306 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
2019-10-07 21:34:22,473 INFO 10306 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:32 - populate-multitenant-programs switch is ON
2019-10-07 21:34:32,510 INFO 10306 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:49 - Skipping site example.com. No configuration.
2019-10-07 21:34:37,517 INFO 10306 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:69 - Caching details for 0 programs.

But still I get the Failed to get program UUIDs from the cache error from edx/var/log/lms/edx.log

Could this error have anything to do with this part of lms.env.json?

“JWT_AUTH”: {
“JWT_AUDIENCE”: “SET-ME-PLEASE”,
“JWT_ISSUER”: “http://127.0.0.1:8000/oauth2”,
“JWT_ISSUERS”: [
{
“AUDIENCE”: “SET-ME-PLEASE”,
“ISSUER”: “http://127.0.0.1:8000/oauth2”,
“SECRET_KEY”: “SET-ME-PLEASE”
}
],
“JWT_SECRET_KEY”: “SET-ME-PLEASE”

8

2 days trying, but still haven’t solved this problem :pensive:
Does anyone have any ideas?

9

Hello Scarback,

according to this code you need to add a Django site configuration for default site(example.com) in LMS admin interface - /admin/site_configuration/siteconfiguration/

With single configuration parameter:

{
  "COURSE_CATALOG_API_URL": "http://127.0.0.1:8381/api/v1",
}
10

Thank you, @andrey.kryachko! I fell we’re getting closer to the answer!

With this new configuration I’ve said, the command /edx/bin/{python,manage}.edxapp lms --settings=aws cache_programs now gives me another output:

root@ip-172-20-2-81:/home/ubuntu# /edx/bin/{python,manage}.edxapp lms --settings=aws cache_programs
2019-10-10 14:17:22,470 INFO 8948 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
2019-10-10 14:17:29,239 INFO 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:32 - populate-multitenant-programs switch is ON
2019-10-10 14:17:39,279 INFO 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:87 - Requesting program UUIDs for example.com.
2019-10-10 14:17:42,605 ERROR 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:90 - Failed to retrieve program UUIDs for site: example.com.
2019-10-10 14:17:42,605 INFO 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:95 - Received 0 UUIDs for site example.com
2019-10-10 14:17:42,606 INFO 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:64 - Caching UUIDs for 0 programs for site example.com.
2019-10-10 14:17:47,620 INFO 8948 [openedx.core.djangoapps.catalog.management.commands.cache_programs] cache_programs.py:69 - Caching details for 0 programs.

For some reason the application has not been able to receive UUIDs from programs

11

have you already configured the program on discovery side -http://127.0.0.1:8381/admin/admin/course_metadata/program ?

also the discovery service metadata and index must be populated before you can use /admin interface

export COURSE_DISCOVERY_CFG=/edx/etc/discovery.yml
sudo -Eu discovery /edx/app/discovery/venvs/discovery/bin/python /edx/app/discovery/discovery/manage.py refresh_course_metadata --settings=course_discovery.settings.production
sudo -Eu discovery /edx/app/discovery/venvs/discovery/bin/python /edx/app/discovery/discovery/manage.py update_index --disable-change-limit --settings=course_discovery.settings.production