Issue while configuring discovery service - Unable to refresh the course metadata

Site Operators Site Operations Help
1

Hello Team!

In the process configuring API communication to LMS from discovery service, I have created client id and secret from LMS admin and configured in the discovery.yml in
SOCIAL_AUTH_EDX_OAUTH2_KEY and SOCIAL_AUTH_EDX_OAUTH2_SECRET respectively and restarted the discovery service. I have also created a partner in the discovery admin panel https://test.discovery.example.com/admin/core/partner. I have given LMS URL, LMS Admin URL, Studio URL and Courses API URL parameters.

I have observed that the JWT tokens are already generated and I didn’t disturb them.

Now, when I tried to refresh the course metadata, I got the following error:

discovery@ip-:~/discovery$ ./manage.py refresh_course_metadata 2021-04-13 10:00:01,930 INFO 30229 [algoliasearch_django.registration] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/algoliasearch_django/registration.py:74 - REGISTER <class ‘course_discovery.apps.course_metadata.algolia_models.AlgoliaProxyProduct’> 2021-04-13 10:00:04,247 INFO 30229 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:116 - Command is not using threads to write data. 2021-04-13 10:00:04,254 INFO 30229 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [https://test.lms.example.com/api/courses/v1/] 2021-04-13 10:00:04,402 ERROR 30229 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - CoursesApiDataLoader failed! Traceback (most recent call last): File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py”, line 25, in execute_loader loader_class(*loader_args).ingest() File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 31, in init self.username = self.get_username_from_client(self.api_client) File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 41, in get_username_from_client token = client.get_jwt_access_token() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 270, in get_jwt_access_token self._ensure_authentication() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 254, in _ensure_authentication timeout=self._timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 170, in get_and_cache_oauth_access_token timeout=timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 115, in get_oauth_access_token response.raise_for_status() # Raise an exception for bad status codes. File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/models.py”, line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://test.lms.example.com/oauth2/access_token CommandError: One or more of the data loaders above failed.

Please let me know if the configuration was wrong and how should I rectify it.

Please help.

Thanks,
Mohan.

2

Hello.

Have you set partner informations (https://test.discovery.example.com/admin/core/partner/1/change/) :

Name: Open edX
Code abbrégé : edX
URL du LMS : https://test.lms.example.com/
LMS Admin URL : https://test.lms.example.com/admin/
L’url du Studio : https://test.cms.example.com/
URL de l’API Cours : https://test.lms.example.com/api/courses/v1/
URL de l’API Commerce électronique : https://test.ecommerce.example.com/api/v2/
URL de l’API Organisations : https://test.lms.example.com/api/organizations/v0/

?

3

@Herve_siyou

I’m still facing the same issue even after adding the API information of ecommerce and organization.

2021-04-13 10:57:02,448 INFO 3855 [algoliasearch_django.registration] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/algoliasearch_django/registration.py:74 - REGISTER <class ‘course_discovery.apps.course_metadata.algolia_models.AlgoliaProxyProduct’> 2021-04-13 10:57:03,957 INFO 3855 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:116 - Command is not using threads to write data. 2021-04-13 10:57:03,960 INFO 3855 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [https://test.lms.example.com/api/courses/v1/] 2021-04-13 10:57:04,052 ERROR 3855 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - CoursesApiDataLoader failed! Traceback (most recent call last): File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py”, line 25, in execute_loader loader_class(*loader_args).ingest() File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 31, in init self.username = self.get_username_from_client(self.api_client) File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 41, in get_username_from_client token = client.get_jwt_access_token() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 270, in get_jwt_access_token self._ensure_authentication() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 254, in _ensure_authentication timeout=self._timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 170, in get_and_cache_oauth_access_token timeout=timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 115, in get_oauth_access_token response.raise_for_status() # Raise an exception for bad status codes. File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/models.py”, line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://test.lms.example.com/oauth2/access_token 2021-04-13 10:57:04,057 INFO 3855 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [https://test.pay.example.com/api/v2/] 2021-04-13 10:57:04,108 ERROR 3855 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - EcommerceApiDataLoader failed! Traceback (most recent call last): File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py”, line 25, in execute_loader loader_class(*loader_args).ingest() File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/api.py”, line 295, in init super(EcommerceApiDataLoader, self).init(partner, api_url, max_workers, is_threadsafe, **kwargs) File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 31, in init self.username = self.get_username_from_client(self.api_client) File “/edx/app/discovery/discovery/course_discovery/apps/course_metadata/data_loaders/init.py”, line 41, in get_username_from_client token = client.get_jwt_access_token() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 270, in get_jwt_access_token self._ensure_authentication() File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 254, in _ensure_authentication timeout=self._timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 170, in get_and_cache_oauth_access_token timeout=timeout, File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/edx_rest_api_client/client.py”, line 115, in get_oauth_access_token response.raise_for_status() # Raise an exception for bad status codes. File “/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/requests/models.py”, line 941, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://test.lms.example.com/oauth2/access_token CommandError: One or more of the data loaders above failed. discovery@ip-:~/discovery$

Do you think the Client ID and Secret can be stored in discovery.yml? Using SOCIAL_AUTH_EDX_OAUTH2_KEY and SOCIAL_AUTH_EDX_OAUTH2_SECRET? Is it correct?

4

Edit /edx/etc/lms.yml :

COURSE_CATALOG_API_URL: “https://test.discovery.example.com/api/v1”,

Edit /edx/etc/discovery.yml :

EDX_DRF_EXTENSIONS:
OAUTH2_USER_INFO_URL: https://test.lms.example.comoauth2/user_info

JWT_AUTH:
JWT_ISSUERS:
- AUDIENCE: SET-ME-PLEASE
ISSUER: https://test.lms.example.com/oauth2
SECRET_KEY: SET-ME-PLEASE

on Juniper release:

SOCIAL_AUTH_EDX_OAUTH2_ISSUER: https://test.lms.example.com
SOCIAL_AUTH_EDX_OAUTH2_KEY: discovery-sso-key
SOCIAL_AUTH_EDX_OAUTH2_LOGOUT_URL: https://test.lms.example.com/logout
SOCIAL_AUTH_EDX_OAUTH2_SECRET: discovery-sso-secret
SOCIAL_AUTH_EDX_OAUTH2_URL_ROOT: https://test.lms.example.com
SOCIAL_AUTH_REDIRECT_IS_HTTPS: true

restart all services and try again

5

@Herve_siyou

Where should I configure the Client ID and secret that I have generated in LMS admin?
Please suggest.

6

@Herve_siyou

I have done the same as you suggested, but I’m still facing the same issue. Have you done any changes to JWT access tokens?

7

Ok, give me a moment, i will give you a step by step to do it

8

i do not have a Juniper Release now, i will install it today, and i will come back to you with the “step by step” process

9

Thanks a lot @Herve_siyou . Much appreciated.

10

I face somes issues installing juniper.master release, the installation failed:

============================================================

Ansible failed!

Decoded error:
== module_stderr ===========================
Traceback (most recent call last):
File “/root/.ansible/tmp/ansible-tmp-1618325747.33-82765245934478/AnsiballZ_mongodb_rs_status”, line 113, in
_ansiballz_main()
File “/root/.ansible/tmp/ansible-tmp-1618325747.33-82765245934478/AnsiballZ_mongodb_rs_status”, line 105, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File “/root/.ansible/tmp/ansible-tmp-1618325747.33-82765245934478/AnsiballZ_mongodb_rs_status”, line 48, in invoke_module
imp.load_module(‘ main ’, mod, module, MOD_DESC)
File “/tmp/ansible_mongodb_rs_status_payload_LwXOsM/ main .py”, line 133, in
File “/tmp/ansible_mongodb_rs_status_payload_LwXOsM/ main .py”, line 115, in main
File “/usr/local/lib/python2.7/dist-packages/pymongo/database.py”, line 529, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File “/usr/lib/python2.7/contextlib.py”, line 17, in enter
return self.gen.next()
File “/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py”, line 982, in _socket_for_reads
server = topology.select_server(read_preference)
File “/usr/local/lib/python2.7/dist-packages/pymongo/topology.py”, line 224, in select_server
address))
File “/usr/local/lib/python2.7/dist-packages/pymongo/topology.py”, line 183, in select_servers
selector, server_timeout, address)
File “/usr/local/lib/python2.7/dist-packages/pymongo/topology.py”, line 199, in _select_servers_loop
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused

11

@Herve_siyou

I’m new to Open edX and don’t know much about this error. Sorry about that. Thanks for your help.

12

@Herve_siyou

I found the issue. In Juniper, we have to configure the application access and the scope “user_id” to enable the API communication between services and LMS. We have to do this configuration in https://test.lms.example.com/admin/oauth_dispatch/applicationaccess/.

Reference URL: Found detailed explanation here.
https://openedx.atlassian.net/wiki/spaces/COMM/pages/1532395987/Setup+OAuth+Client+for+Internal+Services+Django+Oauth+Toolkit+version

13

@Herve_siyou

I could able to refresh the metadata and updated the index, however I’m unable to run the elasticsearch queries from the site. I can see a response like “The query returned no results.”

But, I can see the courses getting updated in the course metadata in discovery admin.

Error log info:
tail -f /edx/var/log/supervisor/discovery-stderr.log
[2021-04-14 13:41:04 +0000] [10161] [INFO] GET /api/v1/course_runs/

Please suggest.
Thanks.