My manually-generated course-completion certificates seem to disappear (every day?)

Development
1

I am using Koa via Tutor.

I manually generated course completion certificates for ~30 students who had taken a class on my beta testing platform. Somehow the certificates keep disappearing (I’m not sure yet, but I suspect they disappear each day.)

When I come back to the site after disappearance, the certificates page shows no created certs:

certs-disappeared
certs-disappeared984×870 20.5 KB

I then click “Generate Exception Certificates” for “All users on the Exception list”. I then see this success in the logs:

lms-worker_1 | 2021-08-13 12:41:22,791 INFO 1056 [celery.app.trace] [user None] [ip None] trace.py:125 - Task lms.djangoapps.instructor_task.tasks.generate_certificates[2b641d4d-ff74-4c64-a092-3fd66b276aac] succeeded in 16.66959141427651s: {‘action_name’: ‘certificates generated’, ‘attempted’: 32, ‘succeeded’: 32, ‘skipped’: 0, ‘failed’: 0, ‘total’: 32, ‘preassigned’: 0, ‘duration_ms’: 16649, ‘step’: ‘Generating Certificates’}

And then the certificates show back up and work (but seemingly with a wrong date? Unless that’s maybe the first date I made them…):

certs-reappeared
certs-reappeared1042×682 19.8 KB

But then they disappear again at some unknown later time.

What could I use to search for log messages which might tell me why the certificates are disappearing?

Is it possible that students completing the class normally and collecting non-manually-generated certs could be triggering something which breaks manually-generated certs?

2

hi @Jimmy_Wu strange indeed, one thing you could do to check if the certificates actually get deleted or they always exist (and maybe this is a frontend problem) is to go to /admin/certificates/generatedcertificate/ and see if the certificates are there when you don’t see them.

3

It appears the certificates continue to exist at that URL. However when I log in as my test user, despite the existence of the certificate, instead of seeing the “View {classname} Certificate” button, I see “Grade required for a {classname} Certificate: 97%.” text, and lack of button.

Also, the URL for the certificates break whenever the problem happens.

OK, and the “good” news here is that apparently the certificates for the first class I re-generated 5 hours ago have already broken :unamused: . So now at least I have less logs to search through I guess…

There appear to be multiple repeated instances of this text in the log (I grepped for “certificate”)

lms-worker_1     | 2021-08-13 12:41:08,847 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:319 - Certificate generated for student qwerty1 in the course: course-v1:bla+bla+1 with template: certificate-template-bla-bla.pdf. given template: None, user is verified: False, mode is verified: False,generate_pdf is: False
lms-worker_1     | 2021-08-13 12:41:08,848 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:350 - Could not retrieve grade for student 53 in the course 'course-v1:bla+bla+1' because an exception occurred while parsing the grade contents 'None' as HTML. The exception was: 'expected string or bytes-like object'
lms-worker_1     | 2021-08-13 12:41:08,848 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:366 - Student 53 is whitelisted in 'course-v1:bla+bla+1'
lms-worker_1     | 2021-08-13 12:41:08,854 INFO 1056 [root] [user None] [ip None] queue.py:469 - certificate generated for user: qwerty1 with generate_pdf status: False
lms-worker_1     | 2021-08-13 12:41:08,856 INFO 1056 [edx.certificate] [user None] [ip None] api.py:229 - Queued Certificate Generation task for 53 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 12:41:08,857 INFO 1056 [tracking] [user None] [ip None] logger.py:42 - {"name": "edx.certificate.created", "context": {"org_id": "bla", "course_id": "course-v1:bla+bla+1"}, "username": "", "session": "", "ip": "", "agent": "", "host": "", "referer": "", "accept_language": "", "event": {"user_id": 53, "course_id": "course-v1:bla+bla+1", "certificate_id": "2ea8e086d8cd401696a935d5caed63de", "enrollment_mode": "honor", "generation_mode": "batch", "certificate_url": "/certificates/2ea8e086d8cd401696a935d5caed63de"}, "time": "2021-08-13T12:41:08.857216+00:00", "event_type": "edx.certificate.created", "event_source": "server", "page": null}
lms-worker_1     | 2021-08-13 12:41:09,310 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:319 - Certificate generated for student qwerty2 in the course: course-v1:bla+bla+1 with template: certificate-template-bla-bla.pdf. given template: None, user is verified: False, mode is verified: False,generate_pdf is: False
lms-worker_1     | 2021-08-13 12:41:09,312 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:350 - Could not retrieve grade for student 48 in the course 'course-v1:bla+bla+1' because an exception occurred while parsing the grade contents 'None' as HTML. The exception was: 'expected string or bytes-like object'
lms-worker_1     | 2021-08-13 12:41:09,312 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:366 - Student 48 is whitelisted in 'course-v1:bla+bla+1'
lms-worker_1     | 2021-08-13 12:41:09,317 INFO 1056 [root] [user None] [ip None] queue.py:469 - certificate generated for user: qwerty2 with generate_pdf status: False
lms-worker_1     | 2021-08-13 12:41:09,320 INFO 1056 [edx.certificate] [user None] [ip None] api.py:229 - Queued Certificate Generation task for 48 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 12:41:09,320 INFO 1056 [tracking] [user None] [ip None] logger.py:42 - {"name": "edx.certificate.created", "context": {"org_id": "bla", "course_id": "course-v1:bla+bla+1"}, "username": "", "session": "", "ip": "", "agent": "", "host": "", "referer": "", "accept_language": "", "event": {"user_id": 48, "course_id": "course-v1:bla+bla+1", "certificate_id": "8f7fb4d9f9124f38bf2e015ffa7a3d77", "enrollment_mode": "honor", "generation_mode": "batch", "certificate_url": "/certificates/8f7fb4d9f9124f38bf2e015ffa7a3d77"}, "time": "2021-08-13T12:41:09.320471+00:00", "event_type": "edx.certificate.created", "event_source": "server", "page": null}
4

Here is the non-grepped full log of an example of 2 instances of the repeating error:

lms-worker_1     | 2021-08-13 12:41:07,218 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:319 - Certificate generated for student user1 in the course: course-v1:bla+bla+1 with template: certificate-template-bla-bla.pdf. given template: None, user is verified: False, mode is verified: False,generate_pdf is: False
cms_1            | 
lms_1            | 2021-08-13 16:45:43,035 INFO 8 [tracking] [user None] [ip 123.123.123.1] logger.py:42 - {"name": "/", "context": {"user_id": null, "path": "/", "course_id": "", "org_id": ""}, "username": "", "session": "2d873c94aa08f2ebfede7675689b7010", "ip": "123.123.123.1", "agent": "Mozilla/5.0 (Linux; Android 11; SAMSUNG SM-A125F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/13.2 Chrome/83.0.4103.106 Mobile Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/about", "accept_language": "en-GB,en-US;q=0.9,en;q=0.8", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2021-08-13T16:45:43.035256+00:00", "event_type": "/", "event_source": "server", "page": null}
mongodb_1        | 2021-08-10T16:56:12.915+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:35346 #625 (59 connections now open)
cms-worker_1     | 
forum_1          | 172.18.0.9 - - [13/Aug/2021:14:17:07 +0000] "GET /api/v1/db6fc8d1ee632e05082b0c8e99aae2a65ca1635a/threads?page=1&per_page=20&course_id=course-v1%3Abla%2Bbla%2B1&sort_key=activity&user_id=782&context=course&request_id=29ee47a9-caa2-4eaa-b7da-bc949f7f5684 HTTP/1.1" 200 820 0.0128
forum_1          | I, [2021-08-13T14:17:34.334180 #34]  INFO -- : source=rack-timeout id=9b18a590ebc52c16387127e5a3d97a35 timeout=20000ms state=ready
forum_1          | I, [2021-08-13T14:17:34.356331 #34]  INFO -- : source=rack-timeout id=9b18a590ebc52c16387127e5a3d97a35 timeout=20000ms service=22ms state=completed
lms-worker_1     | 2021-08-13 12:41:07,219 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:350 - Could not retrieve grade for student 57 in the course 'course-v1:bla+bla+1' because an exception occurred while parsing the grade contents 'None' as HTML. The exception was: 'expected string or bytes-like object'
caddy_1          | {"level":"error","ts":1628345444.5889046,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            | Traceback (most recent call last):
lms_1            | [pid: 8|app: 0|req: 409814/1436111] 172.18.0.12 () {54 vars in 1391 bytes} [Fri Aug 13 16:45:43 2021] GET / => generated 32792 bytes in 123 msecs (HTTP/1.0 200) 6 headers in 349 bytes (1 switches on core 0)
lms_1            | 2021-08-13 16:45:52,057 INFO 22 [tracking] [user 3] [ip 123.123.123.2] logger.py:42 - {"name": "/courses/course-v1:bla+bla+1/instructor/api/list_instructor_tasks", "context": {"course_id": "course-v1:bla+bla+1", "course_user_tags": {"view-welcome-message": "False"}, "user_id": 3, "path": "/courses/course-v1:bla+bla+1/instructor/api/list_instructor_tasks", "org_id": "bla"}, "username": "Admin", "session": "c5497850e29d5fbdbbbe109135599e5b", "ip": "123.123.123.2", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/instructor", "accept_language": "en-US,en;q=0.9", "event": "{\"GET\": {}, \"POST\": {}}", "time": "2021-08-13T16:45:52.057232+00:00", "event_type": "/courses/course-v1:bla+bla+1/instructor/api/list_instructor_tasks", "event_source": "server", "page": null}
mongodb_1        | 2021-08-10T16:56:12.923+0000 I NETWORK  [conn625] received client metadata from 172.18.0.11:35346 conn625: { driver: { name: "PyMongo", version: "3.10.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1029-aws" }, platform: "CPython 3.8.6.final.0" }
cms-worker_1     | Traceback (most recent call last):
forum_1          | 172.18.0.9 - - [13/Aug/2021:14:17:34 +0000] "PUT /api/v1/users/1870?request_id=9d75c674-2816-4e30-a7f5-1a83fdaf901b HTTP/1.1" 200 47 0.0233
lms-worker_1     | 2021-08-13 12:41:07,219 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:366 - Student 57 is whitelisted in 'course-v1:bla+bla+1'
caddy_1          | {"level":"error","ts":1628345448.009807,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/video_module/video_handlers.py", line 337, in transcript
lms_1            | [pid: 22|app: 0|req: 410533/1436112] 172.18.0.12 () {62 vars in 2071 bytes} [Fri Aug 13 16:45:52 2021] POST /courses/course-v1:bla+bla+1/instructor/api/list_instructor_tasks => generated 17 bytes in 58 msecs (HTTP/1.0 200) 9 headers in 582 bytes (1 switches on core 0)
mongodb_1        | 2021-08-10T16:56:13.185+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:35348 #626 (60 connections now open)
cms-worker_1     |   File "/openedx/venv/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
forum_1          | I, [2021-08-13T14:17:57.240081 #37]  INFO -- : source=rack-timeout id=1ef008f94f52fd614b0dc93f751cee39 timeout=20000ms state=ready
lms-worker_1     | 2021-08-13 12:41:07,225 INFO 1056 [root] [user None] [ip None] queue.py:469 - certificate generated for user: user1 with generate_pdf status: False
caddy_1          | {"level":"error","ts":1628346067.1527333,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |     content, filename, mimetype = get_transcript(
lms_1            | 2021-08-13 16:45:52,410 INFO 8 [tracking] [user 1688] [ip 123.123.123.3] logger.py:42 - {"name": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@video+block@0c86b62749d9450e88c7669aaa33a9b4/handler/xmodule_handler/save_user_state", "context": {"course_id": "course-v1:bla+bla+1", "course_user_tags": {}, "user_id": 1688, "path": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@video+block@0c86b62749d9450e88c7669aaa33a9b4/handler/xmodule_handler/save_user_state", "org_id": "bla"}, "username": "rwxget", "session": "46c79ff90b394bc5bf3932b24e18fb79", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/59e2e6a6c7e74172ab7c681dbb96330f/fc7058c8f5404fc7b90875fe1913659f/?child=first", "accept_language": "en-GB,en;q=0.5", "event": "{\"GET\": {}, \"POST\": {\"saved_video_position\": [\"00:01:22\"]}}", "time": "2021-08-13T16:45:52.409972+00:00", "event_type": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@video+block@0c86b62749d9450e88c7669aaa33a9b4/handler/xmodule_handler/save_user_state", "event_source": "server", "page": null}
mongodb_1        | 2021-08-10T16:56:13.186+0000 I NETWORK  [conn626] received client metadata from 172.18.0.11:35348 conn626: { driver: { name: "PyMongo", version: "3.10.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1029-aws" }, platform: "CPython 3.8.6.final.0" }
mongodb_1        | 2021-08-10T16:56:13.289+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:35350 #627 (61 connections now open)
forum_1          | I, [2021-08-13T14:17:57.252500 #37]  INFO -- : source=rack-timeout id=1ef008f94f52fd614b0dc93f751cee39 timeout=20000ms service=12ms state=completed
lms-worker_1     | 2021-08-13 12:41:07,228 INFO 1056 [edx.certificate] [user None] [ip None] api.py:229 - Queued Certificate Generation task for 57 : course-v1:bla+bla+1
caddy_1          | {"level":"error","ts":1628348833.2765143,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/video_module/transcripts_utils.py", line 1060, in get_transcript
mongodb_1        | 2021-08-10T16:56:13.289+0000 I NETWORK  [conn627] received client metadata from 172.18.0.11:35350 conn627: { driver: { name: "PyMongo", version: "3.10.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1029-aws" }, platform: "CPython 3.8.6.final.0" }
mongodb_1        | 2021-08-10T16:56:13.289+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:35352 #628 (62 connections now open)
redis_1          | 8040:C 12 Aug 2021 05:13:12.127 * RDB: 2 MB of memory used by copy-on-write
lms_1            | 2021-08-13 16:45:52,422 INFO 22 [tracking] [user 1688] [ip 123.123.123.3] logger.py:42 - {"name": "pause_video", "context": {"user_id": 1688, "path": "/event", "course_id": "course-v1:bla+bla+1", "org_id": "bla"}, "username": "rwxget", "session": "46c79ff90b394bc5bf3932b24e18fb79", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/59e2e6a6c7e74172ab7c681dbb96330f/fc7058c8f5404fc7b90875fe1913659f/?child=first", "accept_language": "en-GB,en;q=0.5", "event": "{\"id\": \"0c86b62749d9450e88c7669aaa33a9b4\", \"code\": \"Sr7VUql1wT8\", \"duration\": 1393, \"currentTime\": 82.36285309346009}", "time": "2021-08-13T16:45:52.421446+00:00", "event_type": "pause_video", "event_source": "browser", "page": "https://bla.com/courses/course-v1:bla+bla+1/courseware/59e2e6a6c7e74172ab7c681dbb96330f/fc7058c8f5404fc7b90875fe1913659f/?child=first"}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:44 +0000] http://bla.com "POST /event HTTP/1.1" 200 7 "https://bla.com/courses/course-v1:bla+Dbg1011_WinDbg1+1/courseware/6178d296890a499c97b26eb6c3d0788f/29274cde69094eefa772f7e6c85febb7/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67" "98.223.208.142"
forum_1          | 172.18.0.9 - - [13/Aug/2021:14:17:57 +0000] "GET /api/v1/users/1870?complete=True&request_id=38f6457d-4438-4998-a28d-6ab9d61f0154 HTTP/1.1" 200 223 0.0131
caddy_1          | {"level":"error","ts":1628350396.7578673,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |     return get_transcript_from_contentstore(
lms-worker_1     | 2021-08-13 12:41:07,229 INFO 1056 [tracking] [user None] [ip None] logger.py:42 - {"name": "edx.certificate.created", "context": {"org_id": "bla", "course_id": "course-v1:bla+bla+1"}, "username": "", "session": "", "ip": "", "agent": "", "host": "", "referer": "", "accept_language": "", "event": {"user_id": 57, "course_id": "course-v1:bla+bla+1", "certificate_id": "e7dba9f53c3b4271b2d392ffa3e61678", "enrollment_mode": "honor", "generation_mode": "batch", "certificate_url": "/certificates/e7dba9f53c3b4271b2d392ffa3e61678"}, "time": "2021-08-13T12:41:07.228954+00:00", "event_type": "edx.certificate.created", "event_source": "server", "page": null}
mongodb_1        | 2021-08-10T16:56:13.290+0000 I NETWORK  [conn628] received client metadata from 172.18.0.11:35352 conn628: { driver: { name: "PyMongo", version: "3.10.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1029-aws" }, platform: "CPython 3.8.6.final.0" }
cms-worker_1     |     R = retval = fun(*args, **kwargs)
redis_1          | 1:M 12 Aug 2021 05:13:12.182 * Background saving terminated with success
redis_1          | 1:M 12 Aug 2021 05:18:13.045 * 10 changes in 300 seconds. Saving...
lms_1            | [pid: 22|app: 0|req: 410534/1436113] 172.18.0.12 () {60 vars in 1905 bytes} [Fri Aug 13 16:45:52 2021] POST /event => generated 7 bytes in 30 msecs (HTTP/1.0 200) 7 headers in 360 bytes (1 switches on core 0)
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:45 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 149 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | I, [2021-08-13T14:17:57.265330 #37]  INFO -- : source=rack-timeout id=2b49b851c39c05907ea604868b17b269 timeout=20000ms state=ready
caddy_1          | {"level":"error","ts":1628350396.7579746,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/video_module/transcripts_utils.py", line 72, in wrapper
lms-worker_1     | 2021-08-13 12:41:07,676 INFO 1056 [lms.djangoapps.grades.course_grade_factory] [user None] [ip None] course_grade_factory.py:223 - Grades: Update, Course: course_key: course-v1:bla+bla+1, version: 61166743f3bf03a202cfc03b, edited_on: 2021-08-13 12:36:19.705000+00:00, grading_policy: 07//7l6rp2+Iomia2ACBrZKoFkI=, User: 33, Course Grade: percent: 0.0, letter_grade: None, passed: False, persisted: False
mongodb_1        | 2021-08-11T02:03:16.580+0000 I NETWORK  [conn627] end connection 172.18.0.11:35350 (61 connections now open)
cms-worker_1     |   File "/openedx/venv/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
redis_1          | 1:M 12 Aug 2021 05:18:13.047 * Background saving started by pid 8041
lms_1            | [pid: 8|app: 0|req: 409815/1436114] 172.18.0.12 () {60 vars in 2340 bytes} [Fri Aug 13 16:45:52 2021] POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@video+block@0c86b62749d9450e88c7669aaa33a9b4/handler/xmodule_handler/save_user_state => generated 17 bytes in 86 msecs (HTTP/1.0 200) 6 headers in 324 bytes (1 switches on core 0)
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:45 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 150 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:46 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 151 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | D, [2021-08-13T14:17:57.265623 #37] DEBUG -- : source=rack-timeout id=2b49b851c39c05907ea604868b17b269 timeout=20000ms service=0ms state=active
caddy_1          | {"level":"error","ts":1628350396.757906,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |     return func(*args, **kwds)
lms-worker_1     | 2021-08-13 12:41:07,681 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:319 - Certificate generated for student user2 in the course: course-v1:bla+bla+1 with template: certificate-template-bla-bla.pdf. given template: None, user is verified: False, mode is verified: False,generate_pdf is: False
mongodb_1        | 2021-08-11T02:03:16.580+0000 I NETWORK  [conn628] end connection 172.18.0.11:35352 (60 connections now open)
cms-worker_1     |     return self.run(*args, **kwargs)
redis_1          | 8041:C 12 Aug 2021 05:18:14.019 * DB saved on disk
lms_1            | 2021-08-13 16:45:52,711 INFO 22 [tracking] [user 634] [ip 123.123.123.3] logger.py:42 - {"name": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4/handler/toggle_button", "context": {"course_id": "course-v1:bla+bla+1", "course_user_tags": {}, "user_id": 634, "path": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4/handler/toggle_button", "org_id": "bla"}, "username": "neongen", "session": "12cb0a4a64e1ff02d74fb08315290f8a", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/8a24513bf3d1491ba663801ce5ea9f96/d584444aa35e43138018f4c05e262ba5/?child=first", "accept_language": "en-US,en;q=0.9", "event": "{\"GET\": {}, \"POST\": {\"{\\\"done\\\":true}\": [\"\"]}}", "time": "2021-08-13T16:45:52.711241+00:00", "event_type": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4/handler/toggle_button", "event_source": "server", "page": null}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:46 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 152 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:46 +0000] http://bla.com "POST /event HTTP/1.1" 200 7 "https://bla.com/courses/course-v1:bla+Dbg1011_WinDbg1+1/courseware/6178d296890a499c97b26eb6c3d0788f/29274cde69094eefa772f7e6c85febb7/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67" "98.223.208.142"
forum_1          | I, [2021-08-13T14:17:57.272842 #37]  INFO -- : source=rack-timeout id=2b49b851c39c05907ea604868b17b269 timeout=20000ms service=8ms state=completed
caddy_1          | {"level":"error","ts":1628350396.7579138,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"client disconnected"}
cms_1            |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/video_module/transcripts_utils.py", line 940, in get_transcript_from_contentstore
lms-worker_1     | 2021-08-13 12:41:07,682 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:350 - Could not retrieve grade for student 33 in the course 'course-v1:bla+bla+1' because an exception occurred while parsing the grade contents 'None' as HTML. The exception was: 'expected string or bytes-like object'
mongodb_1        | 2021-08-11T02:03:16.580+0000 I NETWORK  [conn625] end connection 172.18.0.11:35346 (59 connections now open)
cms-worker_1     |   File "/openedx/edx-platform/lms/djangoapps/grades/tasks.py", line 104, in compute_grades_for_course_v2
redis_1          | 8041:C 12 Aug 2021 05:18:14.023 * RDB: 2 MB of memory used by copy-on-write
lms_1            | 2021-08-13 16:45:52,730 INFO 8 [tracking] [user 634] [ip 123.123.123.3] logger.py:42 - {"name": "edx.done.toggled", "context": {"user_id": 634, "path": "/event", "course_id": "course-v1:bla+bla+1", "org_id": "bla"}, "username": "neongen", "session": "12cb0a4a64e1ff02d74fb08315290f8a", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/8a24513bf3d1491ba663801ce5ea9f96/d584444aa35e43138018f4c05e262ba5/?child=first", "accept_language": "en-US,en;q=0.9", "event": "{\"done\": true}", "time": "2021-08-13T16:45:52.730240+00:00", "event_type": "edx.done.toggled", "event_source": "browser", "page": "https://bla.com/courses/course-v1:bla+bla+1/courseware/8a24513bf3d1491ba663801ce5ea9f96/d584444aa35e43138018f4c05e262ba5/?child=first"}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:47 +0000] http://bla.com "POST /event HTTP/1.1" 200 7 "https://bla.com/courses/course-v1:bla+Dbg1011_WinDbg1+1/courseware/6178d296890a499c97b26eb6c3d0788f/29274cde69094eefa772f7e6c85febb7/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67" "98.223.208.142"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:48 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 154 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | 172.18.0.9 - - [13/Aug/2021:14:17:57 +0000] "PUT /api/v1/users/1870?request_id=d3787005-13fe-4db4-ab86-15b9645e87c6 HTTP/1.1" 200 47 0.0084
caddy_1          | {"level":"error","ts":1628350396.758072,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            |     raise NotFoundError('No transcript for `{lang}` language'.format(
lms-worker_1     | 2021-08-13 12:41:07,682 INFO 1056 [lms.djangoapps.certificates.queue] [user None] [ip None] queue.py:366 - Student 33 is whitelisted in 'course-v1:bla+bla+1'
lms-worker_1     | 2021-08-13 12:41:07,691 INFO 1056 [root] [user None] [ip None] queue.py:469 - certificate generated for user: user2 with generate_pdf status: False
mongodb_1        | 2021-08-11T02:03:16.580+0000 I NETWORK  [conn626] end connection 172.18.0.11:35348 (58 connections now open)
redis_1          | 1:M 12 Aug 2021 05:18:14.050 * Background saving terminated with success
lms_1            | [pid: 8|app: 0|req: 409816/1436115] 172.18.0.12 () {64 vars in 1998 bytes} [Fri Aug 13 16:45:52 2021] POST /event => generated 7 bytes in 46 msecs (HTTP/1.0 200) 7 headers in 360 bytes (1 switches on core 0)
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:48 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 155 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | I, [2021-08-13T14:17:57.276851 #28]  INFO -- : source=rack-timeout id=0a00a03fdcf7196386e0fd1f10c87a93 timeout=20000ms state=ready
caddy_1          | {"level":"error","ts":1628350396.7579436,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 172.18.0.3:443->88.230.228.188:58568: write: connection refused"}
cms_1            | xmodule.exceptions.NotFoundError: No transcript for `en` language
cms-worker_1     |     raise self.retry(kwargs=kwargs, exc=exc)
lms-worker_1     | 2021-08-13 12:41:07,694 INFO 1056 [edx.certificate] [user None] [ip None] api.py:229 - Queued Certificate Generation task for 33 : course-v1:bla+bla+1
mongodb_1        | 2021-08-11T02:03:51.848+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:47176 #629 (59 connections now open)
redis_1          | 1:M 12 Aug 2021 05:23:15.021 * 10 changes in 300 seconds. Saving...
lms_1            | 2021-08-13 16:45:52,781 INFO 22 [tracking] [user 634] [ip 123.123.123.3] logger.py:42 - {"name": "edx.grades.problem.submitted", "context": {"course_id": "course-v1:bla+bla+1", "course_user_tags": {}, "user_id": 634, "path": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4/handler/toggle_button", "org_id": "bla", "module": {"display_name": "Completion", "usage_key": "block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4"}}, "username": "neongen", "session": "12cb0a4a64e1ff02d74fb08315290f8a", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/8a24513bf3d1491ba663801ce5ea9f96/d584444aa35e43138018f4c05e262ba5/?child=first", "accept_language": "en-US,en;q=0.9", "event": {"user_id": "634", "course_id": "course-v1:bla+bla+1", "problem_id": "block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4", "event_transaction_id": "8cc96bc0-f481-41ec-b89f-80a7b073ffa0", "event_transaction_type": "edx.grades.problem.submitted", "weighted_earned": 1, "weighted_possible": 1}, "time": "2021-08-13T16:45:52.780803+00:00", "event_type": "edx.grades.problem.submitted", "event_source": "server", "page": null}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:48 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 156 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | I, [2021-08-13T14:17:57.287055 #28]  INFO -- : source=rack-timeout id=0a00a03fdcf7196386e0fd1f10c87a93 timeout=20000ms service=10ms state=completed
caddy_1          | {"level":"error","ts":1628350396.7579858,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"client disconnected"}
cms_1            | [pid: 15|app: 0|req: 882/3138] 172.18.0.12 () {56 vars in 2118 bytes} [Sun Jul 25 13:15:31 2021] GET /xblock/container/block-v1:bla+bla+1+type@vertical+block@65aea3af5134475988fb1699e02c06aa => generated 14574 bytes in 169 msecs (HTTP/1.0 200) 6 headers in 301 bytes (1 switches on core 0)
cms-worker_1     |   File "/openedx/venv/lib/python3.8/site-packages/celery/app/task.py", line 706, in retry
lms-worker_1     | 2021-08-13 12:41:07,694 INFO 1056 [tracking] [user None] [ip None] logger.py:42 - {"name": "edx.certificate.created", "context": {"org_id": "bla", "course_id": "course-v1:bla+bla+1"}, "username": "", "session": "", "ip": "", "agent": "", "host": "", "referer": "", "accept_language": "", "event": {"user_id": 33, "course_id": "course-v1:bla+bla+1", "certificate_id": "b0e68ff618d143feb964af3e8ddfa4fe", "enrollment_mode": "honor", "generation_mode": "batch", "certificate_url": "/certificates/b0e68ff618d143feb964af3e8ddfa4fe"}, "time": "2021-08-13T12:41:07.694457+00:00", "event_type": "edx.certificate.created", "event_source": "server", "page": null}
mongodb_1        | 2021-08-11T02:03:51.848+0000 I NETWORK  [conn629] received client metadata from 172.18.0.11:47176 conn629: { driver: { name: "PyMongo", version: "3.10.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-1029-aws" }, platform: "CPython 3.8.6.final.0" }
redis_1          | 1:M 12 Aug 2021 05:23:15.024 * Background saving started by pid 8042
lms_1            | 2021-08-13 16:45:52,792 INFO 22 [celery_utils.logged_task] [user 634] [ip 123.123.123.3] logged_task.py:25 - Task lms.djangoapps.grades.tasks.recalculate_subsection_grade_v3[ff1bb438-3316-46e1-a779-9a94ffa240dd] submitted with arguments None, {'user_id': 634, 'anonymous_user_id': None, 'course_id': 'course-v1:bla+bla+1', 'usage_id': 'block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4', 'only_if_higher': None, 'expected_modified_time': 1628873152, 'score_deleted': None, 'event_transaction_id': '8cc96bc0-f481-41ec-b89f-80a7b073ffa0', 'event_transaction_type': 'edx.grades.problem.submitted', 'score_db_table': 'csm', 'force_update_subsections': False}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:51 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 158 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | 172.18.0.9 - - [13/Aug/2021:14:17:57 +0000] "GET /api/v1/cfa1a38ae81df9cbc9204cc161bef8ecbcb60e39/threads?page=1&per_page=20&course_id=course-v1%3Abla%2Bbla%2B1&sort_key=activity&user_id=1870&context=course&request_id=b45f8200-6d86-4b5b-8231-3e40f9657680 HTTP/1.1" 200 772 0.0111
caddy_1          | {"level":"error","ts":1628350396.7579386,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"client disconnected"}
cms_1            | [pid: 7|app: 0|req: 876/3139] 172.18.0.12 () {56 vars in 2215 bytes} [Sun Jul 25 13:15:32 2021] GET /preview/xblock/block-v1:bla+bla+1+type@video+block@d39890c085c8497a892e4dc5bf4e50bb/handler/transcript/translation/en?videoId=3_yD_123123 => generated 0 bytes in 89 msecs (HTTP/1.0 404) 5 headers in 283 bytes (1 switches on core 0)
cms-worker_1     |     raise_with_context(exc)
lms-worker_1     | 2021-08-13 12:41:08,141 INFO 1056 [lms.djangoapps.grades.course_grade_factory] [user None] [ip None] course_grade_factory.py:223 - Grades: Update, Course: course_key: course-v1:bla+bla+1, version: 61166743f3bf03a202cfc03b, edited_on: 2021-08-13 12:36:19.705000+00:00, grading_policy: 07//7l6rp2+Iomia2ACBrZKoFkI=, User: 51, Course Grade: percent: 0.0, letter_grade: None, passed: False, persisted: False
mongodb_1        | 2021-08-11T02:03:51.990+0000 I NETWORK  [listener] connection accepted from 172.18.0.11:47178 #630 (60 connections now open)
redis_1          | 8042:C 12 Aug 2021 05:23:15.987 * DB saved on disk
lms_1            | 2021-08-13 16:45:52,793 INFO 22 [tracking] [user 634] [ip 123.123.123.3] logger.py:42 - {"name": "edx.done.toggled", "context": {"course_id": "course-v1:bla+bla+1", "course_user_tags": {}, "user_id": 634, "path": "/courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4/handler/toggle_button", "org_id": "bla", "module": {"display_name": "Completion", "usage_key": "block-v1:bla+bla+1+type@done+block@2b749c3c3bf3440b9899d314a1b72ff4"}, "asides": {}}, "username": "neongen", "session": "12cb0a4a64e1ff02d74fb08315290f8a", "ip": "123.123.123.3", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36", "host": "bla.com", "referer": "https://bla.com/courses/course-v1:bla+bla+1/courseware/8a24513bf3d1491ba663801ce5ea9f96/d584444aa35e43138018f4c05e262ba5/?child=first", "accept_language": "en-US,en;q=0.9", "event": {"done": true}, "time": "2021-08-13T16:45:52.793460+00:00", "event_type": "edx.done.toggled", "event_source": "server", "page": "x_module"}
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:17:52 +0000] http://bla.com "POST /courses/course-v1:bla+bla+1/xblock/block-v1:bla+bla+1+type@problem+block@8651da3797ea4ca28055b3e50af96f1d/handler/xmodule_handler/input_ajax HTTP/1.1" 200 160 "https://bla.com/courses/course-v1:bla+bla+1/courseware/ce583049085a47ce9256a26e2b5f5c1f/aff48fa2b31d478b9039ecf273344ddb/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" "41.217.82.227"
forum_1          | I, [2021-08-13T14:18:15.313495 #28]  INFO -- : source=rack-timeout id=d0631f27094b2a52f04b239caab75e30 timeout=20000ms state=ready
caddy_1          | {"level":"error","ts":1628350396.7578652,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
cms_1            | [pid: 15|app: 0|req: 883/3140] 172.18.0.12 () {62 vars in 1998 bytes} [Sun Jul 25 13:15:32 2021] POST /event => generated 0 bytes in 20 msecs (HTTP/1.0 204) 6 headers in 313 bytes (1 switches on core 0)
cms-worker_1     |   File "/openedx/edx-platform/lms/djangoapps/grades/tasks.py", line 102, in compute_grades_for_course_v2
5

Actually I just noticed all those log entries I cited were from before the timestamp of recreation of certs (I’m just leaving them up incase someone things they hold a clue.)

Here’s the first instances of “certificate” mentioned after the 12:41:22,791 timestamp where I regenerated things (LMK if you’d like to see additional logging around a particular timestamp):

lms-worker_1     | 2021-08-13 12:41:22,791 INFO 1056 [celery.app.trace] [user None] [ip None] trace.py:125 - Task lms.djangoapps.instructor_task.tasks.generate_certificates[2b641d4d-ff74-4c64-a092-3fd66b276aac] succeeded in 16.66959141427651s: {'action_name': 'certificates generated', 'attempted': 32, 'succeeded': 32, 'skipped': 0, 'failed': 0, 'total': 32, 'preassigned': 0, 'duration_ms': 16649, 'step': 'Generating Certificates'}
lms-worker_1     | 2021-08-13 12:59:47,147 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 280 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 12:59:47,150 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 280 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 12:59:59,725 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 297 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 12:59:59,727 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 297 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:00:07,650 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 307 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:00:07,653 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 307 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:00:33,158 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 335 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:00:33,161 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 335 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:01:57,649 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 54 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:01:58,909 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 475 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:01:58,911 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 475 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:02:25,677 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 34 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:02:27,749 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 516 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:02:27,752 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 516 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:03:56,070 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 547 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:03:56,072 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 547 : course-v1:bla+bla+1
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:50:21 +0000] http://bla.com "GET /static/images/certificates/audit.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/courses/course-v1:bla+bla+1/courseware/e1334266b4344792b3c03df870947355/3cf6205822cf48bcb3226ce9a399cf0c/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36" "123.123.123.1"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:50:21 +0000] http://bla.com "GET /static/images/certificates/verified.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/courses/course-v1:bla+bla+1/courseware/e1334266b4344792b3c03df870947355/3cf6205822cf48bcb3226ce9a399cf0c/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36" "123.123.123.1"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:50:21 +0000] http://bla.com "GET /static/images/certificates/professional.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/courses/course-v1:bla+bla+1/courseware/e1334266b4344792b3c03df870947355/3cf6205822cf48bcb3226ce9a399cf0c/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36" "123.123.123.1"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:16:50:21 +0000] http://bla.com "GET /static/images/certificates/honor.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/courses/course-v1:bla+bla+1/courseware/e1334266b4344792b3c03df870947355/3cf6205822cf48bcb3226ce9a399cf0c/?child=first" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36" "123.123.123.1"
lms-worker_1     | 2021-08-13 13:07:58,038 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 5 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:13,574 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 57 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:14,952 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 33 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:15,917 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 51 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:16,863 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 53 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:17,839 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 48 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:18,830 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 71 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:19,823 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 90 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:21,075 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 67 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:22,061 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 38 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:22,954 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 31 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:23,666 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 181 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:24,374 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 230 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:25,079 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 75 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:25,785 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 95 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:26,758 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 26 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:27,462 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 30 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:28,202 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 28 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:28,886 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 36 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:29,577 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 60 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:30,259 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 66 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:31,215 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 79 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:31,898 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 86 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:32,590 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 94 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:33,280 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 161 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:33,982 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 206 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:35,067 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 100 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:35,760 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 175 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:36,457 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 202 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:37,176 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:92 - Certificate marked not passing for 211 : course-v1:bla+bla+1 via failing grade: Course Grade: percent: 0.0, letter_grade: None, passed: False
lms-worker_1     | 2021-08-13 13:08:54,987 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 1261 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:08:54,989 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 1261 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:09:40,568 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 1353 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:09:40,571 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 1353 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:10:06,762 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 1386 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:10:06,764 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 1386 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:10:51,465 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 204 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 13:10:51,468 INFO 1056 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 204 : course-v1:bla+bla+1
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:25 +0000] http://studio.bla.com "GET //admin/certificates/generatedcertificate/ HTTP/1.1" 404 7982 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:25 +0000] http://studio.bla.com "GET /static/studio/bundles/commons.fbb780f7a5c85ecfe619.d74b88139ac4.js HTTP/1.1" 200 2012599 "https://studio.bla.com//admin/certificates/generatedcertificate/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:30 +0000] http://studio.bla.com "GET /admin/certificates/generatedcertificate/ HTTP/1.1" 404 7982 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:38 +0000] http://bla.com "GET /admin/certificates/generatedcertificate/ HTTP/1.1" 200 45351 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:38 +0000] http://bla.com "GET /admin/jsi18n/ HTTP/1.1" 200 3223 "https://bla.com/admin/certificates/generatedcertificate/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:54 +0000] http://bla.com "GET /admin/certificates/generatedcertificate/?q=user1 HTTP/1.1" 200 7808 "https://bla.com/admin/certificates/generatedcertificate/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:12:54 +0000] http://bla.com "GET /admin/jsi18n/ HTTP/1.1" 200 3223 "https://bla.com/admin/certificates/generatedcertificate/?q=user1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:18:30 +0000] http://bla.com "GET /certificates/b8532cae92e44a569584d9b19891cf43 HTTP/1.1" 404 9460 "https://www.linkedin.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
lms-worker_1     | 2021-08-13 14:00:19,947 INFO 1063 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 623 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 14:00:19,950 INFO 1063 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 623 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 14:03:28,182 INFO 1063 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:150 - Entered into Ungenerated Certificate task for 623 : course-v1:bla+bla+1
lms-worker_1     | 2021-08-13 14:03:28,183 INFO 1063 [lms.djangoapps.certificates.signals] [user None] [ip None] signals.py:178 - Certificate Generation task failed for 623 : course-v1:bla+bla+1
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:14 +0000] http://bla.com "GET /static/js/certificates/factories/certificate_invalidation_factory.e197f9758f9f.js?raw HTTP/1.1" 200 1160 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:14 +0000] http://bla.com "GET /static/js/certificates/factories/certificate_whitelist_factory.bdaec0c2298f.js?raw HTTP/1.1" 200 1954 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/collections/certificate_invalidation_collection.js HTTP/1.1" 200 575 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/models/certificate_exception.js HTTP/1.1" 200 1206 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/views/certificate_invalidation_view.js HTTP/1.1" 200 5876 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/collections/certificate_whitelist.js HTTP/1.1" 200 2236 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/views/certificate_whitelist.js HTTP/1.1" 200 4960 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/views/certificate_whitelist_editor.js HTTP/1.1" 200 5053 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/views/certificate_bulk_whitelist.js HTTP/1.1" 200 10444 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/models/certificate_invalidation.js HTTP/1.1" 200 1159 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:21:15 +0000] http://bla.com "GET /static/js/certificates/factories/certificate_whitelist_factory.js HTTP/1.1" 200 1954 "https://bla.com/courses/course-v1:bla+bla+1/instructor" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "123.123.123.2.adminIP"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:26:17 +0000] http://bla.com "GET /static/images/certificates/verified.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 OPR/77.0.4054.298" "123.123.123.3"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:26:18 +0000] http://bla.com "GET /static/images/certificates/professional.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 OPR/77.0.4054.298" "123.123.123.3"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:26:18 +0000] http://bla.com "GET /static/images/certificates/honor.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 OPR/77.0.4054.298" "123.123.123.3"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:26:18 +0000] http://bla.com "GET /static/images/certificates/audit.24fb9e64fdae.png HTTP/1.1" 200 13211 "https://bla.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 OPR/77.0.4054.298" "123.123.123.3"
nginx_1          | 172.18.0.3 - - [13/Aug/2021:17:28:15 +0000] http://bla.com "GET /static/js/courseware/certificates_api.41fbb2dd40c8.js HTTP/1.1" 200 752 "https://bla.com/courses/course-v1:bla+bla+1/progress" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36 OPR/77.0.4054.298" "123.123.123.3"

Basically this all seems to have something to do with me having set a required grade for certificates, and then the system not liking that I manually generated certs for beta test students who don’t have the required grade, and so then it kills their certs.

This seems like a likely bug in Open edX where the code which processes the COURSE_GRADE_NOW_FAILED signal (which is apparently what causes that “Certificate marked not passing”… error message), is not taking into account the certificate exceptions list of something? (cc @staubina since I see a post about reworking the code for certificates.)

6

Hiya.

Yep, the course certificate code previously didn’t respect the certificate exception list (aka certificate allowlist) when it received a failing grade signal. This was fixed in this PR: https://github.com/edx/edx-platform/pull/26356

Additional cert fixes and improvements have been made in Lilac and Maple. See https://openedx.atlassian.net/wiki/spaces/COMM/pages/2023915819/Lilac and https://openedx.atlassian.net/wiki/spaces/COMM/pages/2603816700/Maple

7

great to know! Is that fix in Koa though? Or do I need to upgrade to Lilac to get it? (I was planning on skipping Lilac and upgrading to Maple once it’s out)

8

That fix is not in Koa. You’d need to upgrade to Lilac and then enable the certificates_revamp.use_allowlist CourseWaffleFlag (as mentioned in the notes linked above).

Alternately, if you don’t want to upgrade and are comfortable writing code and redeploying your instance, you could reference the PR and manually add the fix yourself.

9

Well, it’s my understanding that with Tutor creating customizations yourself leads to forking edx-platform which leads to PITA maintenance difficulties. As I’m administering the site myself I doubt I’d have time for that. I guess I’ll see if I can upgrade my beta site to Lilac without everything blowing up and if that works move on to the production site. Thanks for fixing it though!

10

@Jimmy_Wu, the certificates disappear when users visit the Course Progress page. We have prepared a quick fix for the Koa release to patch our instances before the Lilac upgrade. You can find it here.

11

@crice I ended up just upgrading to Lilac to pick up the recent XSS fix too. I’m currently on Lilac.2 + Tutor 12.0.4. When I manually go to my progress page and request the certificate, it still breaks the URL for the previously manually-generated cert. It’s hard to be taken seriously as a platform when my students’ certificates on their linkedin pages keep breaking :frowning: (It’s too soon to tell yet whether things are still just auto-breaking after a time period…)

12

@Jimmy_Wu, have you enabled certificates_revamp.use_allowlist CourseWaffleFlag, as described above?

14

OK, that’s my bad! Too much time had passed, and I forgot that was required (I was thinking just the Lilac upgrade would fix the issue.) Adding certificates_revamp.use_allowlist to the waffle flags seems to fix it! Thanks!