I’m running Nutmeg and my VM on AWS became unresponsive over ssh on Sunday. I was forced to reboot it. I checked the filesystem and it wasn’t a problem that it had no space remaining or anything like that.
Since I rebooted, I am no longer able to run grade reports (they all just say “QUEUING” and never complete), I can’t import or export a class (it is permanently stuck on “Exporting” for exports even when I leave the page and come back, and “Unpacking” for imports), and when I test as a student account, I can’t submit answers to multiple choice problems, and completion xblocks that are marked done are persisted as marked done, but they’re not captured as done according to the course Progress page (completion xblocks are normally treated like 1-point problems.)
These symptoms tend to make me think perhaps there’s something broken about the mysql database (but not the mongodb where supposedly I read in the past completion xblocks’ information is stored)?
What is the best way to debug if there is a database error going on? (I haven’t been able to find anything interesting in the typical tutor local logs --tail=1000 output.)
Thus far I have tried the typical rebuilding of the docker container and restarting via tutor local quickstart but the error has persisted.
I also note that there is one error during “tutor images build openedx”
Step 22/103 : RUN curl -fsSL https://github.com/open-craft/edx-platform/commit/3d54f284f82b61e693ad652d8d6e46a226fcb36d.patch | git am
---> Running in f0ae43c43caf
curl: (22) The requested URL returned error: 404
Logs grepped for “error”, with nothing obvious showing up to me:
tutor local logs --tail=1000 | grep -i error
cms-worker_1 | [2023-02-27 18:33:19,383: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
cms-worker_1 | [2023-02-27 18:33:21,398: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
cms-worker_1 | [2023-02-27 18:33:25,419: ERROR/MainProcess] consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
lms-worker_1 | 2023-02-27 18:33:19,937 ERROR 1 [celery.worker.consumer.consumer] [user None] [ip None] consumer.py:451 - consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
lms-worker_1 | 2023-02-27 18:33:21,951 ERROR 1 [celery.worker.consumer.consumer] [user None] [ip None] consumer.py:451 - consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
lms-worker_1 | 2023-02-27 18:33:25,959 ERROR 1 [celery.worker.consumer.consumer] [user None] [ip None] consumer.py:451 - consumer: Cannot connect to redis://redis:6379/0: Redis is loading the dataset in memory.
caddy_1 | {"level":"error","ts":1677522807.8127453,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"1.1.1.1:58431","proto":"HTTP/2.0","method":"POST","host":"my.site.com","uri":"/courses/course-v1:MySite+Class1+V1/instructor/api/list_instructor_tasks","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":33.453857414,"size":0,"status":403}
caddy_1 | {"level":"error","ts":1677522834.8444057,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"1.1.1.1:58431","proto":"HTTP/2.0","method":"POST","host":"my.site.com","uri":"/courses/course-v1:MySite+Class1+V1/instructor/api/list_instructor_tasks","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.594305346,"size":0,"status":403}
caddy_1 | {"level":"error","ts":1677522894.3375554,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"1.1.1.1:58431","proto":"HTTP/2.0","method":"POST","host":"my.site.com","uri":"/courses/course-v1:MySite+Class1+V1/instructor/api/list_instructor_tasks","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.088728878,"size":0,"status":403}
caddy_1 | {"level":"error","ts":1677522902.3015223,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"1.1.1.1:58431","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/enterprise/api/v1/enterprise-learner/?username=User1","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.250758609,"size":9504,"status":404}
caddy_1 | {"level":"error","ts":1677522954.4726045,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"1.1.1.1:58431","proto":"HTTP/2.0","method":"POST","host":"my.site.com","uri":"/courses/course-v1:MySite+Class1+V1/instructor/api/list_instructor_tasks","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.219338132,"size":1019,"status":403}
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-26T14:24:14,818Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-10523025246243657187, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
caddy_1 | {"level":"error","ts":1677523358.741058,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"3.3.3.3:56224","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/static/xblock/resources/done/public/check-full.png","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.235543799,"size":9510,"status":404}
caddy_1 | {"level":"error","ts":1677523363.6825814,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/api/user_tours/v1/User3","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.080648439,"size":0,"status":404}
caddy_1 | {"level":"error","ts":1677523366.4745123,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/static/xblock/resources/done/public/check-empty.png","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.147255096,"size":9508,"status":404}
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-26T14:51:20,285Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-17380630067652297396, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
caddy_1 | {"level":"error","ts":1677523369.1003125,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/apps.my.site.com/src/course-home/outline-tab/LmsHtmlFragment.css","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.160525264,"size":9508,"status":404}
caddy_1 | {"level":"error","ts":1677523369.2665875,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/api/edx_proctoring/v1/user_onboarding/status?is_learning_mfe=true&course_id=course-v1%3AMySite%2BClass1%2BV1&username=User3","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.09508568,"size":67,"status":404}
caddy_1 | {"level":"error","ts":1677523369.4550166,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/apps.my.site.com/src/course-home/outline-tab/LmsHtmlFragment.css","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.218113326,"size":9508,"status":404}
caddy_1 | {"level":"error","ts":1677523369.5173433,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/api/user_tours/v1/User3","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.105177614,"size":0,"status":404}
caddy_1 | {"level":"error","ts":1677523369.7331588,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50847","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/apps.my.site.com/src/course-home/outline-tab/LmsHtmlFragment.css","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.142734102,"size":9508,"status":404}
caddy_1 | {"level":"error","ts":1677523370.826845,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
caddy_1 | {"level":"error","ts":1677523395.9024441,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:50882","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/apps.my.site.com/src/course-home/outline-tab/LmsHtmlFragment.css","tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.26288684,"size":8295,"status":404}
caddy_1 | {"level":"error","ts":1677523397.7204595,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"3.3.3.3:56224","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/api/certificates/v0/certificates/User2/","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.079329645,"size":63,"status":403}
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-26T17:00:35,833Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-14550384431915640575, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
caddy_1 | {"level":"error","ts":1677523509.6227825,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"3.3.3.3:56224","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/api/certificates/v0/certificates/User2/","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.148598795,"size":63,"status":403}
forum_1 | /app/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:205:in `pid=': Already running on PID:14 (or pid=/tmp/forum_unicorn.pid is stale) (ArgumentError)
forum_1 | /app/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:205:in `pid=': Already running on PID:14 (or pid=/tmp/forum_unicorn.pid is stale) (ArgumentError)
caddy_1 | {"level":"error","ts":1677523589.4326928,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"3.3.3.3:56224","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/static/xblock/resources/done/public/check-empty.png","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.160680944,"size":9510,"status":404}
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-26T17:10:26,144Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-2847894554714149840, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
forum_1 | /app/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:205:in `pid=': Already running on PID:14 (or pid=/tmp/forum_unicorn.pid is stale) (ArgumentError)
forum_1 | /app/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:205:in `pid=': Already running on PID:14 (or pid=/tmp/forum_unicorn.pid is stale) (ArgumentError)
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-26T18:11:56,233Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-8115212296896162819, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
caddy_1 | {"level":"error","ts":1677523803.8927326,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:51168","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/static/xblock/resources/done/public/check-empty.png","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.153876993,"size":9508,"status":404}
elasticsearch_1 | {"type": "server", "timestamp": "2023-02-27T18:32:51,680Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "e9c9df28b1e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-12070135537405910665, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
caddy_1 | {"level":"error","ts":1677524030.795671,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"2.2.2.2:51168","proto":"HTTP/2.0","method":"GET","host":"my.site.com","uri":"/static/xblock/resources/done/public/check-full.png","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"my.site.com"}},"user_id":"","duration":0.232267946,"size":9508,"status":404}
One thing I noticed in the logs around hits for “grade_course” on the failing server is this:
2023-02-27 21:35:48,938 INFO 7 [openedx.core.djangoapps.cors_csrf.helpers] [user 3] [ip 1.2.3.4] helpers.py:64 - Origin 'https://my.site.com' was not in `CORS_ORIGIN_WHITELIST`; full referer was 'https://my.site.com/courses/course-v1:MyOrg+MyClass+V1/instructor' and requested host was 'my.site.com'; CORS_ORIGIN_ALLOW_ALL=False
On a working server, I don’t see that. Could that have anything to do with it? (I don’t see anything set under the working site’s admin/cors_csrf/ page though FWIW, and I’ve never set that anywhere AFAIK before.
So no, I don’t think I’d be able to find how to check things. I’m just offering to run any commands you request on a system in a known-broken state and provide the log output. If you’re not interested, that’s fine, but LMK so I can delete the Amazon snapshots and save some money.