Caddy 502 Errors During Load Testing

Hello Team!

I hope I have tagged the correct group.

We are in the process of upgrading from Koa to Nutmeg. We are doing some load testing on our site and when we start ramping up the concurrent users we start to see some 502 errors that seem to be coming from Caddy (errors below).

We were wondering if anyone else has had similar experiences and if they had to make any configuration changes in the Caddy file?

Koa was on NGINX and we had more experience tuning there. We have looked a little bit on the Caddy site but didn’t see any configurations that seemed to align with our errors.

We ran a separate test where we replaced Caddy with NGINX and our concurrent tests ran without any errors. Our goal is to use the infrastructure that aligns with the community so that we can upgrade to the next versions without a lot of custom changes so we would like to stay with Caddy if we can get these errors resolved.

Thanks in advance for your help!

502 Errors:

{ "level": "error",
"ts": 1680726239.7636938,
"logger": "http.log.access.log0",
"msg": "handled request",
"request": { "remote_addr": "xxx.xx.x.xxx:29732",
"proto": "HTTP/1.1",
"method": "POST",
"host": "xx.xx.xxx.edu",
"uri": "/courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@500c03fd346047089efae3f6daf6b31e/handler/publish_completion" },
"user_id": "",
"duration": 0.000553214,
"size": 0,
"status": 502 }


{ "level": "error",
"ts": 1680726239.7636938,
"logger": "http.log.access.log0",
"msg": "handled request",
"request": { "remote_addr": "xxx.xx.x.xxx:29732",
"proto": "HTTP/1.1",
"method": "POST",
"host": "xx.xx.xxx.edu",
"uri": "/courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@500c03fd346047089efae3f6daf6b31e/handler/publish_completion",
"headers": { "X-Forwarded-Port": [ "443" ],
"Sec-Fetch-Mode": [ "cors" ],
"Referer": [ "https://xx.xx.xxx.edu/courses/course-v1:UNIx+C05a+2020_T1/courseware/73e1fe9a452b4560863c2848756e4d9f/75efae50589f4ccca67bb268107cf11d/3?activate_block_id=block-v1:UNIx+C05a+2020_T1+type@vertical+block@a4e0dbf674604b71beb48f42ac52c642" ],
"Accept-Language": [ "en-US,en;q=0.5" ],
"Accept": [ "application/json,
text/javascript,
*/*; q=0.01" ],
"Origin": [ "https://xx.xx.xxx.edu" ],
"Sec-Fetch-Dest": [ "empty" ],
"X-Csrftoken": [ "xx.xx.xxx" ],
"X-Forwarded-For": [ "xx.xx.xxx.xxx" ],
"X-Amzn-Trace-Id": [ "Root=1-xxx" ],
"X-Requested-With": [ "XMLHttpRequest" ],
"Content-Type": [ "application/x-www-form-urlencoded; charset=UTF-8" ],
"Accept-Encoding": [ "gzip,
deflate,
br" ],
"User-Agent": [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0" ],
"Cookie": [ "experiments_is_enterprise=false; edxloggedin=true; edx-user-info=\"{\\\"version\\\": 1\\\\054 \\\"username\\\": ...; csrftoken=xx.xx.xxx; openedx-language-preference=en" ],
"X-Forwarded-Proto": [ "https" ],
"Content-Length": [ "16" ],
"Sec-Fetch-Site": [ "same-origin" ] } },
"common_log": "xxx.xx.x.xxx - - [05/Apr/2023:20:23:59 +0000] \"POST /courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@500c03fd346047089efae3f6daf6b31e/handler/publish_completion HTTP/1.1\" 502 0",
"user_id": "",
"duration": 0.000553214,
"size": 0,
"status": 502,
"resp_headers": { "Server": [ "Caddy" ] } }


{ "level": "error",
"ts": 1680726239.7636597,
"logger": "http.log.error.log0",
"msg": "EOF",
"request": { "remote_addr": "xxx.xx.x.xxx:29732",
"proto": "HTTP/1.1",
"method": "POST",
"host": "xx.xx.xxx.edu",
"uri": "/courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@500c03fd346047089efae3f6daf6b31e/handler/publish_completion",
"headers": { "Origin": [ "https://xx.xx.xxx.edu" ],
"Sec-Fetch-Dest": [ "empty" ],
"X-Forwarded-For": [ "xx.xx.xxx.xxx" ],
"X-Amzn-Trace-Id": [ "Root=1-xxxx" ],
"X-Requested-With": [ "XMLHttpRequest" ],
"Content-Type": [ "application/x-www-form-urlencoded; charset=UTF-8" ],
"Accept-Encoding": [ "gzip,
deflate,
br" ],
"User-Agent": [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0" ],
"X-Csrftoken": [ "xx.xx.xxx" ],
"Cookie": [ "experiments_is_enterprise=false; edxloggedin=true; edx-user-info=\"... openedx-language-preference=en" ],
"X-Forwarded-Proto": [ "https" ],
"Content-Length": [ "16" ],
"Sec-Fetch-Site": [ "same-origin" ],
"X-Forwarded-Port": [ "443" ],
"Sec-Fetch-Mode": [ "cors" ],
"Referer": [ "https://xx.xx.xxx.edu/courses/course-v1:UNIx+C05a+2020_T1/courseware/73e1fe9a452b4560863c2848756e4d9f/75efae50589f4ccca67bb268107cf11d/3?activate_block_id=block-v1:UNIx+C05a+2020_T1+type@vertical+block@a4e0dbf674604b71beb48f42ac52c642" ],
"Accept-Language": [ "en-US,en;q=0.5" ],
"Accept": [ "application/json,
text/javascript,
*/*; q=0.01" ] } },
"duration": 0.000553214,
"status": 502,
"err_id": "5nn18jqgp",
"err_trace": "reverseproxy.statusError (reverseproxy.go:886)" }

{ "level": "error",
"ts": 1680724782.2203991,
"logger": "http.log.access.log0",
"msg": "handled request",
"request": { "remote_addr": "xxx.xx.x.xx:17494",
"proto": "HTTP/1.1",
"method": "POST",
"host": "xx.xx.xxx.edu",
"uri": "/courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@7d1d5bc37cb641499f293e13ca7b043d/handler/publish_completion",
"headers": { "X-Forwarded-For": [ "xx.xx.xxx.xxx" ],
"Sec-Fetch-Dest": [ "empty" ],
"X-Csrftoken": [ "xx.xx.xxx.xxx" ],
"Cookie": [ "experiments_is_enterprise=false; edxloggedin=true; edx-user-info=\"... openedx-language-preference=en" ],
"X-Forwarded-Port": [ "443" ],
"Content-Length": [ "16" ],
"Content-Type": [ "application/x-www-form-urlencoded; charset=UTF-8" ],
"User-Agent": [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0" ],
"Origin": [ "https://xx.xx.xxx.edu" ],
"X-Requested-With": [ "XMLHttpRequest" ],
"X-Amzn-Trace-Id": [ "Root=1-xxx" ],
"Sec-Fetch-Mode": [ "cors" ],
"Sec-Fetch-Site": [ "same-origin" ],
"Accept-Language": [ "en-US,en;q=0.5" ],
"Accept-Encoding": [ "gzip,
deflate,
br" ],
"X-Forwarded-Proto": [ "https" ],
"Referer": [ "https://xx.xx.xxx.edu/courses/course-v1:UNIx+C05a+2020_T1/courseware/e4e2597a0c72435e92d5172c115ca07e/54985ac9631d4ec384adad8ba8aafe9d/3?activate_block_id=block-v1:UNIx+C05a+2020_T1+type@vertical+block@3e9d29b067464afab8e2b18ea3f6e9cb" ],
"Accept": [ "application/json,
text/javascript,
*/*; q=0.01" ] } },
"common_log": "xxx.xx.x.xx - - [05/Apr/2023:19:59:42 +0000] \"POST /courses/course-v1:UNIx+C05a+2020_T1/xblock/block-v1:UNIx+C05a+2020_T1+type@html+block@7d1d5bc37cb641499f293e13ca7b043d/handler/publish_completion HTTP/1.1\" 502 0",
"user_id": "",
"duration": 0.000865048,
"size": 0,
"status": 502,
"resp_headers": { "Server": [ "Caddy" ] } }