Frequently comment service time out error

Site Operators Site Operations Help
1

this error happens randomly multiple times a day causing bad experience for q&a activities.

I’m working on upgrading but by now I have a dogwood instance and I want to at least extend this timeout duration. I have made these changes but amazingly it has no effect on the timeout duration.
changes I’ve made:

lms/lib/comment_client/utils.py
- timeout=5
+ timeout=40

forum/cs_comments_service/config/unicorn.rb
- timeout 25
+ timeout 40

forum/forum_env
- export WORKER_PROCESSES="4"
+ export WORKER_PROCESSES="6"

Any help would be appreciated

2

@mahyard rather than adjusting the timeout, have you looked into why requests might be responding slowly?

What kind of system are you hosting on, are resources like CPU or memory being exhausted that might be causing the comments service to take excessive time to respond?

Is the mongo database responding slowly and causing the forum service to take too long to respond?

3

Thank you @joshm for reopening this post.
actually we have a single open stack which hosts all of our services. as I mentioned above currently we are on dogwood but in a couple weeks we will switch to hawthorn. our machine has 4 vCore processors and 20GB Memory which I think should be enough for 45k users who send 1 request per second. unfortunately I’m not familiar with ruby and I can’t trace to find and solve the issue so I thought may be somebody could help me to buy a few more time until we get ready to migrate. thanks in advanced.

4

We ran into the same issue.

We crafted a command to be able to predictably reproduce the issue using ab (Apache Benchmark). It looked like this:

ab -H "X-Edx-Api-Key: our_secret_key" -n10 -c10 "http://localhost:18080/api/v1/users/7?complete=True&request_id=id-copypasted-from-logs"

After restarting the forum daemon we observed the problem was gone by running the above command again. So for now we set the forum to restart each night.

Hope this helps.

1 Like
5

thank you very much Silvio for your kindness. Actually we have upgraded to hawthorn now. so I can’t test what you said. But I appreciate your answer and I hope this would help somebody else. @silviot

1 Like