ConnectionForced: Too many heartbeats missed

Site Operators Site Operations Help
1

Hi,
I am getting ConnectionForced: Too many heartbeats missed errors from time to time. Any idea how to troubleshoot and get the root cause?

I’m running Ironwood.

Here is the full traceback:

Sep 15 17:05:00 campus [service_variant=lms][celery.worker.consumer][env:sandbox] WARNING [campus  2346] [consumer.py:291] - consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 280, in start
blueprint.start(self)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
step.start(parent)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 884, in start
c.loop(*c.loop_args())
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
next(loop)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/async/hub.py", line 281, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/async/hub.py", line 140, in fire_timers
entry()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/async/timer.py", line 64, in __call__
return self.fun(*self.args, **self.kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/async/timer.py", line 132, in _reschedules
return fun(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/connection.py", line 277, in heartbeat_check
return self.transport.heartbeat_check(self.connection, rate=rate)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 135, in heartbeat_check
return connection.heartbeat_tick(rate=rate)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/amqp/connection.py", line 915, in heartbeat_tick
raise ConnectionForced('Too many heartbeats missed')
ConnectionForced: Too many heartbeats missed