Getting rate-limited request error on devstack LMS

Development
,
1

I’m getting the following error when trying to login to my devstack lms at http://localhost:18000/login page.

Too Many Requests
Your request has been rate-limited. Please try again later.

Any idea why I’m receiving this error? I see this new security announcement Security: Patch for rate-limiting in edx-platform concerning rate-limiting being added, however, it doesn’t tell me how to disable it.

Also, when I try to login to http://localhost:18000/admin (Django Admin) page I get the following error.

RuntimeError: Unable to create a new session key. It is likely that the cache is unavailable.
2022-05-16 15:31:14,447 ERROR 169 [django.request] [user None] [ip None] log.py:224 - Internal Server Error: /admin/login/
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/user_authn/views/login.py", line 641, in redirect_to_lms_login
    return admin.site.login(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/admin/sites.py", line 414, in login
    return LoginView.as_view(**defaults)(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/decorators/debug.py", line 89, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/auth/views.py", line 63, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/auth/views.py", line 92, in form_valid
    auth_login(self.request, form.get_user())
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 111, in login
    request.session.cycle_key()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 344, in cycle_key
    self.create()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/contrib/sessions/backends/cache.py", line 50, in create
    raise RuntimeError(
RuntimeError: Unable to create a new session key. It is likely that the cache is unavailable.
[16/May/2022 15:31:14] "POST /admin/login/?next=/admin/ HTTP/1.1" 500 403411

cc: @Tim_McCormack

2

Hmm… even if devstack were misconfigured and using the wrong IP, if there’s only one user, it shouldn’t matter. Misconfigurations around IP address determination should only produce the following two situations:

  • A large number of users are put into the same rate-limiting bucket, incorrectly
  • A single user can pretend to be coming from multiple IP addresses

Neither of those should be relevant to devstack.

(For what it’s worth, the relevant configuration is here: edx-platform/devstack.py at open-release/maple.master · openedx/edx-platform · GitHub as well as any NUM_PROXIES override, if there is one.)

RuntimeError: Unable to create a new session key. It is likely that the cache is unavailable. seems relevant, though. I wonder if a failure to talk to Redis/memcache/whatever is getting reported as a rate-limiting failure because the rate-limiter is failing to store information in the cache.

3

@Tim_McCormack I recently added a cache table and additional configuration for our Qualtrics XBlock to use. Could that have anything to do with it? I’ve commented this additional Qualtrics configuration and restarted devstack but no luck getting past that RuntimeError: Unable to create a new session key. It is likely that the cache is unavailable. error. I’ll debug further and let you know what I find.

# This creates a table at edxapp.qualtrics_backends_api_tokens
./manage.py lms createcachetable --settings=devstack_docker

# edx-platform/lms/envs/common.py
#################### Qualtrics Settings #######################
QUALTRICS_API_BASE_URL = None
QUALTRICS_BACKEND = 'qualtrics.backends.qualtrics.qualtricsBackend'
QUALTRICS_API_VERSION = "v3"
QUALTRICS_API_TOKEN_EXPIRATION = 3599  # 1 hr
QUALTRICS_API_TOKEN_CACHE = 'qualtrics_api_token_cache'

# edx-platform/lms/envs/devstack.py
 CACHES[QUALTRICS_API_TOKEN_CACHE] = {
     'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
     'LOCATION': 'qualtrics_backends_api_tokens'
}
4

@Tim_McCormack I do see this in my devstack memcached container logs.

> Executing task: docker logs --tail 1000 -f 6cf51b76d0b81bfd73fe82ba87367cf549ab6c4d86c041f5a3b7f8d13c3b722a <

Signal handled: Terminated.
Signal handled: Terminated.
Signal handled: Terminated.
Signal handled: Terminated.
Signal handled: Terminated.
5

Yeah, sounds like a memcache issue. Unfortunately, I have very little experience with that subsystem.

6

@Tim_McCormack I figured it out. I was adding a configuration override in my edx-platform/lms/envs/private.py file for a CACHES setting but was importing as from .common import CACHES to bring in that dictionary value to override. That .common import includes 'LOCATION': ['localhost:11211'], which localhost for memcached container cannot be network resolved from the LMS container but the memcached container has rather be found on the docker network through the alias edx.devstack.memcached.

# edx-platform/lms/envs/common.py

############################ Cache Configuration ###############################

CACHES = {
    ...
    'default': {
        'VERSION': '1',
        'KEY_FUNCTION': 'common.djangoapps.util.memcache.safe_key',
        'LOCATION': ['localhost:11211'],
        'KEY_PREFIX': 'default',
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
    },
}

To fix my issue I imported using from .production import CACHES since the settings using in the environment /edx/etc/lms.yml are being used instead which reflects the correct memcached alias on the docker network.

# /edx/etc/lms.yml

CACHES:
    ...
    default:
        BACKEND: django.core.cache.backends.memcached.MemcachedCache
        KEY_FUNCTION: common.djangoapps.util.memcache.safe_key
        KEY_PREFIX: default
        LOCATION:
        - edx.devstack.memcached:11211
        VERSION: '1'
1 Like