dave
January 29, 2025, 5:53pm
1
There is a new DEPR proposal for the removal of the XQueue sevice:
opened 07:07PM - 25 Jan 25 UTC
depr
### Proposal Date
2025-01-29
### Target Ticket Acceptance Date
2025-02-12
##… # Earliest Open edX Named Release Without This Functionality
Ulmo
### Rationale
The XQueue service adds a lot of operational complexity for the capability that it provides. While it did originally do more (e.g. push grading), that functionality was removed because it was prone to operational issues. The current form of XQueue mostly serves as a holding area for submissions, waiting for xqueue-watcher to poll for them.
A larger deprecation was proposed in https://github.com/openedx/public-engineering/issues/214, but we concluded that the off-site grading functionality that XQueue represented was still valuable to preserve.
### Removal
The XQueue service will be discontinued and the [xqueue repo](https://github.com/openedx/xqueue) will be archived.
The [following settings](https://github.com/openedx/edx-platform/blob/a5b9b873906aadde6e73eabd79ef1611edef54f7/lms/envs/common.py#L1091-L1100) will be removed from edx-platform:
* `XQUEUE_INTERFACE`
* `XQUEUE_WAITTIME_BETWEEN_REQUESTS`
We will maintain support for [xqueue-watcher](https://github.com/openedx/xqueue-watcher) and ProblemBlock use of code graders.
When xqueue references are eventually completely removed from edx-platform (post-Teak), we will also execute https://github.com/openedx/public-engineering/issues/192 (MATLAB removal). The two systems are tightly coupled enough where it makes sense to remove them at the same time.
### Replacement
Instead of having a separate XQueue service, the [edx-submissions](https://github.com/openedx/edx-submissions) repo will be expanded to cover the same use case while running in-process in the LMS. The LMS will also re-implement the REST API that xqueue-watcher interacts with.
The LonCapaSystem service [XQueueService](https://github.com/openedx/edx-platform/blob/a5b9b873906aadde6e73eabd79ef1611edef54f7/xmodule/capa/xqueue_interface.py#L159) will be replaced with a generic XBlock runtime service that is not ProblemBlock-specific.
### Deprecation
The LMS will emit warning log messages in Teak whenever the old xqueue-callback URL for ProblemBlocks is hit.
### Migration
* No content changes will be required for authored problems. Existing content is intended to work exactly as it always has.
* No code changes will be required for `xqueue-watcher`.
* Configuration changes _will_ be required for `xqueue-watcher`, to point it at the LMS instead of at the XQueue service.
In the Teak release, all code grader submissions will be written to the new backend in edx-submissions, and will be available for xqueue-watcher to pull directly from the LMS. If `XQUEUE_INTERFACE` is also configured for edx-platform, the LMS will additionally post the request to the XQueue service. If the LMS gets a response from the XQueue service, it will delete the queued submission in edx-submissions.
This should allow us a smooth transition path where operators could make their transition by deploying updated configuration for their xqueue-watcher. Operators that are very downtime sensitive could run multiple xqueue-watcher instances concurrently during the transition, falling back to the old behavior if necessary.
#### Open question:
1. Should duplicate grading attempts for the same submission be "first one wins" or "last one wins"?
2. Do we need to preserve an alias for the old Loncapa XQueueService? This would only need to be done if people have extended their instance ProblemBlocks with custom input/response-types that use that service... which seems unlikely?
Support for posting to the XQueue service would be removed after 2025-08-12, and all code and tests in edx-platform related to xqueue will be removed at that point.
### Additional Info
We ultimately want to iterate on this and create a much more flexible off-platform grading capability that can:
1. Make better use of other communications mechanisms, like the message queue backend that openedx-events can utilize.
2. Allows for multiple interfaces, and so accepts things like LTI's grade passbacks, and possibly bulk grading.
3. Understands team-related grading, like ORA does.
4. Remove some of the contortions we make to allow LTI blocks to authenticate non-user requests for grade passback purposes.
But the first step is to preserve the existing functionality while reducing operational complexity of running this system, which this DEPR represents.
### Task List
_No response_
Unlike the prior proposal , this new one only removes the XQueue service, while keeping backwards compatibility with xqueue-watcher. Existing ProblemBlocks that make use of XQueue for code grading will continue to function. This is mostly about folding in functionality from XQueue to run in-process in the LMS, with an eye towards refactoring it for more generalized use for off-platform grading in the future.
The new system and XQueue will be supported concurrently in Teak, with support for the XQueue service dropped entirely sometime after 2025-08-12, i.e. Ulmo will not have XQueue.
Target acceptance date is 2025-02-12. Please see the DEPR ticket for details.
dave
February 14, 2025, 3:38pm
2
This DEPR has been marked as “Accepted”.
1 Like