tCRIL Funded Contribution - Redis Streams as Event Bus

Hello all!

tCRIL is pleased to announce a new funded contribution project to help advance Open edX platform. This project aims to investigate and provide a reference implementation of Redis Streams as a message bus for cross-application communication in Open edX deployments. We are seeking proposals from existing community members as well as qualified organizations looking for a first project to join the community. All proposals from providers interested in undertaking this work must be submitted by February 1, 2023 and tCRIL expects to make a provider selection by February 8, 2023.

At a high level this project hopes to establish Redis Streams as a second message bus, built on the patterns of the existing Kafka bus, and having a reference implementation at near-parity to the Kafka bus. We additionally expect to have documented any blocking issues to full adoption, as well as future work necessary to provide a fully functional, production ready product. We are pursuing this work to help drive message bus adoption across the ecosystem by offering a low cost, lightweight alternative to Kafka using technology that site operators are already familiar with.

We will be looking for proposals with a strong understanding of the technologies in play, their limitations, and the existing Open edX message bus use case(s) when evaluating bids. Price, timeline, and a proven ability to work with the Open edX code base will also be important factors in our decision.

Details on the project status, as well as a draft Statement of Work explaining the proposed project in more detail, are located in the FC-0008 Funded Contribution page on Confluence. The draft Statement of Work can (and will) be amended as conversations with the providers unfold but should be considered the most authoritative source of what we are looking for from these proposals.

Please send all proposals to funded-proposals@tcril.org and cc myself at bmesick@tcril.org. Feel free to contact me directly with any questions on Slack at @Brian Mesick or comment here.

We are looking forward to your questions and proposals!

13 Likes

Update to this contribution:

The draft statement of work linked on the Confluence page includes a reference to finishing this PR: https://github.com/openedx/openedx-events/pull/85

That PR has changed in scope to just include the producer side, and merged. The remaining work to implement an abstraction layer on the consumer side still needs to be done, and there is a work-in-progress branch here GitHub - openedx/openedx-events at timmc/load-consumer for working on the consumer side.

So the scope of work is somewhat smaller on that side of things, but there are likely to be numerous architectural discussions about the abstraction of the consumer API. Make sure your proposals include time for those discussions and PR revisions as we build a solution that will work for both.

Please see this conversation in Slack for a lot more information and context: Slack

1 Like

Aaaaaand the email address to send proposals to is actually funded-contributions@tcril.org. The one in the original post will bounce!

1 Like