OEP-52: Event Bus Technology draft available

Announcements Architecture
1

OEP-52 discusses the need for an event bus technology in the Open edX platform, as well as the (provisional) technology choice. This OEP does not yet have an arbiter or deadline. However, we are actively trying out the technology options, so we wanted to share this for discussion sooner rather than later.

See OEP-52: Event Bus Technology.

1 Like
2

The way this OEP is phrased leads me to believe that the choice of Kafka has already been made, unless benchmarking reveals major flaws. Kafka is a piece of software that is very complex to deploy and administer. Please keep in mind that thousand of system administrators around the world need to deploy Open edX; adding yet another heavy component on top of Open edX would only make their lives worse. I would strongly recommend to reconsider this option – or at the very least to abstract away the actual implementation such that the end user can choose an alternative. (my vote would go to Redis Streams)

2 Likes
3

I think it definitely makes sense to abstract this layer out, regardless of what server we end up with. @regis: I’ve never used Redis Streams in production–are there any major differences in capability/worldview that would make it challenging to make an abstraction that covers both? Ordering guarantees, scaling up and down, etc.? Are there features we should avoid leveraging in order to make compatibility easier?

1 Like
4

Thanks for your thoughts. I will ultimately address everything, but not sure if we should consolidate feedback on the OEP PR with a new thread for each of these topics?

What I will address is your initial concern. The point of the OEP process here is exactly to illicit this kind of feedback. That doesn’t mean we’ll reach consensus before deciding, but it does mean an arbiter should agree that all points have been addressed.

Also, I think that we should discuss all of these points further (on the OEP), and I don’t want Dave’s feedback to make it seem like an abstraction layer is a done deal, because there are also downsides to an abstraction layer, and each of these topics will need a more complete discussion.

Thank you.

1 Like