The way this OEP is phrased leads me to believe that the choice of Kafka has already been made, unless benchmarking reveals major flaws. Kafka is a piece of software that is very complex to deploy and administer. Please keep in mind that thousand of system administrators around the world need to deploy Open edX; adding yet another heavy component on top of Open edX would only make their lives worse. I would strongly recommend to reconsider this option – or at the very least to abstract away the actual implementation such that the end user can choose an alternative. (my vote would go to Redis Streams)
2 Likes