There has been much interest and discussion around alternative options for the existing discussion forum in Open edX. One of the options for such an alternative forum/discussion platform has been Discourse. Discourse itself doesn’t need much introduction here, it is the discussion platform that powers this site itself.
To that end we have investigated an approach to enabling integration between Discourse and Open edX. A detailed breakdown of that approach can be seen in this discovery document.
A rough overview of the approach is as follows:
Create a plugin for edx-platform that integrates with the extensive Discourse API.
The plugin can create per-course and per-cohort categories and set up proper user account permissions.
The plugin can also provide a course tab for embedding a category of discussions for the course itself.
Create a plugin for Discourse that allows embedding a topic or category. Discourse already support embedding a topic, but that feature has major limitation. The current embed is read-only, but we can start from there.
Create an XBlock that uses the above to embed a topic or category in a unit.
The Discourse UI itself will be available as well so that moderators can take advantage of the rich moderation tools provided by Discourse. It can also be opened up to students to allow them to have meta, or cross-course discussions.
Now I feel bad for not keeping up with edxchange in a while…
This sounds great, but same boat of not being able to access the doc. Perhaps something for Confluence?
I think Discourse has a lot of issues that still prevents it from being as good a course discussion solution than it could be, but similarly it’s the best of a lot of bad options out there. Did some discovery into discussion replacements a few years back that basically resulted in “I don’t really like the UX of Discourse much, but every other solution is appalling”. So I’m mostly in favour of this as it’s still better than default discussions, but I wish there was a better solution.
I think something worth considering with this proposal is how it affects the Teams feature. Discourse would need to somehow be a part of both, or you’re going to end up with two different discussion experiences on the same course if you use Teams.
This has been my finding as well. I don’t really dislike the Discourse UX, and I’m not sure it’s possible to have a UX that’s universally liked, but in general I’ve run into more people who like it than not. Of course that is just anecdotal
I’m very much in favor of this overall. As you mention in the document, just switching to Discord solves a lot of issues we’ve been struggling with for more than six years. I’m sure it will raise others, but it looks like a substantial gain right off the bat.
I’m excited that you folks are investigating this, and I agree that Discourse is the best thing out there at the moment. I would also frankly be overjoyed to retire our forums in favor of integration with a more mature, actively developed alternative. The existing forums are such an oddball in our stack.
One of the biggest sticking points we’ve had in the past when evaluating other solutions is the granularity of the forum space and admin/moderation. Most forums simply aren’t meant to be used in a way that has ten thousand different isolated spaces with different sets of moderators in each. The design doc has a proposal for mapping courses and individual inline discussion spaces into categories, and I’m guessing we’d use something like this to allow for per-category moderators. But I’m not clear on how Discourse behaves when you literally have a million categories (a two year old thread implies that we’d want to cap at around a thousand), or how operationally complex it would become if we start partitioning across many Discourse instances.
To be clear, I’m not trying to say that Discourse integration as a whole won’t work. My concern is more that modeling our use of Discourse to have precise 1:1 analogs with our existing forums system might not scale. But I wonder if we could take a couple of steps back and think of a way to integrate the two in a way that’s more natural for Discourse, even if it means we lose a few existing features or have to do more work on the integration side.
For instance, we could drop the per-unit categories and make the integration layer keep track of the associations between content Units and Discourse threads (maybe even make that M:M). Or we could step even further back and see if it makes sense to share a common forum space across multiple courses in a significantly different interaction model than what currently exists today.
I am quite ignorant of Discourse’s data model and operational behavior, so please forgive me (and correct me!) if I’m missing something obvious.
This was one of the reasons why I structured the categories the way I did, to make it easier to manage and track permissions. Each Course has its own category, and at this level we can directly give the broadest permissions to course staff/admins/moderators, then we categorise by cohort and can use group permissions to handle that. i.e. create a course-slug-cohort-slug group and give it permissions in that category.
I don’t want to start optimising too early, but for users at least the idea was to not create an account until the user actually wanted to comment. I don’t think the same can work for categories and topics.
There are definitely Open edX-specific requirements that make the integration more complex, but there are also optimisations that are possible in this use case that aren’t for others. For instance the course/cohort/… hierarchy we’ve set up is meaningless to Discourse itself but is more meaningful to us. So for instance when a user is posting from inside a course, we don’t need/want to display or query any data outside that category structure.
Another thing is, courses are shorter-lived that a typical Forum categories are supposed to be. A new course will pop up, run for a while and then close including all discussions in it. For Discourse this is a burden, but we can hopefully better optimise that.
There is definitely going to be a Discourse plugin, and potentially PRs against Discourse for this. The plugin is definitely meant to handle category and topic embeds, but beyond that it’s vague right now. One additional thing it can do is to let Discourse ignore archived/closed courses unless explicitly requested.
Potentially these can even be exported or otherwise backed up and then deleted.
Per-unit categories are not compulsory. The aim is to enable both approaches, i.e. per-unit category, and per-unit thread/topic. Sharing a common forum space could also be possible without much difficulty, we could allow toggling the permissions such that these categories are not hidden, and locked to particular groups, it would simplify some things, but also make other things more complex since it doesn’t get rid of the constantly growing categories and threads as new courses are run and closed.
I definitely think there will be changes to the interaction model, and switching from one system to the other will not be a 1:1 exchange. Features will be lost along the way, but with the explicit understanding that the new features we gain in exchange make up for it.
When I say new features make up for it, I don’t mean the new features will provide the same functionality, but that they will enhance the experience in other ways.
What I’ve realised (and this is just my own personal view) is that a lot of the issues with discussions are hard to avoid if it’s a component that’s part of a larger solution. I can deeply empathise with why it isn’t feasible to just tack on a CMS into edx-platform. While that would be possible, it could never catch up to Wordpress, Drupal or other such CMSs, but most people would just want that. The people who contribute to other CMS solutions won’t be interested in contributing to a CMS that’s locked into another platform.
Discussions and Forums I feel are pretty much in a similar spot. Improving an Open edX integrated forum or CMS will only ever improve the experience for those who use the Open edX platform, and as such it will always have a smaller audience, fewer developers and improve at a slower rate than an external forum or CMS that has a wider user base. So it’s better (I feel) to integrate with an external system that provides the features that are needed, (or accept that fewer features are the cost of deeper integration).
Right. I know you folks are already thinking about these things, which is great. I am just very wary of committing to an integration mapping model before prototyping how it behaves at the scale of thousands or tens of thousands of courses. Premature optimization can be dangerous, but having some tracer bullets to test the limits the data model would help mitigate a lot risk.
Take the example of wanting to enable both the per-unit category and per-unit thread/topic. That represents a two order magnitude difference in terms of the number of categories. For ten thousand courses, maybe that’s the difference between 10K and 1M categories. I would much rather enable only per-unit threads in the first iteration, and then see how strong the demand for per-unit categories is, as well as how it affects Discourse performance.
There are multiple ways we can do the Unit:Threads association. My big worry is that we will create a popular feature (e.g. per-unit categories) in a way that doesn’t scale past a hundred courses, and then we will have to scramble to create more drastic, backwards compatible, and operationally complex measures in order to maintain site stability (e.g. shard it out to ten instances of Discourse). We’ve all been there, and it is not pretty.
FWIW, I completely agree with both your assessment of the tradeoffs and your conclusion about the best course of action. I very much hope this effort succeeds. If this integration proves to be popular and operationally sane (and I think the odds are good), I would be happy to deprecate our existing forums solution altogether in favor of supporting only an integration layer, with Discourse being the recommended service to integrate with. To be clear: I don’t have the final say in that on my own. But I would certainly advocate for that position.
Sure, I definitely think it might be valuable to do some benchmarks here and have a good picture of what these limits our, especially in this use case.
Sure. Given that Discourse needs more work to support embedding categories that sounds like a good way to pace the development. Threads are closer to being embeddable in any case.
@xitij2000 Thank you for this proposal. I have some thoughts on how it would be beneficial and directionally aligned to implement this as an edX Frontend Plugin, rather than an XBlock or an edX backend plugin. I have added a few comments on the doc to this affect.
@nimisha There are a few reason for why I felt this would need a plugin, I have outlined them in my reply on the doc.
Could you help point me to the latest discussion/documentation on frontend plugins? I would love to follow that and get a better understanding of how that could be used here.
Very exciting. As an aside, there’s currently discovery within edX into moving to Kubernetes and making it the default Open edX deployment strategy, so the fact that Discourse is deployed using Docker is great.
Is this initiative still in the planning/proposal phase, or is it being actively developed now?
Yes - this initiative actually still needs financial sponsors. We are figuring out how to do this technically, but we still need to find funders for the actual development effort, afterwards. So if anyone has an interest in ensuring that this will happen, let us know.
Since we run instructor paced courses that have a begin and end date, we have been wanting to form an alumni community space for learners once they complete a course. It looks like Discourse could provide this since it is decoupled from a course instance in edX.
@dave, we did some performance testing on Discourse. We created 1e4 + 9 categories for checking how they would affect the pages. It turns out that loading the full page is indeed slow with many categories - loading it with 1e4 + 9 categories takes around 6-10s. However, when we have it loaded and switch to a specific category, it takes around 130ms. I’m providing the metrics below.
Example metrics for 1e4 + 9 categories:
Reloading whole page: 9745.9ms.
Switching to the category: 129.2 ms.
Loading “about” thread": 1450.3 ms.
For 5 categories it looks like the following:
Refreshing page| 196.6 ms.
Switching page|: 132.4 ms,
Loading “about” thread": 167.8 ms.
So it seems that this part of the code is responsible for this slowness while loading the page. If we would be using API with a custom UI, it wouldn’t affect us - as the metrics above state, loading single category, nor its topic, is not affected. But if we decide to use the full Discourse UI, we will need to create an option to disable calculations performed on all existing categories, but this will require more context of the codebase.
There was also a small problem with loading single thread, when IDs from this condition were being evaluated and as a huge query to the DB, taking 1.5s to process (for 1e4 + 9 categories). We found two fixes for this:
Btw. we also checked that the profiler didn’t have any significant impact on the page loads. Disabling it didn’t bring any noticeable performance differences.
The other idea is creating one category per course and then adding tags (which are lightweight and don’t affect full UI performance even after creating 1e6 of them) to specific topics with something like ${course_key}-${cohort_id} (or even having a flat, categoryless, structure). However with this solution we lose the permission management on the Discourse side. What do you think about this?
I really like this proposal, Discourse is clearly a great (if not the best) choice for providing a forum experience now.
However, currently (as far as I know) there is no official WYSIWYG editor plugin for Discourse, the users need to use the Markdown based editor that is used on this Discourse instance as well.
I don’t know if this is a problem at all, I really like the current editor of Discourse, but I don’t have a problem with writing Markdown
It hasn’t seen many updates, but since it is built on top of React, it could be used not just as an editor on the Discourse site, but maybe as an embedded editor directly in Open edX as well.
Xavier, as of today, with Juniper, is there a (very simple) way to integrate discourse into open edx instead of the current, quite antiquated, forum system ?
