Tutor Enhancement Proposal (TEP): Plugin indices

Community
1

Hey y’all, I hope you had a good summer holiday :sunny:

I’d like to introduce a change in Tutor in how plugins are managed. This change would serve different purposes:

  1. Make it more clear, convenient and reliable to install/uninstall/upgrade/search plugins.
  2. Publicly recognize the contribution of plugin maintainers and developers.
  3. Incentivize Open edX developers to distribute their contributions via Tutor plugins.

Objectives

Plugin management

To resolve item #1, we will introduce the following commands, strongly inspired by the apt CLI:

$ tutor plugins search plonk
plonk/14.0.1
    plonkify the fooberator
pweeet/14.2.0
    ungrunt the plonkifactor
$ tutor plugins show plonk
Version: 14.0.1
Maintainer: John "Xbl0ck3r" Doe <johndoe@xblock.com>
Homepage: https://github.com/overhangio/tutor-plonk
URL: https://pypi.org/project/tutor-plonk
$ tutor plugins install plonk
...  # pip install tutor-plonk<15.0.0,>=14.0.0
$ tutor plugins upgrade plonk
... # pip install --upgrade tutor-plonk<15.0.0,>=14.0.0
$ tutor plugins upgrade
... # upgrade all installed plugins
$ tutor plugins remove plonk
... # pip uninstall tutor-plonk

Behind the scenes, most of these commands will call the pip CLI. The list of plugins will be fetched from a public index. The default will only include the official plugins, stored for example at: https://raw.githubusercontent.com/overhangio/tutor-plugins/master/nutmeg/official.yaml

It will be possible to add and remove indices. This will be useful in at least two use cases:

  1. Contrib (non-official) plugins.
  2. Non-open source plugins

To achieve that, we will introduce the following commands:

$ tutor plugins index add https://raw.githubusercontent.com/overhangio/tutor-plugins/master/contrib.yaml
$ tutor plugins index list
https://raw.githubusercontent.com/overhangio/tutor-plugins/master/nutmeg/official.yaml
https://raw.githubusercontent.com/overhangio/tutor-plugins/master/nutmeg/contrib.yaml
$ tutor plugins update
...

The syntax of the index files is not well defined, yet.

We want to be able to install plugins from different sources:

  1. Pypi-compliant projects that implement the JSON API.
  2. Git repositories.
  3. URLs that point at single-file Python plugins. (for instance: GitHub gists)

Note that different Open edX releases will have different indices. EDIT: There will also be extra “nightly” indices for the nightly branch.

Better recognition of Tutor maintainer contributions

Item #2 was a sub-goal of the Tutor maintainers TEP that I didn’t implement in a satisfying way. There is a public list of Tutor maintainers (here) but until now we are (really: I am) doing a bad job at highlighting the extent of their contributions. I hope to improve that by including the plugin maintainer’s name/company in:

  • the “maintainer” field of the plugin’s setup.py file, which will be displayed by tutor plugins show and the pypi page.
  • the plugin README file, which will be displayed on GitHub and in the Pypi description.

Encourage plugin creation

I want to encourage the creation of non-official Tutor plugins by making it possible to install plugins from contrib/3rd-party (or even private) indices. See above for the CLI. Thoses indices might be listed in the official github/overhangio/tutor-plugins repo, but their authors are encouraged to advertise them themselves.

Because indices are clearly different from one release to the next, this should encourage plugin maintainers to quickly upgrade the plugins they maintain after a new release.

Feedback

Comments will be appreciated! Please reply to this topic below. I am especially interested in the opinion of @tutor-maintainers and plugin developers (:eyes: @fghaas @insad @kmccormick @mgmdi) If you don’t have any specific comment to make but you think this would be a welcome change, just hit the :heart: button.

Thanks!

17 Likes
2

It would be nice to have a way to ensure also third party plugins are being kept up to date with new open edX / tutor releases. I’m seeing the lack of upgraded and tested plugins holding back the implementation of the latest release. And I’m not even speaking of the open edX platform native problems, like the payment processor drama that goes on and on.
Wouldn’t mind to put my (only) plugin on some kind of common github account, where e.g. also tutor maintainers have merge rights, so if one day a coconut falls on my head, somebody can take over maintenance.

2 Likes
3

Overall, I really like the idea. A few thoughts:

  1. In addition to a set of indices for each Open edX release, will there be Nightly set of indices?
  2. What standard will a plugin need to be to make it into the upstream “contrib” indices?
  3. Who will maintain upstream indices from release to release? While I totally trust your judgement in curating these indices @regisb, we are currently in a place where if there were an Open edX release scheduled while you were on holiday, or busy, or recovering from being hit in the head with a coconut, we’d have to fork Tutor in order to make the release happen. If you’re willing to share responsibility maintaining the official plugin index, though, then I believe it’d help us increase the bus factor of the release process, without having to grant anyone else direct merge access to overhangio/tutor.
2 Likes
4

Hello there! Some -random?- thoughts :dancer:t3::

  • This is nice! I think most of the pip-commands I use are there :heavy_heart_exclamation:
  • Since not all plugins are updated by the time of the release, it would be nice that the command tutor plugins upgrade doesn’t crash completely when a few plugins are incompatible with the new version. This could also serve as an indicator for which plugins are left to update. But if you have already decided on the behavior, let us know!
  • We could borrow the OEP-55 maintainers’ program readme recommendations & add the maintainers’ section to the tutor plugin repos! The OEP recommends adding backstage info of the repo, but this could be another type of metadata/identification.

I’m tagging you here @Alecar since your day-to-day is tutor plugins :stuck_out_tongue_winking_eye:

4 Likes
5

I would like to ask how item #2 will work with the single python file plugins shared via Github Gists since they won’t feature a setup.py or a README file.

Will there be some kind of metadata required for that type of plugin?

1 Like
6

I am assuming this was meant to be just an example, but just in case it wasn’t: that URI is currently a 404. :slight_smile:

But more importantly, are you sure about the default to include only the official plugins? Shouldn’t it at least also include tutor-contrib-* plugin packages that their authors have published on PyPI?

Allow me to add a comment to this, speaking for a team that maintains several contrib plugins: I’m afraid that plugin users will just have to deal with the fact that Open edX “open” releases, Tutor releases matching the updated edx-platform, and then Tutor plugin releases matching the updated Tutor can only happen in succession. So that means that when a new Open edX release drops, it’s simply not realistic to be able to upgrade to it, using Tutor and contrib plugins, immediately.

I think plugin users will further need to accept the fact that plugin authors often run Open edX and Tutor in production themselves, and until they upgrade their own platform to a new Open edX or Tutor release, they may have just a limited incentive to upgrade their plugins. So, for anyone who needs a Tutor plugin updated for a new release before the plugin authors do, I guess the standing request is to please consider sending a PR.

1 Like
7

Hello everyone! :stuck_out_tongue_winking_eye:

Excellent, I have been waiting for these commands for a long time.
In TVM We have been working using the plugin commands but with this, we could deprecate the commands in future versions and recommend using these.

2 Likes
8

Thank ya’ll for your insights! I’ll try to address your comments one by one.

Yes, definitely. Delays in upgrades have always plagued the Open edX community. I think that by creating release-specific indices we can better keep track of which ones are not up-to-date. Because we will have a “maintainer” email address, we can get in touch with the authors individually and encourage them to upgrade.

I don’t necessarily agree. Yes, coconuts are deadly. But it’s important that you are clearly identified as the author and maintainer of a plugin; this is why the repo should be hosted in your GitHub account.

That being said, it’s important to prepare for a coconut scenario. But that’s what open source is for: anyone can fork your project if they decide to maintain it. And then in the contrib index we can switch to someone else’s fork if you’re coconutted for too long.

Yes, good point, I did not think of it. I’ll edit my TEP.

That’s a tough one :slight_smile: I don’t have a definite answer, to be honest. But of course it’s a very important question.

I think that we should try to emulate the Ubuntu “universe” repository (which was the inspiration behind our “contrib” index): ContributeToUbuntu - Ubuntu Wiki

Basically, anyone can submit new packages to the MOTU provided that they are committed to their maintenance: Debian -- Work-Needing and Prospective Packages

We could probably adopt the same policy. We will have to pay close attention to the pace of upgrades for each plugin. Which leads us to your following question…

The official and contrib indices should be managed by Tutor maintainers – and not just myself. To be honest, I expect that these indices will need very little work. When a new release candidate is created, all plugins will be commented out from the index. And then we will re-introduce them one by one as maintainers upgrade them to the new release.

Yes, you’re right, and I definitely don’t want to be a bottleneck. I mean, you never know when an African swallow might fly over your house, right?

We now have Tutor maintainers for some (not all, unfortunately) of the official Tutor plugins. I did not grant commit rights to anyone else for the core repo, yet; the reason for that is that I’d rather wait until the Open edX code is extracted from the core (something that I mentioned here). I hope that we can reach that state soon, maybe before Olive.

In any case, the indices will have Tutor maintainers with commit rights, so at least these will not have to be forked when I’m coconutted.

Good point. I think that this feature can be part of the tutor plugins upgrade CLI – for instance with a -p/--partial option.

Yes, good point.

For these plugins we will have to add extra metadata to the index yaml file. But I don’t have the specs for these files, so I can’t give a more precise answer. I’m sure we’ll figure out something, though :slight_smile:

yes :slight_smile:

I think that we should make it very easy to enable the contrib index, but that it should not be enabled by default. Enabling it should be a conscious decision. If not, people will assume that contrib plugins are officially supported, which is not the case.

Absolutely. This is the current situation, but I’m sure that it can be improved by figuring out a better pacing for the release candidates. Hopefully, we should be able to work with plugin maintainers such that all plugins are ready to upgrade in time for the actual release. Of course, this can only happen if we communicate with plugin maintainers during the two months period between the release branching and the actual release date.

To @kmccormick’s point, I think that timely maintenance should be mandatory for plugins in the “contrib” index. What that “timely” means is of course the crux of the matter; my opinion is that it’s reasonable to expect that contrib plugins are either updated or forked before the release date, but I’m totally open to a discussion here.

For non-contrib, 3rd-party plugins/indices: maintainers should be free to upgrade whenever they want. And that’s the implicit contract of open source.

6 Likes
9

I support this :+1:

Makes total sense. It’s great to hear you’re still interested in the core extraction and I’m looking forward to helping where I can.

Awesome, this is great step forward!

This TEP gets my :seal: of approval.

1 Like
10

Hello, I was passing by to comment that I like the proposal. :blush: :raised_hands:

The only thing I wanted to add is that it would be good somewhere in the documentation to specify how the versioning of the plugins should be because it is not so evident that the plugins must follow the tutor version. The install and upgrade commands could generate problems if that versioning is not complied with.

2 Likes
11

I’m happy to announce that I’ve opened a PR to implement this TEP: feat: plugin indexes by regisb · Pull Request #783 · overhangio/tutor · GitHub
Indexes maintained by Overhang.IO will be stored here: GitHub - overhangio/tutor-plugins-index
All reviews are welcome :slight_smile:

3 Likes