OEP-45 and replacing edx-configuration with another reference deployment strategy

Development Architecture
1

@billatedx and @Felipe, I see OEP-45 is ready to be merged, but at Opencraft we’re still concerned about the absence of wording regarding a reference implementation. We raised the issue as comments to the OEP itself, and I brought it up in two instances of the contributor’s meetup. I hate to be so pushy (it comes from an honest belief that this will be best for the community), but would it be too much to ask for us to get together one last time before the merge? The next contributor’s meeting is too far away, and, in any case, I think this issue warrants more than the few minutes we’d have for it there.

Basically, the idea I want to sell is that if edX don’t make it clear from the get-go that there will be an open source reference implementation to replace edx-configuration, also one that is used by edx.org itself (even if it’s community-maintained!), then we’re missing out on an opportunity to bring everybody together under one devops roof.

What do you say? Can I send out a Doodle with proposed times? If not, that’s fine, but at the very least we can use this thread to give the subject the dedicated space it deserves.

Thanks!

CC: @nedbat, @regis, and @antoviaque.

This is the Doodle, with proposed times for the week of 2020-05-31:

4 Likes
2

If you send a Doodle I’ll be happy to take part in this conversation. The last couple days were pretty busy with the Juniper release, and this OEP is unrelated anyway. Also, I think this conversation goes beyond the general scope of the contributor’s meetup. Thus, it makes sense to spawn a separate conversation on this topic.
(note that I deleted the other posts you created on that same topic)

3

Awesome! I’m just waiting to see if Bill and Felipe are up for it.

Definitely.

Thanks! :+1:

4

Maybe I’m too much of an operations newb to understand the subtleties here. The abstract of OEP-45 in the pull request includes:

The configuration for running IDAs in different environments will be done through a config file per unique [application, environment] combination. How those files are generated and managed is the responsibility of each operator.

All standard operations for running and managing any Open edX IDA will be documented. How those procedures are executed is the responsibility of each operator.

Those paragraphs (especially the “responsibility of each operator”) seem to explicitly state that there will not be one reference implementation that everyone uses.

@arbrandes: is your goal of “everybody together under one devops roof” at odds with the stated goals of OEP-45?

5

Not at odds, no. An extension of them, perhaps. I would welcome, for instance, the addition of a few words to the first sentence of the last paragraph you quote:

" All standard operations for running and managing any Open edX IDA will be documented, and a reference implementation provided."

The part of it being the responsibility of each operator can, and should, remain. After all, though edx-configuration is widely used, one is not obligated to do so. But I defend that it is still a good idea for such a reference to exist.

Note that though the change suggested above would be a good outcome as far as we’re concerned, it is not the only possible one. I can imagine, for example, coming up with a separate OEP, just for the purposes of a reference implementation.

6

@arbrandes I purposefully left reference implementations out of this OEP based on the conversation that was had at the contributor’s meeting which I took to be consensus that an edX blessed implementation is not necessary since most of the setup complexity would be captured in the container. I am happy to get on a call and try to understand which parts of a reference implementation would be valuable and see if we can come to some shared understanding.

7

Ok, great! I’ll update the top post with the Doodle, but here it is. Basically some slots next week:

Hopefully by the end of the week we can get Felipe’s attention as well.

8

I agree that the existence of edx/configuration helps operators a lot as a reference. It does give an idea of how can the services be deployed. And from that point on the operators can move their separate ways depending on their requirements.

I think that the reference deployment implementation for Docker/Kubernetes should just focus on a basic deployment of all applications, even if the reference mentions in-container complexities handled manually.

1 Like
9

I am also available to chat about the reference implementation. I filled the doodle already. @arbrandes please leave it open for a while to see if more member of the community are interested in participating.

1 Like
10

Thanks for the feedback, @NIXKnight! As @Felipe says, you’ll also be welcome to join the meeting once a time is decided.

And @Felipe, thanks for picking a time as well! I’ll leave the Doodle open until at least the weekend.

11

Thanks @arbrandes I have also filled the doodle.

12

Alright, seems like tomorrow at 15:00 UTC is a good time for most. Here’s the Zoom invite:

OpenCraft GmbH is inviting you to a scheduled Zoom meeting.

Topic: OEP-45 and reference deployment implementation
Time: Jun 3, 2020 15:00 PM UTC

Join Zoom Meeting
Launch Meeting - Zoom

Meeting ID: 850 6323 5545

I’m also uploading an ICS file for easy import into calendars: meeting-2020-06-03.ics (676 Bytes)

See you folks there! (CC @billatedx, @Felipe, @regis - and I hear @jill may also be interested.)

1 Like
13

I will probably try to attend too.

14

Roger that! Everybody’s welcome.

15

A quick recap of the meeting, for those interested, starting with the conclusion first:

Conclusion

While it was decided that OEP-45 is not the right place to refer to a reference deployment implementation directly, edX agreed to add some wording to it addressing the community’s most pressing concern: that the possible future deprecation of edx-configuration could lead to the loss of visibility into how edX does deployments.

In other words, while not everyone actually uses edx-configuration for deployment, a lot of community members do use it as a reference; if it goes away (at once or incrementally), prior to the meeting there was no guarantee, written or otherwise, that information coded exclusively in edx-configuration would continue to be provided. The primary example given was the LMS nginx configuration: it is currently an important bit of deployment lore that would be deprecated by OEP-45, but for which OEP-45 offers no provisions.

With that in mind, @billatedx was already kind enough to add some wording to alleviate this concern. In short, if something is taken out of edx-configuration and it’s important for deployment, it will be replaced by an equivalent source, and made publically available. Thanks, Bill! :slight_smile:

But will there be a reference deployment implentation, after all?

The short answer, for now, is: possibly. And one of the main options is to make it the devstack.

We arrived at this possibility by recognizing that even if edX doesn’t publish the full solution that it actually uses for production deployment after OEP-45, or if it does publish it but doesn’t make it a one-stop-shop like edx-configuration, then if:

  1. a simplified but feature-complete deployment implementation is made available, and
  2. it uses the same Docker files (though not necessary the same images) that are to be produced as part of OEP-45, and
  3. it is subjected to CI/CD so that everybody’s alerted to deployment problems ASAP,

such an implementation would probably be a good-enough, stable, and guaranteed-to-work-with-master reference deployment for the community to use as a basis for their own projects.

If the devstack were to conform to the above, then it would be a prime candidate. Particularly given that edX does actually use the devstack to develop edX. (I wasn’t sure, but Bill confirmed it. ;)) Plus, such an outcome would mean that the devstack would move closer in step with the master branches it uses, resulting in a likely more stable development experience.

To reiterate, though, this is for now only a possibility. edX has agreed to continue discussion on the topic once OEP-45 is approved and progress starts on an actual implementation plan.

Thanks to all that participated! If I missed something, or got anything wrong, please feel free to correct me.

6 Likes