Proposal: Devops Interest Group Structure

Working Groups Build-Test-Release
, ,
1

In the Contributor Meetup a week ago we discussed the creation of a Devops related Working Group that is trying to focus energy and expertise around deployment of the platform.

We currently have three groups that are focused on operations topics:

  • Build, Test, Release (BTR), predominately focused on work to produce official platform releases twice annually.
  • Developer Experience, which is focused on containerization as a developer accelerator. This group is working on both legacy Devstack and Tutor alternatives.
  • The Devops Group, which has been focused on deployments with Kubernetes, but is interested in advancing other operational topics as well.

I believe that all this work is related in a couple ways:

  1. These topics are likely to be of interest to the same set of people, we should make the most of scarce contributor time and headspace.
  2. Given our commitment to containerization, we get the most value with differences between our development environments and production environments are minimal.

I also believe that coordination of similar efforts increases value and decreases waste.

Given that, I think we should create an over-arching DevOps Working Group with two sub-groups or task forces. The sub-groups would be:

  • Build, Test, Release
  • Developer Experience

In order to do this we need an overall leader or leaders for the DevOps Working Group and a leader to assume responsibility for running BTR and shipping the next release, Palm.

Please discussions questions, concerns, and other ideas in this topic. If you are interested in any of these leadership opportunities, please also note that.

1 Like
2

(FYI @Rebecca_S_Graber who co-chairs DevExp with me)

Well, BTR is chairless, and DevExp WG is still deciding when to have a first meeting, so I guess it’s as good a time as any to make organizational changes :stuck_out_tongue: There is certainly overlap between the three groups, so having some way to coordinate would be good.

As long as we can find two folks to step up for the BTR and DevOps chair roles, I am OK with this. I don’t feel very strongly, though, so I am interested to hear what others say.

1 Like
3

I agree with this proposal. We need to strike the right balance between too few/broad and too many/focused working groups. I think that merging BTR with developer experience and devops will lead to more collaboration across a diversity of projects. For instance, many Open edX developers would have a lot to learn from devops people.

That being said, I’m wary of all-hands meetings. We should be careful not to schedule multiple meetings every week where a large number of participants are invited. Instead, we should encourage smaller meetings between people who would like to work together.

I’d like to apply to the position of the DevOps working group chair :slight_smile:

4 Likes
4

I’m unclear on what this would mean in practical terms. In particular, my questions are:

What would the responsibilities of the DevOps group be with respect to the subgroups?

Would members of DevOps work on distinct issues/tickets from BTR and DevEx or would it be more of a coordinating group just to make sure there’s no unnecessary overlap and that BTR and DevEx are aware of what the other is doing?

Would DevOps just comprise various members of BTR and DevEx or would it be a different group of people (with presumably some overlaps)?

I can see the need for coordination between BTR and DevEx but am not sure how DevOps would … operate if its only role was coordinating two other groups.

2 Likes
5

Perhaps I can try to answer that one :slight_smile:

The way I see it, the BTR, developer experience working groups and future efforts aimed at improving the Kubernetes integration have a lot of overlap. That’s perfectly fine; it’s actually great that people are working on closely related topics. But it means that engineers who are interested in contributing to these topics or learning from these groups will not necessarily know who to contact. I’m thinking in particular of edX engineers. Having a single point of entry (the DevOps WG) makes it easier to join existing projects.

Moreover, it’s important that people working on one of these topics interact with others. For instance, the developer experience working group would greatly benefit from research made by the Kubernetes group. Similarly, the BTR should discuss with the developer experience group to make sure that the next release does not break the developer workflow.

Finally, it’s difficult for smaller groups to make themselves heard whenever they need help. If we have a single communication channel, it will be easier to ask for support outside of the existing members.

To summarize, in my opinion, the role of the overarching DevOps WG will be mostly of coordination:

  • To facilitate the external communication of the working groups with the rest of the world, in both directions.
  • To foster cooperation internally, between members of the working groups.

And to answer your questions:

The DevEx and BTR would become teams of the Devops WG. They will still have complete freedom to organize as they need.

New DevOps needs will emerge over time. It will be the role of the DevOps working group to build new teams to work on these projects and to build bridges between teams.

Working group membership is a fluid concept anyway :slight_smile: If we consider BTR and DevExp as teams of the DevOps working group, then yes their members will be part of DevOps. BTR and DevExp will be a subset of DevOps. New members of DevOps will join one of the existing teams or create new ones. These new teams may be temporary; for instance, when they need to work on scoped projects with an end date.

In practical terms, we will:

  • collect community feedback in a GitHub project
  • explicitely flag “good first issues” to attract new members
  • nudge DevOps members to tackle issues
  • communicate about new projects in the forum
  • detect cooperation opportunities by following the progress of each team. Basically, we will periodically collect the following information:
    • What is each team working on, and what is the expected deadline?
    • What issues are you facing?
    • How can we help?
    • What have you completed that we need to communicate to the rest of the community?

@Rebecca_S_Graber do you think this would be a good approach? @e0d I’m curious if this matches what you had in mind.

1 Like
6

Thanks for your thorough response!
To further clarify, would the flow for a particular project be something like:
Someone in the community adds an issue to the DevOps GH project
The DevOps WG decides if this issue fits best with DevEx or BTR or whether it might require another team entirely
Whatever team adds the issue to their own project and takes ownership of working on it
Periodically the team reports progress on the issue back to the rest of the DevOps WG and/or asks for help if needed
When completed to everyone’s satisfaction, the DevOps WG as a whole takes responsibility for communicating this out to the rest of the community

I’m also curious how we think the DevEx team at 2u (aka Arch BOM) will fit into this structure. Will they communicate with DevOps or with DevExp?

7

Yes, very much so. And I don’t think we need to answer every question upfront, e.g., a what level Arch BOM should plug in. As long as there are mechanisms that increase coordination and communication, while preserving as much autonomy as practical, I think we’ll have a winning formula.

8

Any further thoughts from the community? Any other interest in chairing or co-chairing the group? If there’s no disagreement perhaps it’s time to talk about the practical steps to take to make this happen.

9

  • First I am happy to see this proposal/change, I definitely believe those areas intersect with each other as you already have said, and sometimes it’s important for a particular change (be it proposed from BTR, DevExp, or DevOps) to be communicating widely so say if we do arch change for BTR it doesn’t break other areas in the platform…etc.

  • Regarding the roles of the new structure, my conclusion so far

    • Regis chair DevOps
    • Kyle and Rebecca chairing DevExp.
    • BTR as of the last meeting Adolfo announce Jorge Londoño slack message and meeting

Given all that aside, as someone who has been active in BTR mainly, I am happy to continue to devote time to resolve pain points between those groups but mainly focus on BTR.

  • If I have a project/proposal that I think crosses all groups what is the way to move forward? where do I submit my thoughts about it?
10

I’ll start bootstrapping the various GitHub projects, Jira pages, etc. but I need help from an administrator for the following tasks:

Create the following project and make me (@regisb) an admin: https://github.com/openedx/devops-wg/
Create a “DevOps” subcategory of the Working Groups category in this forum.
On Slack, create the #wg-devops channel.

If you know for a fact that an issue should be addressed by the BTR or DevExp, then by all means go to them. If not, go to DevOps. If necessary we’ll dispatch the issue/conversation later.

11

@regis I can help with this. Just to clarify, do you want a new repository, a new project, or both? Keep in mind that issues can be created in any repo, and then added to any number of projects. For example, openedx/wg-developer-experience is where we create general DevExp issues, which I then add to the DevExp project along with issues from other repos, including edx-platform and various tutor repos.

12

It would be great if you could create both.

Right. I think it’s best if issues are tracked in just a single project, so I expect to keep issues that belong to DevExp in their own project. Issues that are created in the devops project will be moved to BTR/devexp if need be.

Do you think this is a sensible approach?

13

Yup, separate projects sound good to me.

What doesn’t seem sensible, now that we have a project for every working group, is also having separate repositories for every working group. I’d rather have one big working-groups repository for all broad issues, and then have all technical issues go in the most-related technical repository. Each WG’s issues would then be pulled together in a single place–their project board. Moving issues between WGs would just simply involve moving them between project boards.

I digress. You don’t need to worry about that for this new WG :stuck_out_tongue: I’ll go get the devops repo and project created.

14

Repo: https://github.com/openedx/wg-devops

Project: https://github.com/orgs/openedx/projects/42

1 Like
15

Yeah, I think I agree. Except that the GitHub project could be used not just among working groups, but with the entire Open edX community. Basically, this would be a place to centralize issues, which would then be dispatched to specialized teams/working groups. It’s better than to have issues spread out over so many repositories/projects. It’s also better than Jira.

Thanks for creating the project & repo :slight_smile:

1 Like
16

4 posts were split to a new topic: Single GitHub project for all working groups?

17

@nedbat can you help me with the latter two items?

18

I have morphed the Developer Experience group into the Tutor Users’ Group, which seems like it still belongs under the DevOps umbrella.

That said, since this nested working group structure is now a year old, has it led to any new coordination between the sub-groups?

19

Not really :-/ I am doing a poor job at managing this devops working group, mostly because I want to avoid creating another synchronous meetup. I’m all open to ideas on how to manage this group better, though.

20

No sweat @regis ; on the contrary, I think you’ve been doing a great job providing devops leadership–the Tutor maintenance program, your thoughtful PR and ADR reviews, nudging forward the upstream improvements that you know operators are dying to see–those are all at least as important as literally running a working group :slightly_smiling_face:

As for DevOpsWG, I suggest two approaches you could take:

  1. Dissolve the DevOps WG umbrella. Let Build-Test-Release, Large Instances, and the Tutor Maintainers continue doing their respective jobs and coordinating informally as they need to.
  2. Keep the DevOps WG umbrella, and have a monthly checkin between yourself (as DevOps Chair) and the leaders of the various sub-working-groups. If you’d rather not have a standing meeting, it could just be a Slack thread or a forum post.