Core contributors commitment standard

I’m creating a dedicated thread to host the conversation about the commitments the organizations of core contributors make when they join the program - it was originally brought up in the discussion about work scope, which is on a separate topic – the two are related, but having them separated will give us more time to discuss on the current thread, without blocking the other review.

I’m also linking to a document I had started drafting earlier to start the conversation about setting a specific standard for commitments:

https://openedx.atlassian.net/wiki/spaces/COMM/pages/3594256444/Core+Contributor+-+Organization+Commitment+Standard

2 Likes

Thanks for your work Xavier, I think it’s great that we have a more precise definition of what counts as core contributor work. It’s also great that participation in the various Tutor projects also counts towards CC work.

Do you think that having better scoped roles will allow us to stop evaluating CC participation in terms of number of hours of work? If we can track CC work through issues, then surely it’d be better to evaluate core contributors based on the issues they help resolve? For instance, we could assign complexity tags to each issue (easy/medium/hard/nightmare) and count for each CC the number of closed issues. This would be a much better proxy for participation in the CC program than work time.

3 Likes

My understanding is that the hours are a way to communicate what is expected of a core contributor in return for the rights they get. The ultimate goal being to get more value to the community at large, rather than just scratching-your-own-itch.

That said, I’m with you in believing that hours are a poor proxy for value: it’s why more experienced consultants get paid more by the hour, after all. But given the disparity in contributor roles - it’s not just code we’re talking about - I don’t think there are many other common denominators if we’re aiming for objectivity.

I do think we should seriously revisit this, though. I have a gut feeling that the tally of hours might not be achieving what it’s supposed to, and can in fact be hindering people from joining the CC program. IMHO we would all benefit from a proper retrospective.

At first glance I don’t see many advantages to this over hours. It’s as poor a proxy as, say, measuring lines of code. For instance, I’ve sometimes spent months in a single ticket (ask @cmuniz about OEP-58), or days and weeks agonizing over bugs that were fixed by a single character (thanks, C pointers).

I have a third option to propose: what if we decoupled any kind of objective tally from implicit or explicit requirements, instead relying on a decentralized, communal mechanism to periodically check whether a CC is still up to the task? We already vote to let people in. All that would be missing is a way to vote people out. :man_shrugging:

As for any perceived loss in public recognition, there’s been talk of having a ritual where all contributors are noted. And one way to recognize the most valuable ones could be to, say, have a public ranked vote. The conference would be a pretty good place for this.

My R$0,02.

1 Like

I agree that time is a flawed metric, but I also think it’s the only common currency that we share, and so I think it’s the best one we have.

This came up at the BoF where we had these initial discussions, because someone asked whether donating $$ could be another way to become a core contributor. I don’t remember if there was a consensus about the idea of donating $$, but the value in it was in buying time for someone to contribute, which to me, strengthens the argument that time is the currency of interest here.

It’s true that some people can earn more $$ for their hours, but I don’t think that makes the time “worth more” to the individual that donates that time. The value to the community may be higher from one CC to the next, but the cost of the gift is the same – each of us has the same number of hours in a day.

I also acknowledge that working for OpenCraft biases me here – we’re contractors, so we’re accustomed to measuring everything in hours, and so the reporting part isn’t onerous to us. But we’re also not demanding perfect accuracy in reporting either – CCs can volunteer as much granularity and detail as they want to.

Sure, but what would be the criteria for voting people out? How do we keep this objective? I think this would end up just being another way of asking the same question.

3 Likes

I’m glad we can have an open conversation about work time as a metric. As far as I remember, at no point did we decide as a community to rely on work time to rank core contributors. I think it’s a tremendously important conversation to have.

Right, I’ve been there as well. In such as situation, I think it would be fair to say that you closed a “nightmare”-level issue, right? This would grant you a badge that recognizes that you have extensive experience with Open edX. Because beginner contributors are unable to resolve difficult/nightmare issues.

I think that we should not be ranking core contributors according to any single metric or KPI. As soon as we start relying on a single KPI, that metric instantly becomes flawed because people start optimizing for it. But “time spent on a issue” is an even worse metric because the incentive becomes for contributors to become less efficient. Core contributors are rewarded for spending more time on any single issue. For instance if I spend 50 hours working on a few easy issues I will be catapulted to the top of the core contributor leaderboard. I think everyone would agree that we do not want to encourage this kind of behaviour.

If we agree it’s a flawed metric, then please let’s not use it to rank contributors! We do not need to rank core contributors, so let’s stop doing it.

I think that this metric is actually worse than “flawed”: it’s downright terrible and toxic. Consider this from the perspective of an Open edX outsider. Would you join an open source project that encourages presenteeism? Where your contributions are unfairly compared people who work full-time on the project? Have we seen this metric being used in any other successful open source project? This metric acts as a deterrent for other people to join the project.

This is a great point.

What behaviour do we want to encourage? I think that we can all agree that we do want to attract $ donations, right? So we need to find a way to publicly recognize the impact of financial donations and sponsorship. From this perspective, I think that we should include financial sponsors in the CC program. Sponsors would not receive CC positions, but it would make sense to create an unlimited number of “CC sponsor” seats.

This is false. The fact that an individual can earn more money per hour definitely makes their time worth more. When they contribute for free to an open source project the loss of opportunity means they will make less money. So the cost of the “gift” is not the same at all.

Anyhow, I’m not asking either that we rank core contributors by hourly rate, or the financial value of their contributions. Just to reiterate my position: we should not be ranking core contributors at all.

Reporting hours isn’t onerous to me either. This is not an issue for me. What is an issue is that reporting hours makes me feel like another cog in the machine. It’s killing the joy of contributing to an open source community. I want to feel like I’m a unique snowflake, not a nine to five employee – and I suspect many other contributors are like me.

Let’s ask the same question again: what do we want to achieve? What sort of behaviour do we want to encourage? I want to encourage people to fix issues, implement features, make interesting conference/meetup talks, communicate with the community, etc. So let’s do that! This is basic management: let’s start by publicly acknowledging the contributions of individual community members who make exceptional work.

3 Likes

I have been thinking long and hard on this topic for the last few weeks. Especially after the recent interviews made by @sarina @antoviaque @Dean

It might be a francophone thing, but I totally agree with @regis on this one.

As an Open edX operator with a relatively small team, I realize more and more than 20 hours a month can be onerous. Especially if you have been designated first as a Core Contributor for Translations and second as a Quality Assurance role for testing.

Now that Machine Translations have been put in place and that the language I have been responsible for is almost always around 95% to 100% translated, there is no way I can review strings for up to 20 hours a month. It is definitely much less than that.

I may also have been contributing a lot in terms of testing since Aspen, but there are more and more Core Contributors that have been testing the newest releases and I find myself in a situation where I spend less of my time on testing and just checking that the testing has been done correctly.

As a result, it is clear that I cannot achieve an average of 20 hours per month. I have been seriously thinking about resigning from my Core Contributor role if we go with hours as a measure of evaluation of Core Contributors. I am not there yet.

As an Open edX operator with a relatively small team, I don’t have a ton of ressources available to me to make sure I can understand what is going on in various working groups and understand what needs to be done in order to maintain my own instance of the Open edX ecosystem. Larger teams may be able to have different ressources assigned to different part of the ecosystem. I can’t. There is just me and one other technical ressource on my team.

In the last 2 or 3 years, I have found myself to be more of a Core Observer or a Core Watcher or even a Core Participant. I attend working group meetings where have interests (BTR, Translations, Contributors) in order just to keep up with what is going on because I need to in order to maintain our operations. And because of that I also know that I raise issues on a regular basis. Issues that might be important to small operators as well as big operators. It also allow us to have a voice at the table. But being more of a system administrator than a devops or a developer, it is difficult for me to “contribute” code. I feel that my contribution is at another level. And I will only mention that it took 2 years for a PR I opened on Insights to be reviewed… It didn’t give me much interest in contributing more to be honest.

Bear in mind that I am not looking for any special status based on my history with Open edX. But the current direction of using “time” as a method of evaluating contributors does not sit well with me. I may not need to be a Core Contributor to achieve what I need to do and what I will continue to do in the future. I am not resigning yet, but I may have to.

4 Likes

As far as I know Core Contributors are definitely not officially individually ranked by hours by tCRIL. It’s just used as a common denominator to easily find out if anyone is being idle.

I can understand, therefore, why displaying hours in the Discuss post in a ranked order can be misleading to everyone. As a solution, we could either (A) stop ordering them by Ascending/Descending order (and also remove the names of people who have zero hours), or (B) just go ahead and totally remove the hours from the Discuss posts. I would prefer option A, but I would be okay with Option B too.

Since I currently author those posts, I thought I would address this issue.

Thanks for your answer Dean. I know that you’re doing great work for the Open edX community and I don’t want to disparage that.

If the purpose of the working hours tally is to figure out who is active and who isn’t, could we use instead a binary flag? “Yes I was active and I worked on this and that” vs “I wasn’t active because I was on holiday/busy plowing snow”.

I’m just doing meager non-technical contributions, so really I am just here to help ensure that technical contributors are happy :smiley:. Oh and I don’t get disparaged easily, and do totally enjoy the “democracy” of everything we do!

Wow that’s a neat idea about adding a binary flag. This is something we’ll have to run past @antoviaque @sarina and of course the Listaflow developer @Fox_Piacenti.

This is possible to do, though I think we’ll want to consider the design and purpose of it. There may not need to be a binary flag at all if there’s no particular automation acting upon the information. If someone marks down zero hours and adds a comment about being on vacation, that should be good enough, I’d think, especially since everyone sees the comments in the report.

In just about any case that someone would be reviewing the hours and saying ‘Hey, this person hasn’t contributed hours in a while’, they should be able to see the comment as to why, and evaluate whether that comment fits within expectations/standards (vacations verses ‘Didn’t get around to it’ or what have you).

We are integrating award ceremonies annually at the Open edX conference to award outstanding contributions moving forward. There appears to be a lot of concerns related to the evaluation of quality by the number of hours contributed and I can understand those concerns. It may be better for us to have a moderator for each contribution forum (CC’s, WGs, etc.) to identify which group members have contributed to successful outcomes of the group overall (measurable outcomes) and award people accordingly. This will create an incentive for others to be nominated/awarded and eliminates the feeling of people thinking that they are being “ranked”, based on the hours that they contribute, rather than being rewarded for the quality & engagement in driving valuable results.

2 Likes

Just for context, I manage both the Marketing WG and the Translation WG. For each of these forums, I have selected a community co-organizer, which can be rotated annually. Additionally, Transifex allows me to pull reports on the most active profiles assisting with translations and review of translations. It allows me to see that @sambapete for example, is actively engaged in the management of French-ca, without seeing the number of hours he contributed. This then allows me to conclude that he is highly engaged in ensuring our target completion rates of 90% and above for the group’s selected target languages. Similarly, the Marketing WG has created page content for the Open edX website, that have increased our overall site traffic, which I am able to track in our GA account. Hence, the measurement criteria does not have to be uniform across the board, as long as we can track measurable results and award contributions. The measurement of tracking community performance should always be viewed as positive, without negative connotation. We are tracking because we want to reward, not to deter people through processes of micro-management.

3 Likes

This is my feeling as well. In the name of objectivity, we’re over-simplifying the measurement of value. But we’re losing too much in the simplification. We can still be objective and fair, but if we’re using the same measuring stick everywhere, not both.

2 Likes

First, I’d like to mention that the discussion about work hours is a separate discussion from the original thread topic, which was meant to focus on defining the scope of work as a core contributor. :slight_smile: It might make sense to move it to a separate thread at some point, to allow to conclude the current document review.

Hours as a commitment metric?

I agree with @jill’s points, and I would even say that hours is the worst metric – but to the exclusion of every other. ;p I would love to find a better one, but that’s the only one I know that avoids having to regularly judge the relative merits of different people’s work. It’s a great equalizer – it doesn’t matter whether I think that your work was easier than the one I did, or more valuable, or more rare - these are not comparisons I would be looking forward to make regularly, personally.

If you spend 1h on any task that contributes to the project, I know that it will have been an effort in any case - that’s 1h you didn’t spend on something else! And if we agree on the scope of things that are useful or necessary to the project, then I feel that it is fair to consider it the equal of 1h of my own time, regardless of what the respective tasks were. The comparison is imperfect, but the effort look similar enough to me.

Counting hours an issue?

It’s also relevant to note that, in the interviews, most people were actually fine with the concept of counting or committing hours - the issues were more due to things like not knowing what counts or doesn’t, not knowing that hours can be a rough estimate, not being able to plan well enough, not getting enough time from their organization, etc. And often the discussion that was generated by noticing a low number of hours was very constructive, often leading to finding solutions that allowed to fix the blockers. Without the clear & simple indicator of hours tallies, I don’t think we would have had many of these discussions, as the issues would have been much harder to detect.

The goal of having a metric

It might be useful to remember that ultimately the goal is to get organizations to consistently commit resources to the maintenance of the project. Only a very small %age of the time worked on Open edX projects is contributed to the project itself – it is a long standing core issue which greatly limits the pace of the project.

Having a clear metric is meant to address that tragedy of the commons. The idea is to hold organizations accountable, by tying the ability to influence the project to the amount of resources being committed to the project itself. For better or worse, we are almost (?) all working on the project professionally - and as such, we have to prioritize contribution work in the middle of the rest of our work.

Keeping organizations accountable

Unlike a lot of open source projects, we don’t only have a personal interest in working on Open edX, but also a professional one. And it is also our professional interest to make sure the project that brings us all work keeps going and improving, and to make sure that we all chip in consistently. On the long term, this will be as important, or arguably more important, than any individual client work that we do - it’s just often less urgent, on top of the “someone else will do it” effect. To counter that tragedy of the commons, it’s important to have a mechanism that ensure that all our respective organizations properly prioritize their core contributor’s contribution work in the middle of the rest of our professional obligations, and are held as accountable to that.

In the core contributors interviews we did recently, most people usually wanted to spend more time on core contributor work, but weren’t always being provided with that time by their organization. Ensuring it is provided in a dedicated and quantified way helps securing and prioritizing that time within the context of our organizations. This is the language organizations speak. It helps us as core contributors internally in discussions about priorities, but also externally by allowing the project to escalate to organization leads when an organization doesn’t provide the time it committed for to its core contributors.

As for the volume, if 20h/month is too much @sambapete we could always find a better level - or maybe adapt it to the size of the organization/team to make it lighter on smaller ones, as has been suggested in the past? We haven’t really set a standard yet, 20h/month was an arbitrary pick. What would be the right level of effort?

Hours ranking

For ranking, it’s imho another separate topic, which is more tied to the discussion about leaderboards, etc - if we try to sort out everything at once we’ll likely not be able to decide on anything. :stuck_out_tongue: But in any case, as @Dean mentioned we don’t currently have an official ranking - the report as a whole is a contributed post on the forum, and anyone is free to post their own summary of core contributor activity, as well as alternate metrics or rankings. Each would represent a unique and interesting standpoint, and I don’t think we should forbid to post certain stats or summaries. By definition, any stat will always be a simplified view of reality, but it’s a contribution and a point of view nonetheless. Other ways to present the information are welcomed, and having more diverse perspectives to compare would also allow to experiment, and help evolve them iteratively over time.

1 Like

Sound great and I do agree with your points. However why don’t we have a metric for the other way around. i.e. if an organization is commiting X (hours or whichever metric we chose to), then shouldn’t we also look at the incentive for the orgnization to continue doing so? Things like 1) Ensuring that organization is getting leads/clients or being sustainable and/or 2) prioritizing that organization for blending project.

Speaking transparency, this might not the be the case for every organization speically those which are already established. Probably more for small size organization, like myself where I am the only one in the organization.

Again to be clear I do contribute to the project, because I like to do so in the first the place, to me the Open edX project is of important. But to keep doing so consistently one need to have other incetives.

1 Like

The idea of time tracking is basically a good idea to record the contributions of an Open edX provider (especially when it comes to non-code contributions like testing and/or investigation). On the other hand, Open edX providers who do not actively contribute to the outcomes of the individual working groups can be localized. Even if it is in itself somewhat contradictory to the open source spirit and philosophy to track and control the work of each.

We at Abstract Technology are therefore of the opinion that time tracking is basically a good idea. However, we would like to suggest at this point that time tracking should not be related to individuals, but to companies/service providers. It is clear that a large company can spend many more hours and do it consistently throughout the year, while for a small company it is almost impossible to have 3+ people working on non-project related tasks for a minimum of 20 hours a month. And if they manage to do it then not consistently throughout the year.

We therefore propose to continue time tracking, but to change the standard approach of individual time tracking to company related time tracking. What do others think about it?

I will also tag my colleagues which are also listed as contributor: @Ilaria_Botti @abstract-technology

2 Likes

Wouldn’t that penalize operators who contribute but who have less contributors than large companies? If I understand what you are recommending, a team of 3 and a team of 1 would have a 20 hours a month contribution in both cases?

@sambapete thank you for your reaction.
The snippet you are referring to was more meant as an example. Currently it is like that, everybody has to contribute 20 hours a month, no matter if he/she comes from a big or small company. And here is the difference: A big company can easily manage that one person of the team regularly contributes 20 hours to the community. For a small company it is not easy to manage.

In case the community decides to go ahead with the time tracking we would suggest tracking the time and contribution on a company basis. Each company can decide how to divide the contribution hours between the employees. Currently, this is not the case.

Is it more clear, what I describe?

That’s exactly how I understood it. Thanks @Nicole_Kessler

But we’re still in the situation where a company, or an operator, with a single core contributor would have to contribute 20 hours per month. Right now, I can’t reach it every month.

@sambapete What do you think would be a good level of contribution for you?

As I mentioned above, we haven’t really decided or even discussed on that at all yet – the 20h/month type of commitment came up organically as the most common commitment by organizations, but we can go with something different and more deliberate, based on our experience so far.