In the distant pass, edX/Open edX had a Flaky Test Process. It was written up in the wiki (before the 2U acquisition, and before OEPs existed). The old process page went to the 2U wiki, and was dropped from the Open edX wiki. Basically, we decided that flaky tests should be removed until they can be fixed.
I continue to follow that process, so I made a copy of the old Flaky Test Process page in the Open edX wiki. I believe this is a past decision that should be revived as an OEP, and updated to remove all the 2U specific ticketing, etc.
I agree with this in general, if we can show that a test is flaky, then we should eject it and write up a ticket about it so that we can follow up in the future but we don’t want to slow down development because tests are flaky.
I just want to say I haven’t seen many flaky tests these days. So I think this has worked well!
A much more common problem now is some unknown error happening on github that causes some entire test shards to not run at all, without any specific error message. I don’t know why that happens but it’s now much more common for me than a flaky test.
)