ORA has a serious evaluation bias : when attributing evaluations to peers it’s designed to be a FIFO.
.
this is what our audit of the code by Opencraft @clemente revealed :
- The query does not order by assessment count
- It orders by creation time, so it is indeed a FIFO
- It excludes submissions that are already being graded by someone else, until a lease limit of 8 hours is reached
- It excludes submissions that are already fully graded
when using peer evaluation “along the way” it’s fine because the first to submit their assignment are the first to be graded.
but when all assignments are submitted “all at a due date” and then there is a peer evaluation phase, the FIFO peer evaluation attribution is biased
=> the first students who peer evaluate, evaluate the earliest assignments only
=> the latest assignment are only evaluated by the students who do their peer evaluate late
This is a big bias in our use case : in order to be fair and impartial peer evaluation should be randomized thus grading is biased
.
;
there seems to be another problem : instead of grading all assignments, latest submitted assignements can be left out : no one grades them
