ORA peer evaluation bias - first students who evaluate mark the earliest assignments only

ORA has a serious evaluation bias : when attributing evaluations to peers it’s designed to be a FIFO.

.

this is what our audit of the code by Opencraft @clemente revealed :

  • The query does not order by assessment count
  • It orders by creation time, so it is indeed a FIFO
  • It excludes submissions that are already being graded by someone else, until a lease limit of 8 hours is reached
  • It excludes submissions that are already fully graded

when using peer evaluation “along the way” it’s fine because the first to submit their assignment are the first to be graded.

but when all assignments are submitted “all at a due date” and then there is a peer evaluation phase, the FIFO peer evaluation attribution is biased

=> the first students who peer evaluate, evaluate the earliest assignments only
=> the latest assignment are only evaluated by the students who do their peer evaluate late

This is a big bias in our use case : in order to be fair and impartial peer evaluation should be randomized thus grading is biased

.
;

there seems to be another problem : instead of grading all assignments, latest submitted assignements can be left out : no one grades them

image

@antoviaque @marcotuts @jpdujay @Paul_Humbert @yvain

1 Like

there is also a serious problem with the number of peer evaluations,

asking for 5 peer evaluations, a lot of submissions getting 4 but some get 6 7 or even 8 peer evaluations

Capture d’écran 2020-10-12 093913

This is a serious problem and keep few students in middle to get score due to incomplete peer count.

Great Findings!