# ORA peer evaluation bias - first students who evaluate mark the earliest assignments only

**URL:** <https://discuss.openedx.org/t/ora-peer-evaluation-bias-first-students-who-evaluate-mark-the-earliest-assignments-only/3367>\
**Category:** Collaborative Proposals\
**Created:** [October 7, 2020, 8:44am UTC](https://discuss.openedx.org/t/ora-peer-evaluation-bias-first-students-who-evaluate-mark-the-earliest-assignments-only/3367 "2020-10-07T08:44:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Remi\_Bachelet](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/remi_bachelet/32/507_2.png) [@Remi\_Bachelet](https://discuss.openedx.org/u/Remi_Bachelet)\
**Post date:** [October 7, 2020, 8:44am UTC](https://discuss.openedx.org/t/ora-peer-evaluation-bias-first-students-who-evaluate-mark-the-earliest-assignments-only/3367/1 "2020-10-07T08:44:49Z")

</div>

ORA has a serious evaluation bias : when attributing evaluations to peers it’s designed to be a FIFO.

.

## this is what our audit of the code by Opencraft @clemente revealed :

- The query does not order by assessment count
- It orders by creation time, so it is indeed a FIFO
- It excludes submissions that are already being graded by someone else, until a lease limit of 8 hours is reached
- It excludes submissions that are already fully graded

* * *

when using peer evaluation “along the way” it’s fine because the first to submit their assignment are the first to be graded.

but when all assignments are submitted “all at a due date” and then there is a peer evaluation phase, the FIFO peer evaluation attribution is biased

=\> the first students who peer evaluate, evaluate the earliest assignments only  
=\> the latest assignment are only evaluated by the students who do their peer evaluate late

This is a big bias in our use case : **in order to be fair and impartial peer evaluation should be randomized** thus grading is biased

.  
;

_there seems to be another problem : instead of grading all assignments, latest submitted assignements can be left out : no one grades them_

![image](https://us1.discourse-cdn.com/flex020/uploads/openedx/original/2X/2/22c045c50be8f1a813b6fb6cae1f739f3205b4d5.jpeg)

@antoviaque @marcotuts @jpdujay @Paul_Humbert@yvain

---

<div class="post-metadata">

**Author:** ![Remi\_Bachelet](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/remi_bachelet/32/507_2.png) [@Remi\_Bachelet](https://discuss.openedx.org/u/Remi_Bachelet)\
**Post date:** [October 12, 2020, 7:56am UTC](https://discuss.openedx.org/t/ora-peer-evaluation-bias-first-students-who-evaluate-mark-the-earliest-assignments-only/3367/2 "2020-10-12T07:56:09Z")

</div>

there is also a serious problem with the number of peer evaluations,

asking for 5 peer evaluations, a lot of submissions getting 4 but some get 6 7 or even 8 peer evaluations

![Capture d’écran 2020-10-12 093913](https://us1.discourse-cdn.com/flex020/uploads/openedx/original/2X/3/3a9b6a5ef2396afee4e6ec9bc8469469beebcb9b.png)

---

<div class="post-metadata">

**Author:** ![Manish\_Hada](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.openedx.org/manish_hada/32/1651_2.png) [@Manish\_Hada](https://discuss.openedx.org/u/Manish_Hada)\
**Post date:** [October 13, 2020, 7:38am UTC](https://discuss.openedx.org/t/ora-peer-evaluation-bias-first-students-who-evaluate-mark-the-earliest-assignments-only/3367/3 "2020-10-13T07:38:27Z")

</div>

This is a serious problem and keep few students in middle to get score due to incomplete peer count.

Great Findings!