Note: creating this thread on behalf of OpenCraft.
One of our clients is having issues where users are consistently forgetting to submit ORA2 assessments. They upload their files and/or save their progress in the text response, but never come back to submit before the due date. We suspect it is because it’s not clear enough that their response will not be recorded unless they click “Submit”.
We would be interested to know if others have had this experience, and if there are any existing proposed changes to the ORA2 UX to help with this? Alternately, what research or changes would edX be interested in pursuing in this direction?
Ideas that we have had so far include:
automatically marking an in-progress response as submitted once the due date passes
allowing multiple submissions, effectively making every saved response and/or uploaded file event become the latest submission. (we researched this and the previous point and determined that these would be very complex to implement correctly)
modifying the UI layout to make the steps required clearer (simple but would require some research to work well)
adding behaviour to alert users that they have unsubmitted in-progress responses and that they will not be recorded until manually submitted. This could be an alert course-wide, or popup after uploading a file or saving a text response. (simplest method, possibly tricky to implement in a non-annoying manner)
add a button for staff to trigger a submit for all students with in-progress responses (unresearched; similar to auto-submitting but takes out the complexities required for auto-launching jobs at certain times)
automatically send email alerts to students when due-dates draw near warning about un-submitted responses (also still in idea phase)
We’ve seen this happen at HarvardX from time to time. If the long-promised notification functionality were part of the platform, this would seem like a perfect use case for them: “You have saved work for a problem at (link). Go there to complete and submit it.”
We’ve definitely had customers with this issue, and it’s often caused problems, not just with deadlines but also with timed exams. Due dates aren’t the only issue! Anything that can lock learner access to a unit based on a condition causes this problem to rear its head.
That said, it’s not common en masse - at least not that I’ve seen. I think there’s three things I’d like to see here.
An option for staff to force submission for a single learner whose submission is saved but not submitted
An option for staff to force submission for all learners with saved progress
An optional to enable auto submission when the deadline passes
I don’t like 3 as a solution, not every in-progress submission should pollute the pool of responses. I can see this being a big problem on larger courses where someone types a single word or space, never comes back and all of a sudden you’ve got hundreds of worthless responses in the way of grading. But 1 and 2 provide sorely needed support tools for instructors.
It’s interesting to hear that this is a common issue. So to continue the conversation, @marcotuts could you weigh in with your thoughts on the various options for solutions? Specifically, which kind of changes would edX be open to accept or would prefer?
Hi Samuel, Thanks for starting this thread. I have some thoughts I’ve tried to group / share below on the topic:
As @MHaton alludes to I don’t know that this is a very common issue, but I still think a quick review to explore ways to reduce this learner issue, even if not very common, is worthwhile!
*Additionally, @colin.fredericks calls out the benefit of notifications as a way to address this, which I agree with as well. Lacking the infrastructure today to make that kind of application trigger / alert we may need to explore additional options in the short term.
Some of the other suggestions proposed for auto-submission, or separate secondary “Unsubmitted” state, or multiple submission support in ORA2, are all interesting to consider and some I think have separate value (as in the case of multiple submission support which was in ORA1 (or maybe we never added it to ORA2 and I just remember making mockups for that in 2012/2013?).
I can submit / share a proposal that builds on the above discussion using some browser hacks of the current ORA problem experience for feedback next.
Below is a list of proposed changes with some visuals, feedback welcome!
A. As a learner who has saved a draft open response assessment submission I would like to ensure my submission is graded.
When a learner has a saved ORA response that is not submitted update the status bar text & color, and add a status alert message to make this state clear.
If a submission has a saved draft, update status bar color from blue to yellow, change the text from “In Progress” to “In Progress - Not Yet Submitted”.
If a submission has a saved draft add a status message to the top of the “Your Response” step as we have with the “Assess Peers” step when you are waiting for responses / assessments. The status message reads: “You have saved a draft response but you have not submitted it. In order to have your submission assessed you must submit your response below.”
A minor change, but the existing message that is shown to the right of the “Save your progress” button can be given a yellow background echoing the status message above and color bar to make this message (somewhat) more visible.
@marcotuts I think alerting the learner more is a good step towards stopping this happening and like this mockup. I wonder if altering the “Save Your Progress” button to “Save Draft” might be a good step? Makes it a little clearer that it’s not a complete submission.
I also wonder if the position/styling of the buttons themselves needs to be changed. The save button is prominent as it’s right below the box. A lot of people will just go for the first button they see and roll with it. while submit is blue and pops more, but is down on the right outside of the prompt area. If they spot it, it’s easy to find, but I’m wondering if it’s disappearing below the fold on smaller screens etc., or just being overlooked.
I’d still like more instructor tools on this though, chiefly the one that allows a single submission to be moved from draft to submitted - no matter what changes are made to improve the UX, there’s always that one guy…
Good question. I made these mockups based on this conversation, so they are not in the ORA pipeline currently. If you think this would solve your clients concerns and feel comfortable using this for that effort perhaps this is a candidate then to move to PDP / the Collaborative Proposals area.
If you think this would solve your clients concerns and feel comfortable using this for that effort perhaps this is a candidate then to move to PDP / the Collaborative Proposals area.