As you know the Build-Test-Release Working Group has been working on a new comprehensive Test Plan for Nutmeg including a list of (up to now) 214 individual tests. It takes the form of a Google Sheet checklist with graphs and a number of supporting tabs including README, PUBLIC SUGGESTIONS BOX, ROADMAP, LEADERBOARD, and finally the NUTMEG TESTS.
The demo platform will be deployed weekly on Mondays at 7am UTC and will be available at: https://nutmeg.demo.overhang.io/. This means the database is wiped weekly, i.e. your email address will no longer be in the auth_user table, so you need to re-register your account weekly when the platform redeploys each Monday if your testing is spread across multiple weeks.
The credentials for the superuser/staff account are listed in the project readme.
The SMTP service is enabled, but the server is not configured for email authentication (DKIM, SPF, etc.). This means that emails do get sent, but itās very likely that they will end up in spam, depending on your email client configuration.
At the moment, the courseware is most likely not accessible because the demo server is not running the learning MFE yet. The reason for that is that the tutor-mfe plugin needs to be upgraded to Nutmeg, and that is quite a bit of work, which is underway but not completed yet.
TESTING GUIDELINES: HOW TO USE THE SHEET
Step 1: Browse unassigned Test Cases and self-assign from the dropdown list (contact us if your name is not there).
Step 2: When you are ready to perform the test, update the Status column to āIn Progressā.
Step 3: Perform the test.
Step 4: Update the Status column to āCompleteā.
Step 5: Update the Result column to āPassedā or āFailedā.
Step 6: IF test āFailedā, create a GitHub issue and link it in the column. Make sure to add the Test Case ID into GitHub.
Step 7: IF GitHub issue is resolved, go back to Step 2 and start the testing process again.
DOS AND DONTS: LETāS KEEP IT NEAT AND ERROR FREE
Testing should happen on the official instance: https://nutmeg.demo.overhang.io/
Only assign yourself tasks that you will complete within 24 hours. This is to avoid blocking the pipeline for others.
Do not edit this Main Sheet, instead add your suggestions to the SUGGESTIONS BOX.
If you test a component in Studio, please check itās working in the LMS view too.
Most of the sheets are PROTECTED meaning you cannot alter them, apart from a specially selected range that you will be using as tester.
Do you want to join the test volunteers and help test Nutmeg? Contact us or reply to this post.
Thanks and welcome to the test team! Iām excited to have you join us. You may participate in any way that is convenient to you.
I added you to the Slack Buid-Test-Release Working Group so you will see if we post something related to testing.
I will personally keep you in the loop so you will be included.
If there is any feature area or plugin you want to focus on, just let me know?
You can also add your suggestions in the SUGGESTIONS BOX tab in the same shared Google Sheet shared in my original post that is open for public review and commentary.
Your name is now also showing up in the dropdown menu for self-assigning test cases:
@Dean Thanks for quick help. Iām intrested in testing web certificates and xAPI/caliper support. However, Iāll assign other features to myself as appropriate.
Please join the Slack Buid-Test-Release Working Group to keep up to date with the testing communications. I will also keep you in the loop so you will be included.
If there is any feature area or plugin you want to focus on?
You can also add your suggestions in the SUGGESTIONS BOX tab in the same shared Google Sheet shared in my original post that is open for public review and commentary.
Your name is now also showing up in the dropdown menu for self-assigning test cases:
@Dean Thanks for adding me as a test volunteer. Iām interested in testing the whole system on behalf of Edly. Iāll assign features to myself accordingly.
Certificates: there are already a few available test cases (we could potentially add some more test cases around certificates to be more thorough). Feel free to suggest.
xAPI/caliper: we have not added test cases around xAPI/caliper yet, so you could take the lead to suggest some test cases to be added to the sheet, or wait for it to be added.
It has been exactly one (1) week since the draft Test Plan (for Nutmeg and beyond) was released for public comments and suggestions, so I thought I would update everyone on the progress.
Most importantly, please note that testing has NOT started yet, so donāt feel like youāre missing out on anything. This is because we are still preparing the test instance, which as you know takes a lot of configuration and upgrading of multiple apps/MFEs to work in Nutmeg.
So, what can you do now while we wait for the test instance to be ready? If youāre rearing to go, you can read through the test cases and see if there are test cases we have not added yet. There are many still to add, such as xAPI/Caliper, SCORM, LTI, and other test cases. Feel free to add your suggestions to the SUGGESTIONS tab in the sheet.
The demo platform will now be deployed WEEKLY instead of daily, on Mondays at 7am UTC. This means the database is wiped weekly, i.e. your email address will no longer be in the auth_user table, so you need to re-register your account weekly when the platform redeploys each Monday if your testing is spread across multiple weeks.
The credentials for the superuser/staff account are listed in the project readme.
The SMTP service is enabled, but the server is not configured for email authentication (DKIM, SPF, etc.). This means that emails do get sent, but itās very likely that they will end up in spam, depending on your email client configuration.
At the moment, the courseware is most likely not accessible because the demo server is not running the learning MFE yet. The reason for that is that the tutor-mfe plugin needs to be upgraded to Nutmeg, and that is quite a bit of work, which is underway but not completed yet.
2. Test Team volunteers
At least four (4) community members have joined the new test team since posting about it! And I must say they are all truly Open edXĀ® experts so weāre lucky to have them on board. We welcome:
@Dean I think instead of studio that extension is installed in LMS so LMS: EXTENSIONS: XAPI/CALIPER would more appropriate category for that extension.
I placed it under the āauthorā role. If you think thatās wrong, we can choose another role such as DevOps, or add a new role not covered by my list:
DevOps
Global Admin
Author
Instructor
Learner
Beta Testers
Course Data Researcher
Discussion Admins
Discussion Moderators
Group Community TA
Community TA
A warm welcome to the talented @jramnai who joins the volunteer Test Teamā¦ andā¦ he is the first person to report a bug even though testing has not officially started yet. Thank you Jay!
Due Date for Testing: 1 June 202224 hours remaining.
Status: 90.%
Release date: 9 June 2022
We are now moving into the final 24 hours of the 1 month-long testing sprint we have just undertaken.Tomorrow Iāll summarise what we have achieved so far, and list whatās outstanding which will be our focus up until the official release date of 9 June 2022 .Take some rest tonight and revel in what you have achieved over this past month. Thank you all!
