Introducing the Test Plan for Nutmeg and Beyond

Dear Community,

TESTING IS NOW LIVE! :sunglasses: :slight_smile: :partying_face:

As you know the Build-Test-Release Working Group has been working on a new comprehensive Test Plan for Nutmeg including a list of (up to now) 214 individual tests. It takes the form of a Google Sheet checklist with graphs and a number of supporting tabs including README, PUBLIC SUGGESTIONS BOX, ROADMAP, LEADERBOARD, and finally the NUTMEG TESTS.

The first DRAFT of the Test Plan is now available for public review here: Open edX® Community Test Plan (Main Sheet) - Google Sheets

How To Video

Screen Shot 2022-04-11 at 17.01.44

ACCESSIBILITY: A COMMUNITY FOR EVERYONE
Screen reader support is turned on.
Braille support is turned on.
Screen magnifier support is turned on.

TEST INSTANCE ACCESS: THANKS TO OUR INSTANCE SPONSOR RÉGIS BEHMO, FOUNDER OF TUTOR

  • There is a repository to host the deployment scripts here: GitHub - overhangio/nutmeg-demo: Open edX Nutmeg demo platform CD
  • The demo platform will be deployed weekly on Mondays at 7am UTC and will be available at: https://nutmeg.demo.overhang.io/. This means the database is wiped weekly, i.e. your email address will no longer be in the auth_user table, so you need to re-register your account weekly when the platform redeploys each Monday if your testing is spread across multiple weeks.
  • The credentials for the superuser/staff account are listed in the project readme.
  • Changes to the instance configuration can be made as PRs here.
  • The SMTP service is enabled, but the server is not configured for email authentication (DKIM, SPF, etc.). This means that emails do get sent, but it’s very likely that they will end up in spam, depending on your email client configuration.
  • At the moment, the courseware is most likely not accessible because the demo server is not running the learning MFE yet. The reason for that is that the tutor-mfe plugin needs to be upgraded to Nutmeg, and that is quite a bit of work, which is underway but not completed yet.

TESTING GUIDELINES: HOW TO USE THE SHEET
Step 1: Browse unassigned Test Cases and self-assign from the dropdown list (contact us if your name is not there).
Step 2: When you are ready to perform the test, update the Status column to ‘In Progress’.
Step 3: Perform the test.
Step 4: Update the Status column to ‘Complete’.
Step 5: Update the Result column to ‘Passed’ or ‘Failed’.
Step 6: IF test ‘Failed’, create a GitHub issue and link it in the column. Make sure to add the Test Case ID into GitHub.
Step 7: IF GitHub issue is resolved, go back to Step 2 and start the testing process again.

DOS AND DONTS: LET’S KEEP IT NEAT AND ERROR FREE
Testing should happen on the official instance: https://nutmeg.demo.overhang.io/
Only assign yourself tasks that you will complete within 24 hours. This is to avoid blocking the pipeline for others.
Do not edit this Main Sheet, instead add your suggestions to the SUGGESTIONS BOX.
If you test a component in Studio, please check it’s working in the LMS view too.
Most of the sheets are PROTECTED meaning you cannot alter them, apart from a specially selected range that you will be using as tester.

Do you want to join the test volunteers and help test Nutmeg? Contact us or reply to this post.

Test Coordinator: dean@abconlinecourses.com
Test Instance Owner: regis@behmo.com
QA Manager: pierre.mailhot@edulib.org

The current test volunteers include, but are not limited to:

Ned Bachelder (2U)
Kaleb Abebe (MIT Open Learning)
Régis Behmo (Tutor)
Pierre Mailhot (EDUlib)
Peter Pinch (MIT Open Learning)
Brian Grossman (MITx xPRO)
Ghassan Maslami (independent)
Maria Grimaldi (eduNEXT)
Kyle McCormick (tCRIL)
Zia Fazal (Arbisoft)
Fayyaz Ahmed (Edly)
Gabriel D’Amours (OpenCraft)
JayRam Nai (DRC Systems)
Dean Jay Mathew (ABC Online Courses)
Aloka Wimalarathna (ABC Online Courses)
Nuanpunmorakot Wiphatsumrit (ABC Online Courses)
Isanka Wijerathne (ABC Online Courses)

9 Likes

Hi @Dean
I want to join in as test volunteer for Nutmeg

1 Like

Hi @Zia_Fazal

Thanks and welcome to the test team! I’m excited to have you join us. You may participate in any way that is convenient to you.

I added you to the Slack Buid-Test-Release Working Group so you will see if we post something related to testing.

I will personally keep you in the loop so you will be included.

If there is any feature area or plugin you want to focus on, just let me know?

You can also add your suggestions in the SUGGESTIONS BOX tab in the same shared Google Sheet shared in my original post that is open for public review and commentary.

Your name is now also showing up in the dropdown menu for self-assigning test cases:

Thanks again for joining the team!

1 Like

@Dean Thanks for quick help. I’m intrested in testing web certificates and xAPI/caliper support. However, I’ll assign other features to myself as appropriate.

Hi @Dean
I want to join in as test volunteer for Nutmeg

1 Like

Hi @Fayyaz_Ahmed

Sure thing! Welcome to the test team!

Please join the Slack Buid-Test-Release Working Group to keep up to date with the testing communications. I will also keep you in the loop so you will be included.

If there is any feature area or plugin you want to focus on?

You can also add your suggestions in the SUGGESTIONS BOX tab in the same shared Google Sheet shared in my original post that is open for public review and commentary.

Your name is now also showing up in the dropdown menu for self-assigning test cases:

1 Like

@Dean Thanks for adding me as a test volunteer. I’m interested in testing the whole system on behalf of Edly. I’ll assign features to myself accordingly.

1 Like

Sounds good.

Certificates: there are already a few available test cases (we could potentially add some more test cases around certificates to be more thorough). Feel free to suggest.

xAPI/caliper: we have not added test cases around xAPI/caliper yet, so you could take the lead to suggest some test cases to be added to the sheet, or wait for it to be added.

With regards.

Hi @Zia_Fazal

I’m going to add xAPI/Caliper support to the sheet.

My question is, what “location” category would you like me to add it to? How about something like STUDIO: EXTENSIONS: XAPI/CALIPER

I’ll probably also add LTI, SCORM, HP5 and some others during this round.

At the moment we have the following categories:

USER ROLE STUDIO LMS ADMIN INTEGRATIONS
DevOps STUDIO: HOME LMS: THEME ADMIN: DJANGO ADMIN PORTAL AWS: CUSTOM VIDEO UPLOAD MODULE
Author STUDIO: MAINTENANCE LMS: COURSE DISCOVERY
Instructor STUDIO: CONTENT: OUTLINE LMS: AUTHENTICATION
Learner STUDIO: CONTENT: OUTLINE: UNIT LMS: AUTHENTICATION: SSO
Beta Testers STUDIO XBLOCKS LMS: DASHBOARD
Course Data Researcher STUDIO: CONTENT: UPDATES LMS: PROFILE
Discussion Admins STUDIO: CONTENT: PAGES LMS: ACCOUNT
Discussion Moderators STUDIO: CONTENT: FILES & UPLOADS LMS: COURSE ABOUT PAGE
Group Community TA STUDIO: CONTENT: FILES & UPLOADS LMS: COURSE HOME
Community TA STUDIO: CONTENT: TEXTBOOKS LMS: COURSE HOME: TOOLS
STUDIO: SETTINGS: SCHEDULE & DETAILS LMS: COURSE HOME: PROGRESS
STUDIO: SETTINGS: GRADING LMS: COURSE HOME: DISCUSSION
STUDIO: SETTINGS: COURSE TEAM LMS: COURSE HOME: TEAMS
STUDIO: SETTINGS: GROUP CONFIG LMS: MULTILINGUAL
STUDIO: SETTINGS: ADVANCED LMS: INSTRUCTOR
STUDIO: SETTINGS: CERTIFICATES LMS: INSTRUCTOR: COURSE INFO
STUDIO: TOOLS: IMPORT LMS: INSTRUCTOR: MEMBERSHIP
STUDIO: TOOLS: EXPORT LMS: INSTRUCTOR: COHORTS
STUDIO: TOOLS: CHECKLIST LMS: INSTRUCTOR: EXTENSIONS
LMS: INSTRUCTOR: STUDENT ADMIN
LMS: INSTRUCTOR: DATA DOWNLOAD
LMS: INSTRUCTOR: EMAIL
LMS: INSTRUCTOR: ANALYTICS

Your point of view is very welcome if you want to suggest a category.

General Update:

It has been exactly one (1) week since the draft Test Plan (for Nutmeg and beyond) was released for public comments and suggestions, so I thought I would update everyone on the progress.

Most importantly, please note that testing has NOT started yet, so don’t feel like you’re missing out on anything. This is because we are still preparing the test instance, which as you know takes a lot of configuration and upgrading of multiple apps/MFEs to work in Nutmeg.

So, what can you do now while we wait for the test instance to be ready? If you’re rearing to go, you can read through the test cases and see if there are test cases we have not added yet. There are many still to add, such as xAPI/Caliper, SCORM, LTI, and other test cases. Feel free to add your suggestions to the SUGGESTIONS tab in the sheet.

1. Test Instance Status

The README sheet has been updated as follows:

  • The demo platform will now be deployed WEEKLY instead of daily, on Mondays at 7am UTC. This means the database is wiped weekly, i.e. your email address will no longer be in the auth_user table, so you need to re-register your account weekly when the platform redeploys each Monday if your testing is spread across multiple weeks.

  • The credentials for the superuser/staff account are listed in the project readme.

  • The SMTP service is enabled, but the server is not configured for email authentication (DKIM, SPF, etc.). This means that emails do get sent, but it’s very likely that they will end up in spam, depending on your email client configuration.

  • At the moment, the courseware is most likely not accessible because the demo server is not running the learning MFE yet. The reason for that is that the tutor-mfe plugin needs to be upgraded to Nutmeg, and that is quite a bit of work, which is underway but not completed yet.

2. Test Team volunteers

At least four (4) community members have joined the new test team since posting about it! And I must say they are all truly Open edX® experts so we’re lucky to have them on board. We welcome:

@gabrieldamours from OpenCraft
@Zia_Fazal from Arbisoft
@mgmdi from eduNEXT
@Fayyaz_Ahmed from Edly by Arbisoft
@Brian_Grossman from MITx xPro

The whole team so far is sixteen (16) people, in no particular order:

Ned Bachelder (2U)
Kaleb Abebe (MIT Open Learning)
Régis Behmo (Tutor)
Pierre Mailhot (EDUlib)
Peter Pinch (MIT Open Learning)
Brian Grossman (MITx xPRO)
Ghassan Maslami (independent)
Maria Grimaldi (eduNEXT)
Kyle McCormick (tCRIL)
Zia Fazal (Arbisoft)
Fayyaz Ahmed (Edly)
Gabriel D’Amours (OpenCraft)
Dean Jay Mathew (ABC Online Courses)
Aloka Wimalarathna (ABC Online Courses)
Nuanpunmorakot Wiphatsumrit (ABC Online Courses)
Isanka Wijerathne (ABC Online Courses)

We also give thanks to @sarina and @antoviaque who help us in so many ways.

@Dean I think instead of studio that extension is installed in LMS so LMS: EXTENSIONS: XAPI/CALIPER would more appropriate category for that extension.

1 Like

This has been added to the sheet with ID of TC_AUTHOR_149.


I placed it under the “author” role. If you think that’s wrong, we can choose another role such as DevOps, or add a new role not covered by my list:

DevOps
Global Admin
Author
Instructor
Learner
Beta Testers
Course Data Researcher
Discussion Admins
Discussion Moderators
Group Community TA
Community TA

A warm welcome to the talented @jramnai who joins the volunteer Test Team… and… he is the first person to report a bug even though testing has not officially started yet. Thank you Jay!

3 Likes

TESTING IS NOW OPEN!

Watch the How To Video here:

Screen Shot 2022-04-11 at 17.01.44

Fayyaz and Kaleb have been fast out the starting blocks and are racing ahead on our fun leaderboard!

Remember it’s not a race, but there is a finish line and we will all cross it together! So let’s do it!

4 Likes

Hi @Dean,
I am not able to join the slack channel. Kindly add my user fayyaz.ahmed@edly.io to the slack channel.

Hi @Fayyaz_Ahmed,

Great going on Nutmeg testing, everyone was super impressed when I updated the BTR Working Group yesterday.

@nedbat I think you’re the right person to add people to Slack, please correct me if I’m wrong. Can you please add Fayyaz to Slack?

Hi @Fayyaz_Ahmed

Have you tried inviting yourself to Slack? Here is the link: Join the Open edX community on Slack!

1 Like

@Dean,
Thanks. Invited my user.

Nutmeg Testing is coming to a close in 7 days.

For the latest updates please check the Slack Channel: #wg-build-test-release

https://openedx.slack.com/archives/C01AGTSB1LL/p1653561876185169

Update on Nutmeg Testing

  • Due Date for Testing: 1 June 2022 24 hours remaining.

  • Status: 90.%

  • Release date: 9 June 2022

We are now moving into the final 24 hours of the 1 month-long testing sprint we have just undertaken.Tomorrow I’ll summarise what we have achieved so far, and list what’s outstanding which will be our focus up until the official release date of 9 June 2022 .Take some rest tonight and revel in what you have achieved over this past month. Thank you all!