Issues with Enrollment and Course Mode Data Accuracy in Aspect

Site Operators Tutor Help
, , ,
1

Hi everyone,

We’re currently experiencing some issues with data accuracy in Aspect. Specifically:

  1. When a user enrolls in a course, the enrollment count is not increasing in Aspect.

  2. Additionally, we’re not receiving accurate data regarding the course mode (e.g., audit, verified).

We’re using the following repository: GitHub - openedx/platform-plugin-aspects: Aspects plugins for the Open edX LMS and Studio

Has anyone else encountered similar issues with Aspect? If so, how did you resolve them?
Or does anyone have suggestions on how we can troubleshoot or fix this?

2

@Sara_Burns @TyHob any suggestions?

3

Hi @Ronak_Pansuriya1 . There are a few possible causes that may be impacting you, but it depends on a number of factors. If you are running Aspects with the default settings you should be able to see if there are errors in the logs for the LMS or CMS worker containers, or in the Ralph container. This would be the case where events aren’t making it into the database. If you are using one of the other methods of moving events (Vector or event bus) you would need to look at those containers instead.

If the problem is that there are errors in those logs, you can replay the tracking logs to try to catch the missing events. If that doesn’t help you find the issues, I can try to help more, but I’ll have a lot of questions. :slightly_smiling_face: It would be good to review this document to make sure things are set up correctly as well.

4

Thank you for your guidance, @TyHob.

I’ve identified the issue.

When we add a new user through the Instructor Dashboard, the first user is reflected in Superset. However, when we try to add a second user, they are not reflected in Superset.

On the other hand, when users enroll by themselves, their data is correctly reflected in Superset.

Do you have any idea what might be causing this issue?
Could you please check on your end whether this is a setup issue on our side or a potential bug?

Thank you in advance.

5

I can’t think of a reason why the first user would work, but the second not work. However, it you are enrolling users by email address and they don’t have an account yet I think there may be a race condition when they create an account and are automatically enrolled in the course. The enrollment event gets fired before the MySQL transaction creating the user is committed, so when the job to send the event to ClickHouse runs, it errors.

In this case (in the default configuration) you would see errors in the lms-worker like “User does not exist”. It would be great to know if you’re seeing that so we can work on a fix.

6

Thank you for the explanation.

In our case, the users already exist on the platform. We are trying to add users who already have accounts, so it’s not a case where the user is being created at the time of enrollment.

We also tested this behavior across multiple courses and different projects, and we’re consistently facing the same issue — the second user added through the Instructor Dashboard does not get reflected in Superset.

and I haven’t see any errors in the lms-worker

1 Like
7

Interesting. Are you enrolling by username or email address? Are you using the “Auto Enroll” and / or “Notify users by email” checkboxes?

8

I think I’ve found the issue. Events are being triggered for the enrollment, but they are misattributed to the instructor / admin who added them in the UI. I’ll open an issue for it, but it may take a little while to fix since this area of the code is very complicated.

We may, instead, release a bug fix version that uses the event_sink enrollments table, which seems to be more reliable.

10

I’m happy to help resolve this issue if you can guide me on where to look and how I might be able to fix the bug.

11

In fact, it’s somewhat more complicated. The enrollment doesn’t happen until the user activates their account, so if you are enrolling users who exist but haven’t activated the account (for example by clicking the email link) the enrollment won’t show up until they do.

The thing I’m seeing is that when I force activated my test accounts in the Django admin it sent the tracking event as the logged in user (my admin account) instead of the test accounts. Do you know if the accounts you are testing with are definitely active?

12

Yes, I confirm that all my test accounts are active. I also tried this across different projects and courses, but the issue remains the same.

13

Got it, thank you! I’ve confirmed that it still sends the event as the instructor / admin user instead of the student being enrolled in the case where the user is already active. I am writing up a bug for it now and will post a link here once that’s done. It will take a little while to dig through this code path and figure out where the fix may need to go.

14

Thank you @TyHob
Let me know if I can help you somehow.

15

Thanks @Ronak_Pansuriya1 , here’s the issue: Enrollment events can create a statement for the wrong user · Issue #508 · openedx/event-routing-backends · GitHub . I’ve put all of the information I can think of in there, and would be happy to help out if you’d like to take on this task. Otherwise we will try to prioritize this work in the next couple of weeks.

As I mentioned above, we are also looking at moving away from the xAPI events for most enrollment metrics going forward, but that is a much larger task and we’re not sure if/when that work will be prioritized. We’d still want to fix this anyway.

2 Likes
16

@TyHob I fixed the issue by overriding extract_username_or_userid in the BaseEnrollmentTransformer, and it’s now working as expected.

Now, my question is: how can I test older tracking log events that may have different structures?
Is there any way to handle older tracking logs so we can see accurate data in Aspects?

If you have any guidance on this, it would be helpful to me.

17

That’s great! I think that is the safest change to make right now. We don’t have samples of older versions of events to test against unfortunately, someone with old tracking logs would have to go back and compare their earliest events to what we currently have to see if there are differences. In most cases the event docs have notes when events were added/removed/changed and I’m not seeing that for these events.

I think for the change you made it makes the most sense to check for the event.user_id first, then fall back to the way it was checking before with get_data at the top level, which will recursively search until if finds something. Based on the git history it looks like user_id has been in the event data for at least 12 years.

18

Yes, I have old tracking logs, but I’m not sure how to test them or how to replay that data. Do you have any ideas or suggestions?

Also, we have another situation: we’ve customized many things on the registration page, and we want to send that custom data and view it in Aspect. As I understand it, we need to create a custom signal and a custom event for that, right?

Additionally, we have a lot of old data, how can we move that data into Aspect as well?

Could you please guide or suggest the best approach?

19

Docs for running backfills are here and here. It can be a complicated thing to configure. Also please note that by default Aspects is configured to keep 1 year of data for performance and privacy reasons, so if you wish to use a lot of older data you will need to familiarize yourself with this document.

There are several ways to bring custom data into Aspects with various tradeoffs in terms of performance and complexity. Since it’s a very unique decision for each site, and separate from the topic of this question, can we either whittle down the problem space in Slack or start a new topic here for that?

20

I saw that the docs suggest using:

tutor [dev|local|k8s] do transform-tracking-logs --options “…”

But I am not sure exactly what should be inside the --options when I am trying to load a simple local tracking.log file. I tried various combinations but ran into errors about prefix and provider.

Can you provide an example for a local run of this command?

What are the different ways we can bring custom data into Aspects? Could you please guide or suggest the best approach we could use?

Also, should I create a new topic for this, or would you prefer to continue the discussion on Slack?

21

It looks like that doc is out of date, there is a simpler form now. You can see all of the options with:

tutor local do transform-tracking-logs --help

A working example is:

tutor local do transform-tracking-logs --source_provider LOCAL --source_config '{"key": "/openedx/data/", "prefix": "tracking.log", "container": "logs"}' --destination_provider LRS --transformer_type xapi --dry_run

That example will run against the local tracking log file in Tutor local, but just as a test to make sure the file can be found. Nothing will be sent to Ralph or ClickHouse until you remove the --dry_run option.

It’s very likely that your Tutor LMS worker image doesn’t mount the logs directory and only an empty file will be found. I’m hoping to address this with the Tutor maintainers soon, but in the meantime you can temporarily edit this file:

$TUTOR_ROOT/env/local/docker-compose.jobs.yml

To add this line to the lms-job → volumes key right above the depends_on line. Then simply run the command I gave you above and it should find the file. This change will be overwritten the next time you tutor config save.

With --dry_run on you should see a bunch of standard Django warnings then output like:

Found 1 files in logs/tracking.log*
Found 1 source files:
tracking.log - 61888649 bytes
Looking for log files in logs/tracking.log*
Streaming file <Object: name=tracking.log, size=61888649, hash=90ced65510b0c9eb7b1be588e1bed9c0, provider=Local Storage ...>...
Dry run, skipping, but still clearing the queue.
...
Dry run, skipping, but still clearing the queue.
Finalizing 180 events to LRS
Dry run, skipping final storage.
Queued 100180 log lines, could not parse 0 log lines, skipped 5372 log lines, sent 0 batches.

Without --dry_run if will print many, many statements to the screen by default, but the final output will look like:

Queued 100180 log lines, could not parse 0 log lines, skipped 5372 log lines, sent 11 batches.

I’ll need to ask you a bunch of questions to try to understand your needs for getting custom data. Please contact me on Slack and then we can move the conversation back here if that makes sense.
The short version is that some options are:

  • Override the default xAPI statements to include more data, if that makes sense for your case
  • Create new xAPI statements and dbt transforms if the data can be represented as xAPI or you want to use the LRS functionality for this data
  • Make your MySQL data queryable and either call it directly in Superset or make the data available to ClickHouse

The first two options are outlines here and here. The last one has some potential tradeoffs and performance implications on the LMS.

1 Like