Insights and Tutor incompatibility?

[sambapete (Pierre Mailhot)]

Hi everyone,

Over the week-end we migrated our production environment from Koa Native to Nutmeg Tutor. Everything went fine and we are fixing a few bugs here and there.

Unfortunately, we are facing a brand new Insights issue.

We do generate logs in “raw” format with Tutor though a plugin were
“LOGGING["handlers"]["tracking"]["formatter"] = "raw"”

Unfortunately, since we started pushing the logs to S3 in order to be processed by Insights, there are multiple issues with the daily tasks for enrolment and geolocation plus with displaying the data through the analytics-dashboard. I still have to figure out what is going on with the processing of the daily tasks.

On the analytics dashboard, I now get the following error in my Insights logs when I try accessing a course, any course:

ValueError: time data **‘2020-06-22T07:50:00-04:00’** does not match format **‘%Y-%m-%dT%H:%M:%SZ’**

Did the format change between Native and Tutor? The format does seem ok to me. This did not happen before pushing logs from Tutor.

I am guessing we may be the first to use Tutor and Insights. If not, any ideas what is going on and what you did to resolve that issue if you encountered it?

Thanks for any suggestions.

[sambapete (Pierre Mailhot)]

I think I know what is going on, but right now I don’t know how to solve it. If someone has a quick tip, I would appreciate.

My Insights server is on a different server. I need to access the MySQL database in the docker container on my system running Tutor.

I guess I need to expose a port on my MySQL container, right? But how do I call the MySQL container afterwards?

[sambapete (Pierre Mailhot)]

First, port 3306 wasn’t available on the host. This is now fixed through docker-compose.override.yml

Second, I completely forgot to change the configuration for accessing the openedx database (previously edxapp) on our Insights server. This is now fixed too.

And I can remotely access the MySQL database on my Tutor host from my Insights server.

Now all I have to do is relaunch my daily tasks to see if this fixes the errors I noticed during those tasks.

More to come on Tuesday hopefully.

[sambapete (Pierre Mailhot)]

First, port 3306 wasn’t available on the host. This is now fixed through docker-compose.override.yml

Second, I completely forgot to change the configuration for accessing the openedx database (previously edxapp) on our Insights server. This is now fixed too.

And I can remotely access the MySQL database on my Tutor host from my Insights server.

Now all I have to do is relaunch my daily tasks to see if this fixes the errors I noticed during those tasks.

More to come on Tuesday hopefully.

The daily tasks failed again but that was because the /var/lib/analytics-tasks/analyticstack/repo/override.cfg file still contained a reference to edxapp instead of openedx.

After modification, Insights was able to process tracking logs again, including the ones generated by our new Tutor environment.

Unfortunately, I still couldn’t access the courses in the analytics dashboard. It still complained about ValueError: time data '2019-12-11T19:00:00-05:00' does not match format '%Y-%m-%dT%H:%M:%SZ'

This happens in /edx/app/insights/edx_analytics_dashboard/analytics_dashboard/courses/views/__init__.py when it tries to run

start_date = parse_course_date(self.course_info.get('start'))
end_date = parse_course_date(self.course_info.get('end'))

I couldn’t figure out what was really going on, so I temporarily replaced the instructions with

start_date = datetime.now()
end_date = datetime.now()

This only modifies the display of the start_date and end_date for a course on the analytics dashboard.

I can now access again the analytics contents through the links for each course on the dashboard.

I will need to investigate further what is going on with the dates. I suspect something happened on the way from Koa to Nutmeg.

[jhony_avella]

@sambapete hi. Dates problem could be related to the LMS course API response. Course info is obtained from the API (see line here) and according to the API docs here you should get a date like this one: "2015-06-15T00:00:00Z" which matches the format '%Y-%m-%dT%H:%M:%SZ'. The dates you’re obtaining (for instance '2019-12-11T19:00:00-05:00') match the format "%Y-%m-%dT%H:%M:%S%z". You could check your API response in order to determine the date format you’re getting. Not sure it this could be related as well with a timezone configuration in Django

[sambapete (Pierre Mailhot)]

Hi @jhony_avella

There might a be change somewhere between Koa that we used until last Saturday and Nutmeg that we started using last Saturday.

Since my last post, I have finally handled it by making a few modifications in /edx/app/insights/edx_analytics_dashboard/analytics_dashboard/courses/views/__init__.py

First, I transform the start_date and end_date received from the API.
Second, I also modify the DATETIME_FORMAT being used to compare the date.
Finally, I also modify today’s date to use the same DATETIME_FORMAT with the UTC timezone.

As a result, I am now able to display the correct dates in the course page in Insights.