Errors with superset and aspect

[mboisson (Maxime Boissonneault)]

Hi,

I am getting various errors with Superset and Aspect.

First, when I try to login to

https://superset.edx.

I get the error

Invalid client_id parameter value.

This used to work in a previous version (this is with Ulmo).

Then, in the instructor dashboard, if I try to dig through the At risk learners dashboard, engagement tab, and problems subtab, in the problem attempts section, I get an unexpected error with

Error: Received ClickHouse exception, code: 47, server response: Code: 47. DB::Exception: Unknown expression or table expression identifier `course_name` in scope SELECT problem_link AS problem_link_8b6c25, multiIf(graded = true, 'graded', 'ungraded') AS Graded_0bb27c, AVG(avg_correct_attempts_coursewide) AS avg_correct_attemps_coursewide_a34caf, AVG(correct_attempts_by_learner) AS correct_attempts_by_learner_f82422, AVG(incorrect_attempts_by_learner) AS incorrect_attempts_by_learner_b345d4 FROM (SELECT org, course_key, course_order, problem_link, graded, actor_id, avg_correct_attempts_coursewide, avg_incorrect_attempts_coursewide, coursewide_percent_correct, correct_attempts_by_learner, incorrect_attempts_by_learner, selected_learner_percent_correct, selected_learner_percent_incorrect FROM reporting.dim_problem_coursewide_avg INNER JOIN (WITH page_visits AS (SELECT org, course_key, actor_id, emission_time AS last_visited FROM reporting.dim_learner_last_course_visit WHERE (1 = 1) AND (course_key IN (SELECT course_key FROM event_sink.dim_course_names WHERE course_name IN ('Premiers pas sur les superordinateurs'))) AND (emission_time < subtractDays(now(), 7))) SELECT org, course_key, learners.actor_id AS actor_id FROM reporting.dim_student_status AS learners INNER JOIN page_visits USING (org, course_key, actor_id) WHERE (approving_state = 'failed') AND (enrollment_status = 'registered') AND (course_key IN (SELECT course_key FROM event_sink.dim_course_names WHERE course_name IN ('Premiers pas sur les superordinateurs')))) AS at_risk_learners USING (org, course_key, actor_id)) AS virtual_table WHERE (course_name IN ('Premiers pas sur les superordinateurs')) AND (org = 'CQ') AND (course_key = 'course-v1:CQ+CIP101+v1') GROUP BY problem_link, multiIf(graded = true, 'graded', 'ungraded') ORDER BY sum(course_order) ASC LIMIT 1000. Maybe you meant: ['course_order']. (UNKNOWN_IDENTIFIER) (version 25.8.16.34 (official build)) (for url http://clickhouse:8123)

Something is likely incorrectly configured, but I don’t know what. Any idea ?

This is using aspects 2.5.1, installed through tutor-contrib-aspects

[TyHob]

Hi @mboisson did you go through all of the upgrade steps when you did the upgrade? If so we can try to help troubleshoot from there, but usually that fixes things.

[mboisson (Maxime Boissonneault)]

I think I did, but I can run them again. Is there any issue with running them multiple times ?

Also, I stayed on 2.5.1 because I didn’t want to do a major upgrade to 3.x just yet. Any reason I should hold back to 2.5.1 ?

[mboisson (Maxime Boissonneault)]

Mmm, I see that 2.x is not compatible with Ulmo, I guess I’ll upgrade to 3.0.3.

[mboisson (Maxime Boissonneault)]

Ok, so I have upgraded to 3.0.3:

Mar 11 21:40:04 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Tutor::Plugin[aspects]/Tutor::Plugin_dep[aspects]/Package[tutor-contrib-aspects]/ensure) ensure changed '3.0.2' to '3.0.3' (corrective)
Mar 11 21:40:04 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[rm -rf /tutor/.local/share/tutor/env/plugins/aspects/build/aspects-superset/openedx-assets/assets]) Triggered 'refresh' from 1 event
Mar 11 21:40:12 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[tutor config save]) Triggered 'refresh' from 1 event
Mar 11 21:58:08 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Tutor::Plugin[aspects]/Exec[tutor images rebuild openedx aspects aspects-superset --no-cache]) Triggered 'refresh' from 1 event
Mar 11 22:10:33 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Tutor::Plugin[aspects]/Exec[tutor images rebuild mfe --no-cache]) Triggered 'refresh' from 1 event
Mar 11 22:13:11 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[tutor local do init -l aspects]) Triggered 'refresh' from 1 event
Mar 11 22:13:29 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[tutor local do dbt -c "run-operation remove_deprecated_models" --only_changed False]) Triggered 'refresh' from 1 event
Mar 11 22:13:56 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[tutor local start --detach]/returns) executed successfully (corrective)
Mar 11 22:14:17 edx1.int.evolo-dev.calculquebec.cloud puppet-agent[1113734]: (/Stage[main]/Tutor/Exec[tutor local reboot --detach]) Triggered 'refresh' from 2 events
Mar 1

I then reconnected, and I still get

when trying to log to superset.

I now get a different error when looking at “Problem Engagement per Section/Subsection”:

[Sara_Burns]

Hi @mboisson - you can run this and it should fix that error:

tutor [local/dev/k8s] do dbt -c 'run --full-refresh --select fact_problem_engagement' --only_changed False

[mboisson (Maxime Boissonneault)]

Ok, that fixed the error in the “Problem Engagement” page. But it did not fix the “invalid client_id” when trying to log to superset.

I see from the logs that the client_id in the URL does not match any of the client IDs I find in the admin/oauth2_provider/application/ page

I suspect that the client IDs got messed up somehow, maybe they got reinitialized, but got overwritten when I restored a database backup.

How can I reinitialize the oauth link between superset and openedx ?

[TyHob]

Your Tutor config.yml should have the SUPERSET_OAUTH2_CLIENT_* values that you need. You can update the values in the LMS Django admin admin/oauth2_provider/application/ to match.

The dev row and values are only for running in tutor dev so you should just need to update the superset-sso entry. If your hostname has changed you will also need to update the Redirect uris setting, sometimes people run into this when puling production data back to a staging environment.

[mboisson (Maxime Boissonneault)]

I actually followed Troubleshoot Aspects — Open edX Aspects latest documentation and delete the superset-sso and supperset-sso-dev applications and reinitialized aspects.

That changed the error from

“Invalid client_id” to

“Erreur: invalid_request

Mismatching redirect URI.”

I believe this has to do with a proxy thing.

When logging to superset, the redirect url in the URL is:

http%3A%2F%2Fsuperset.edx. ....

but in the app in the django admin page, it is

https://superset.edx. ...

Our whole setup runs behind an external caddy server.

What setting should I change to make superset aware of that ?

[mboisson (Maxime Boissonneault)]

Adding the “http://superset.edx.” url to the redirect URLs in the django admin page does work. So does manually editing the URL to use https instead of http. But it seems that neither solution is good, the redirect url should just match what was created in the DB.

[mboisson (Maxime Boissonneault)]

@TyHob

You can update the values in the LMS Django admin admin/oauth2_provider/application/ to match.

Can we define those values(SUPERSET_OAUTH2_CLIENT_ID, SUPERSET_OAUTH2_CLIENT_ID_DEV, SUPERSET_OAUTH2_CLIENT_SECRET) in the config, so that they are the same between our production and development environments and that the backup restauration just works ?

[TyHob]

It’s a complicated topic since both services can be run in a variety of ways that Tutor may not be aware of. I think that in this case you may be able to set the Tutor setting SUPERSET_ENABLE_PROXY_FIX to True to work around the issue instead of having to manually override the database entry. We added that workaround for something that’s only come up once or twice, but if that fixes it for you we’ll update the docs to surface it.

[mboisson (Maxime Boissonneault)]

@TyHob

I think that in this case you may be able to set the Tutor setting SUPERSET_ENABLE_PROXY_FIX to True to work around the issue instead of having to manually override the database entry. We added that workaround for something that’s only come up once or twice, but if that fixes it for you we’ll update the docs to surface it.

That did work for me!

I deleted the django apps for superset, I defined SUPERSET_ENABLE_PROXY_FIX: true, then I rebuilt aspects, aspects-superset, and reinitialized aspects. The redirect_url now points to the https url rather than the http one, which matches what was created in the DB.

[TyHob]

Glad to hear it! Just so I can document the use case clearly, your setup is:

• Front door proxy not managed by Tutor, running TLS
• Tutor ENABLE_HTTPS is False

So this line of code is generating http links instead of https:

--redirect-uris "{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://{{ SUPERSET_HOST }}/oauth-authorized/openedxsso" \

The proxy fix is on the Superset side and works by changing the expected redirect uri to use one from the X-Forwarded-For header. Right now we don’t surface all of the potential options for that, so folks are limited to 1 proxy. I’m not sure how much of an issue that is given how rarely this has come up, but I’ll add an issue for looking into it.

If that sounds right can I ping you on the doc PR for review?

[mboisson (Maxime Boissonneault)]

@TyHob

Tutor ENABLE_HTTPS is False

We actually have tutor ENABLE_HTTPS to TrueOtherwise, all of the URLs generated by EdX would point to http and not https. It is the external caddy server who handles the https certificate business.

Our configuration is here:

hieradata-common/edx.yaml

We are using puppet to configure tutor.

The external caddy server is provisioned by Magic Castle, our “infrastructure as code” deployment tool (it provisions a whole HPC cluster).

The external caddy server redirects to the internal caddy’s http interface:

$ cat /etc/caddy/conf.d/superset.edx.conf
superset.edx.evolo-dev.calculquebec.cloud {
  import tls
  route {
    respond /robots.txt 200 {
      body "User-agent: *
Disallow: /"
      close
    }
    reverse_proxy http://edx1.int.evolo-dev.calculquebec.cloud
  }
}

[mboisson (Maxime Boissonneault)]

Also… to make things more complicated, caddy version 2.11 changed the way it worked with headers. That broke a bunch of configurations:

reverseproxy: Set `Host` to `{upstream_hostport}` automatically if TLS (#7454)

I don’t know what is the internal version that is used for caddy in tutor, but that’s probably something to be aware of.

[TyHob]

Thanks for helping troubleshoot this. Here are some doc updates that will help surface the proxy fix:

PR: is here, rendered version is here:

Troubleshoot Aspects — Open edX Aspects latest documentation

And an issue to allow more fine grained settings for the proxy fix in the future:

SUPERSET_PROXY_FIX enhancements

Feel free to review or make suggestions on how we can surface this better.

[mboisson (Maxime Boissonneault)]

For the “such as when populating a staging environment from a production database” part, I would suggest documenting that one should manually define the SUPERSET_OAUTH_CLIENT settings, such as we did here. Otherwise, they are randomly generated each time aspect is configured, and they won’t be updated in the django database if they already exist (maybe this last part should be considered a bug ?)

hieradata-common/edx.yaml

[mboisson (Maxime Boissonneault)]

@Sara_Burns

Hi @mboisson - you can run this and it should fix that error:

tutor [local/dev/k8s] do dbt -c 'run --full-refresh --select fact_problem_engagement' --only_changed False

Hi @Sara_Burns

Is this something that should be run every time we upgrade aspects ? Or is this something specific to this upgrade ?

[Sara_Burns]

That was specific to the upgrade. We usually put any release-specific upgrade instructions in the release notes, but I think a few models were missed in the v2.5 notes so I’ve updated those now.