AI extensibility framework: Default workflow lms-content-summary gets no input

Site Operators
,
1

When I first installed the AI extensibility framework, the example “AI summary” button worked like a charm. But it appears to have broken in one of the recent upgrades (we’re on 2.3.2 now).

I use no content patches to the base.summary base template. This results in the following effective configuration:

{
  "actuator_config": {
    "UIComponents": {
      "request": {
        "component": "AIRequestComponent",
        "config": {
          "buttonText": "AI summary",
          "customMessage": "Produce a brief summary of this unit"
        }
      },
      "response": {
        "component": "AIResponseComponent",
        "config": {
          "customMessage": "AI summary assistant"
        }
      }
    }
  },
  "orchestrator_class": "DirectLLMResponse",
  "processor_config": {
    "LLMProcessor": {
      "function": "summarize_content",
      "provider": "default",
      "stream": true
    },
    "OpenEdXProcessor": {
      "function": "get_location_content"
    }
  },
  "schema_version": "1.0"
}

However, if I hit the corresponding button in the Learning MFE, all I get is:

Sure! Please let me know which unit (topic, title, or key concepts) you’d like summarized, and I’ll provide a concise overview for you.

In other words, it appears like the get_location_content function returns no content.

If I consider its definition, then it looks like any processing error should cause the string Error accessing content to turn up in my logs, but it does not. So perhaps that function is somehow not even invoked?

I am a little bit stumped with this one, and I don’t know where to start looking for clues.

If anyone has ideas/pointers for me, I’d be grateful for those. Thanks!

2

@fghaas I’m so pleased to hear you’re playing around with this!

I’m going to tag @Felipe and @Henrry_Pulgarin on this - one of them should be able to help you out.

3

Thanks! Also, may I suggest you define an ai (or similar) tag, to make it easier to find discussions about that framework?

4

Good call, I added an ai tag and added it to your post.

I also belatedly noticed you didn’t tag the post with the release you’re running - it’d be helpful if you added that tag.

5

Sure. We’re on Ulmo; I’ve added a tag to that effect.

6

@Felipe or @Henrry_Pulgarin, do you happen to have any pointers for me? I’m a bit stuck here, and don’t really know where to start looking for clues.

7

Hi @fghaas at first glance you are doing everything correctly. I’ll need to spin up my dev env to test with the scenario you provided. I’ll come back in a while with my findings

8

Actually, I’m almost certain I’m doing something wrong, I just haven’t figured out a way to pinpoint exactly what that is. :slight_smile:

9

I tested in a dev env using v2.3.2 with the same setup: a new profile which uses base.summary with no patches and a scope to show this in my course at the ui-slot: learning-unit-ai-assist.
With all this it did work. That points to the code working.

The first idea that comes to my mind is that a different profile is answering because there is a scope with higher specificity index (please check that you have all other scopes disabled to rule this out)

Also this makes me think that we lack debugging tools for workflows that don’t have a session.
Which actually is the other thing you could test, change the profile to use “examples.openai.chat”. Launch the assistant as it uses the same

    "OpenEdXProcessor": {
      "function": "get_location_content"
    }

but it will give you the option to debug the session in the /admin/ panel. There you could see the results of the get_location_content as it was passed to the lms. This is kind of a long shot.

image
image1884×1080 435 KB

I was going to add a debug log that function (might still do it later), and while doing that, I though of another idea. You can configure a debug logger for this file, set a char_limit to a low value in the OpenEdXProcessor and you will trigger the debug log in the _truncate_unit_text.

10

I don’t think I need to disable any others, because I don’t have any others. :slight_smile:

In other words, that one scope, which has its

  • Course ID, Location Regex, and UI Slot Selector ID fields blank,
  • Specificity Index set to 0,
  • Service Variant set to LMS, and
  • Profile set to lms-content-summary,

is the only one that is presently defined.

I did try that, but it doesn’t result in any data in the “Debug AI Workflow Session Thread” admin panel.

In the Learning MFE, when I click the “Open Chat” button, I see:

“An unexpected error occurred. Please try again later.”

And the admin interface only gives me:

[
  {
    "session_id": "dd3f8bd6-6baa-44ae-bd17-43db5f63d1d0",
    "user": "florian",
    "course_id": "course-v1:edX+DemoX+Demo_Course",
    "location_id": "block-v1:edX+DemoX+Demo_Course+type@vertical+block@ad00988bd5b64608b3ed46c23be7231c",
    "profile": "lms-content-summary",
    "local_submission_id": null,
    "remote_response_id": null,
    "local_thread": null,
    "remote_thread": null,
    "combined_thread": null,
    "local_thread_error": null,
    "remote_thread_error": null,
    "combined_thread_error": null
  }
]

… and otherwise it says “No thread data available for this session.”

Anything else I could be looking into?

11

Can you share which provider are you setting as your default?