Expanded REST APIs

colin.fredericks · August 6, 2025, 4:36pm

I have a tool that downloads exports of all our courses. It has to use web automation to open every download page one by one, click the button, wait, check for the next button, etc. Similarly, I have a tool that adds and removes people from our staff. Same thing - it’s web automation that breaks every time the Course Team page changes, because that page doesn’t even have sensible IDs for its controls any more.

Please: APIs. If I can build these kinds of tools to use REST APIs instead of web automation, it’ll take a quarter as long to write them, it’ll remove a lot of fragility, and they’ll run ten times faster.

I can give you a list of what would be useful, but starting with course export and staff/admin roles would be great.

joel.edwards · August 7, 2025, 9:01am

Hi @colin.fredericks
From what I understand based on a similar conversation (here) there are some technological limitations that make a course export API a little tricky. However there are backend scripts that handle the process when you initiate an export from the frontend, perhaps these scripts might put you in a better position to automate the process without using a browser-based automation tool?

github.com/openedx/edx-platform

cms/djangoapps/contentstore/views/import_export.py

master

"""
These views handle all actions in Studio related to import and exporting of
courses
"""


import base64
import json
import logging
import os
import re
import shutil
from wsgiref.util import FileWrapper

from django.conf import settings
from django.contrib.auth.decorators import login_required
from django.core.exceptions import PermissionDenied
from django.core.files import File
from django.core.files.storage import FileSystemStorage
from django.db import transaction

This file has been truncated. show original

IvoBranco · August 7, 2025, 4:22pm

Hi @colin.fredericks

Context: I’m the NAU platform technical manager. We manage our own platform and aren’t using an external one.

Bulk export / backup

We run a bulk export / backup for all courses that are open or on creation process. This is something that we have developed internally that is run weekly for all open courses for our partners. We have developed 2 new Django commands that export the course content and then upload them to be inside of a S3 Bucket, making it downloadable by the course teams (Course > Instructor > Data download. So in case they need to recover something they have those backups.

github.com/fccn/nau-openedx-extensions

nau_openedx_extensions/studio/management/commands/export_course_content_async.py

ac42f401a


      
          """
          Dispatch celery tasks that export courses to a tar.gz file.
          
          Warn: this django command won't export the course, it will only dispatch the task to export the course.
          Making the CMS celery worker with many pending tasks.
          Please check the CMS_URL/heartbeat?extended
          
          Export a specific course:
            python manage.py cms export_course_content_async --username <my_username> <course_id>
          
          Export multiple courses:

github.com/fccn/nau-openedx-extensions

nau_openedx_extensions/studio/management/commands/transfer_export_course_content.py

ac42f401a


      
          """
          Before executing this command run: export_course_content_async.py
          Then run this command to transfer the exported course content from User Tasks to GRADES_DOWNLOAD storage.
          Making the backup of the course more easily available to the teams.
          
          Transfer a specific course:
            python manage.py cms transfer_export_course_content --username <my_username> <course_id>
          
          Transfer multiple courses:
            python manage.py cms transfer_export_course_content --username <my_username> <course_id_1>,<course_id_2>,<course_id_3>

Bulk generate and export reports and batch enrollment

You have developed something similar with what we have done. I even know that one of our partners tried to implement something similar (selenium), but what we have implemented IMO is a better approach. We didn’t use the browser (selenium or playwright) approach, because the UI tends to change very often on releases… IMO execute an HTTP request directly to what the buttons execute bellow is a better approach. The scripts logins an user and sends an HTTP POST that emulates the user clicks on a specific button. It is kind of using the API…, but in reality isn’t an API, it emulates the click on the right button… Those scripts allow some partners that have many, many courses on our platform to execute some routine tasks. I know that they execute this and then have their own Power BI for monitoring:

bulk generate reports, any report on the LMS > Course > Instructor > Data Download
bulk export reports
batch enroll on course - when our partner need to enroll many, many students manually

IMO this approach is better than you emulating the browser using Xpath.

Hope my comment helps you.

Kind regards, Ivo Branco.

NAU Platform

colin.fredericks · August 8, 2025, 1:35am

Thank you to @joel.edwards and @IvoBranco for the ideas. They definitely sound like more reliable approaches. Unfortunately, I don’t have access to the backend on the platform instance that I’m working with.

IvoBranco · August 8, 2025, 9:38am

@colin.fredericks

For export and download the course backup, you can be inspired by the script that I have sent and emulate the HTTP GET and POST that the browser send, use Chrome Network Tools for that.

To export send a POST to: https://<STUDIO_DOMAIN>/export/<course_id>

Then get this URL https://<STUDIO_DOMAIN>/export_status/<course_id>, parse the payload has JSON, payload example:

When still being exporting.

{
  "ExportStatus": 1
}

When it has finish:

{“ExportStatus”: 3,“ExportOutput”: “https://url-to-file”}

Try it until it has finish with success, run an while with a sleep…

When that happens get the ExportOutput and download the file.

Hope it helps you.

Ivo Branco

marcotuts · August 14, 2025, 3:59pm

This is very helpful context!

Just this past week we were discussing course / library backups and archiving tools and wondering how often course teams or instances need to define regular content backups, and whether considering this a first tier platform feature that can be configured would be used broadly.

We only spent a short time exploring this, but the discussion was centered around a site or organization level configuration that would allow you to specify how frequently you might want a course export / backup to be stored - monthly, quarterly, biannually, annually?

The more frequent you need this the larger the storage requirements, but ideally the capability would recognize when a given course backup is identical to a previous one to reduce unnecessary storage and transfer cost.

I know this was drafted in the context of expanded REST APIs, and my comment is not about APIs exactly, but if we did this work perhaps a narrower API that allowed you to find / check against the stored backups you have might be easier to get than one that is generating a full export on demand?

colin.fredericks · August 19, 2025, 2:53pm

@IvoBranco Thanks - I’ll definitely give that a shot.

@marcotuts We generally do backups every 6 months-ish. If we had a way to get a checksum or something like that and compare against existing backups that would definitely cut down on how many we need to download, but we’d still have to check all 150ish courses for that, because exports can change even if the course doesn’t.

Topic		Replies	Views
Where is the api for importing and exporting courses? Tutor Help api , tutor	2	343	July 24, 2023
APIs to fetch courses on and bulk download of transcripts Development	3	1320	November 22, 2019
Extract grade reports for all my courses (bulk extract) Site Operators	4	33	June 9, 2025
How to Import multiple Courses/Libraries? Authoring	20	2803	May 7, 2022
Developing a new studio experience Development	1	732	June 29, 2020

Expanded REST APIs

Related topics