How to Import multiple Courses/Libraries?

I made multiple libraries and courses in a local VM, and now I want to upload them to the production server. Is there a better way to import multiple classes at once, or do I need to import them all one at a time?

Using the Studio front-end
All you can do is export/import single courses in zip format.

However, if you have access to both servers, you can export all courses using management commands. These commands should be run within the edxapp virtualenv.

# Load the edxapp virtualenv
sudo -Hu edxapp bash
source /edx/app/edxapp_env
cd /edx/app/edx-platform

# Export all courses to a directory
./manage.py cms export_all_courses /your/output/path
# Import all courses from a directory
./manage.py cms import /your/exported/courses/path

Donā€™t forget to replace paths with ones of your choice.
There are similar commands to for libraries : export_content_library and import_content_library. Unfortunately they have some bugs so you canā€™t use them untill they are corrected.

3 Likes

Are you aware of a ticket thatā€™s tracking the export_content_library bug? Because I have way more libraries than courses, so thatā€™d be the more useful functionality to me. So if thereā€™s not a bug for that, then I should probably file one right?

(I searched for hits for this error signature but couldnā€™t find anything:
Traceback (most recent call last):
File ā€œ./manage.pyā€, line 123, in
execute_from_command_line([sys.argv[0]] + django_args)
File ā€œ/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/init.pyā€, line 381, in execute_from_command_line
utility.execute()
File ā€œ/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/init.pyā€, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File ā€œ/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/base.pyā€, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File ā€œ/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/base.pyā€, line 364, in execute
output = self.handle(*args, **options)
File ā€œ/opt/bitnami/apps/edx/edx-platform/cms/djangoapps/contentstore/management/commands/export_content_library.pyā€, line 65, in handle
shutil.copyfileobj(tarball.file, f)
File ā€œ/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/shutil.pyā€, line 205, in copyfileobj
fdst_write(buf)
TypeError: write() argument must be str, not bytes

I could solve the one in export_content_library.py by editing line 64 (put wb instead of w in open mode)

            with open(target, 'wb') as f:

I couldnā€™t go any further with import_content_library. Unless this is solved, itā€™s irrelevant to export your libraries without a way to import them

@oedx Iā€™ve filed a pull request here for bugfixes.
You can check it out and use it to correct your own code and do your import while it gets reviewed.

Thatā€™s very cool of you. Itā€™s unfortunate they put up such a legal roadblock to providing this fix, that it will take forever to get incorporated :-/

@oedx just make the changes for yourself for the time being

Jumping in to clarify a couple of problems with that assertion:

  1. Itā€™s a fact of life that any serious open source project will have a contributorā€™s agreement that needs to be signed before a contribution can be accepted. Itā€™s just how copyright (and left :wink:) works. I urge you to take a look at the agreement. Itā€™s pretty reasonable and understandable, as far as these things go.
  2. While many community contributions indeed take a significant amount of time to get merged, this is largely not because of the contributorā€™s agreement. Most often, the delay is simply due to the maintainers not having enough time to get to them all. The good news is that edX is actively looking to remedy this via the Core Commiters Program - i.e., delegating merge rights to community members that stand up to certain very reasonable criteria.

Which is all to say: worry not, thereā€™s every chance that this fix will get into Lilac, the next Open edX release. :slight_smile:

(And since it looks like youā€™re a heavy library user, you might want to keep tabs on the new Library Authoring experience, which is currently under development.)

PS: @braden, as core commiter and original author of the libraries code, mind if I propose @ARMBouhaliā€™s PR as an opportunity to demonstrate that Open edX is worthy of contributions? :wink:

Abderraouf, think you can get that agreement signed? Unfortunately, without your consent, thereā€™s little else we can do in the meantime.

Sure, thanks for the nudge. Iā€™d be happy to once itā€™s cleared for review :).

How does one ā€œreach out to the Open edX team to explore potential contributions and improvements to this experienceā€? I canā€™t contribute because I canā€™t program in your high level languages, but I can at least say what Iā€™m trying to get out of libraries (since using them as they currently exist is a bit of a hack just because thereā€™s apparently no other way to include material in multiple courses.)

Well, in this case, your best bet is myself. :slight_smile: Iā€™m not with the Open edX team, but I am leading this effort on the communityā€™s side. If you manage to try it out, feel free to jot down your thoughts here. Code contributions always carry more weight, of course, but since this is a relatively new project, feedback is very valuable.

@ARMBouhali, have you managed to sign that agreement, yet? This is a slam-dunk contribution. We just need your consent.

Apparently I canā€™t make this kind of change myself in a Tutor-based environment without forking edx-platform and keeping that fork up to date, which I am loath to do. So itā€™d really be good if you could get this commit pushed through.

(See @arbrandes, this is how things stay broken. Requiring an IP agreement for a 3 line fix is a massive disincentive for potential contributors. This is why GCC moved away from this recently: GCC Will No Longer Require Copyrights Be Assigned to the FSF - Slashdot)

I agree, but unfortunately Iā€™m (very) far from being the one who makes these rules.

The Developer Certificate of Origin idea is a good one, IMHO. We should bring it to the new Open edX leadership, once the process of acquisition is complete.

Someone else fixed this too, but thereā€™s been no movement. Any idea how to get it prioritized?

Tagging @braden since I guess it says above you were the maintainer

I just attempted using the export_content_library.py fix by forcing my way into the tutor container and fixing it upā€¦ but am I understanding correctly that this doesnā€™t actually export all libraries like export_all_courses does, but instead only exports a single library? (If so, it turns out not to be what Iā€™m looking for :slightly_frowning_face: )

@braden is there any easy place I could add a few lines of looping to convert export_content_library.py into an export-all? (My python is weak. Looks like maybe the problem is that thereā€™s no get_libraries() equivalent of get_courses(), only a get_library() singular?)

Iā€™ll see if I can review it.

Yes, where this line is calling module_store.get_library(library_key) you could instead call module_store.get_libraries() or .get_library_summaries() or .get_library_keys()

1 Like

Soā€¦ a second developer tried to contribute and their fix was also rejectedā€¦

At what point do open edx developers just ā€œrewriteā€ the 3 lines to fix an obvious bug?

Hi all, I have a few questions and comments:

This has been merged, FYI. (post was in Oct 2021, merged in Dec 2021)

Your post is from April 2022. What is the fix that was rejected that you are mentioning? Are you talking about a different issue because #28219 has been merged awhile ago (Dec 9, 2021)

Could you explain to me what exactly your issues might be with our current CLA? Iā€™ll note our current CLA does not include assignment of copyright. Are there other provisions in the CLA that would cause friction? Is filling out the CLA itself too onerous? (Iā€™ll mention that weā€™ve moved to a DocuSign, so filling out the CLA is as simple as submitting one web form: https://openedx.org/cla ) Is the two-page CLA too difficult to read or understand?

Please note Iā€™m legitimately curious here. I run the Core Contributor program and spend a lot of time thinking about issues to contribute, how to get more contributions, and how to ease developer burden. I am happy to do investigation on what a DCO is, and its legal ramifications, if thereā€™s sufficient compelling reason to do so. Please do share your thoughts.

1 Like

I finally gave in and forked edx-platform to get this fix on Lilac, so now I can possibly make the export-all modification a try (because I have like 40 libraries needing exporting/importing)ā€¦ except the baseline export doesnā€™t seem to be working quite right for me even after I apply what was in the merge.

If I do:

tutor local run cms ./manage.py cms export_content_library library-v1:Org+LibName ./all_libs

I get

CommandError: Output path "./all_libs/" not found.
Error: Command failed with status 1: docker-compose -f /home/tutor/.local/share/tutor/env/local/docker-compose.yml -f /home/tutor/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm cms ./manage.py cms export_content_library library-v1:Org+LibName ./all_libs/

despite the fact that I definitely created ./all_libs. And if I leave off the ./all_libs, it seems to export, saying

Library "library-v1:Org+LibName" exported to "./library-v1+Org+LibName.tar.gz"

But then I donā€™t see any file.

So I suspect itā€™s reading from / writing to some other filesystem location. But if I do tutor local run lms bash I also donā€™t see the file. So any ideas where itā€™s reading/writing from/to?

(I feel like this is probably definitely a dockerization issue, since I just tested and found tutor local run cms ./manage.py cms export_all_courses ./all_course/ fails for the same path-not-found reasonsā€¦)