I made multiple libraries and courses in a local VM, and now I want to upload them to the production server. Is there a better way to import multiple classes at once, or do I need to import them all one at a time?
Using the Studio front-end
All you can do is export/import single courses in zip format.
However, if you have access to both servers, you can export all courses using management commands. These commands should be run within the edxapp virtualenv.
# Load the edxapp virtualenv
sudo -Hu edxapp bash
source /edx/app/edxapp_env
cd /edx/app/edx-platform
# Export all courses to a directory
./manage.py cms export_all_courses /your/output/path
# Import all courses from a directory
./manage.py cms import /your/exported/courses/path
Donāt forget to replace paths with ones of your choice.
There are similar commands to for libraries : export_content_library and import_content_library. Unfortunately they have some bugs so you canāt use them untill they are corrected.
3 Likes
Are you aware of a ticket thatās tracking the export_content_library bug? Because I have way more libraries than courses, so thatād be the more useful functionality to me. So if thereās not a bug for that, then I should probably file one right?
(I searched for hits for this error signature but couldnāt find anything:
Traceback (most recent call last):
File ā./manage.pyā, line 123, in
execute_from_command_line([sys.argv[0]] + django_args)
File ā/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/init.pyā, line 381, in execute_from_command_line
utility.execute()
File ā/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/init.pyā, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File ā/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/base.pyā, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File ā/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/site-packages/django/core/management/base.pyā, line 364, in execute
output = self.handle(*args, **options)
File ā/opt/bitnami/apps/edx/edx-platform/cms/djangoapps/contentstore/management/commands/export_content_library.pyā, line 65, in handle
shutil.copyfileobj(tarball.file, f)
File ā/opt/bitnami/apps/edx/venvs/edxapp/lib/python3.8/shutil.pyā, line 205, in copyfileobj
fdst_write(buf)
TypeError: write() argument must be str, not bytes
I could solve the one in export_content_library.py by editing line 64 (put wb instead of w in open mode)
with open(target, 'wb') as f:
I couldnāt go any further with import_content_library. Unless this is solved, itās irrelevant to export your libraries without a way to import them
@oedx Iāve filed a pull request here for bugfixes.
You can check it out and use it to correct your own code and do your import while it gets reviewed.
Thatās very cool of you. Itās unfortunate they put up such a legal roadblock to providing this fix, that it will take forever to get incorporated :-/
1 Like
@oedx just make the changes for yourself for the time being
Jumping in to clarify a couple of problems with that assertion:
- Itās a fact of life that any serious open source project will have a contributorās agreement that needs to be signed before a contribution can be accepted. Itās just how copyright (and left
) works. I urge you to take a look at the agreement. Itās pretty reasonable and understandable, as far as these things go. - While many community contributions indeed take a significant amount of time to get merged, this is largely not because of the contributorās agreement. Most often, the delay is simply due to the maintainers not having enough time to get to them all. The good news is that edX is actively looking to remedy this via the Core Commiters Program - i.e., delegating merge rights to community members that stand up to certain very reasonable criteria.
Which is all to say: worry not, thereās every chance that this fix will get into Lilac, the next Open edX release. 
(And since it looks like youāre a heavy library user, you might want to keep tabs on the new Library Authoring experience, which is currently under development.)
PS: @braden, as core commiter and original author of the libraries code, mind if I propose @ARMBouhaliās PR as an opportunity to demonstrate that Open edX is worthy of contributions? 
Abderraouf, think you can get that agreement signed? Unfortunately, without your consent, thereās little else we can do in the meantime.
Sure, thanks for the nudge. Iād be happy to once itās cleared for review :).
How does one āreach out to the Open edX team to explore potential contributions and improvements to this experienceā? I canāt contribute because I canāt program in your high level languages, but I can at least say what Iām trying to get out of libraries (since using them as they currently exist is a bit of a hack just because thereās apparently no other way to include material in multiple courses.)
Well, in this case, your best bet is myself. Iām not with the Open edX team, but I am leading this effort on the communityās side. If you manage to try it out, feel free to jot down your thoughts here. Code contributions always carry more weight, of course, but since this is a relatively new project, feedback is very valuable.
@ARMBouhali, have you managed to sign that agreement, yet? This is a slam-dunk contribution. We just need your consent.
Apparently I canāt make this kind of change myself in a Tutor-based environment without forking edx-platform and keeping that fork up to date, which I am loath to do. So itād really be good if you could get this commit pushed through.
(See @arbrandes, this is how things stay broken. Requiring an IP agreement for a 3 line fix is a massive disincentive for potential contributors. This is why GCC moved away from this recently: GCC Will No Longer Require Copyrights Be Assigned to the FSF - Slashdot)
I agree, but unfortunately Iām (very) far from being the one who makes these rules.
The Developer Certificate of Origin idea is a good one, IMHO. We should bring it to the new Open edX leadership, once the process of acquisition is complete.
Someone else fixed this too, but thereās been no movement. Any idea how to get it prioritized?
Tagging @braden since I guess it says above you were the maintainer
I just attempted using the export_content_library.py fix by forcing my way into the tutor container and fixing it upā¦ but am I understanding correctly that this doesnāt actually export all libraries like export_all_courses does, but instead only exports a single library? (If so, it turns out not to be what Iām looking for 
)
@braden is there any easy place I could add a few lines of looping to convert export_content_library.py into an export-all? (My python is weak. Looks like maybe the problem is that thereās no get_libraries() equivalent of get_courses(), only a get_library() singular?)
Iāll see if I can review it.
Yes, where this line is calling module_store.get_library(library_key) you could instead call module_store.get_libraries() or .get_library_summaries() or .get_library_keys()
1 Like
Soā¦ a second developer tried to contribute and their fix was also rejectedā¦
At what point do open edx developers just ārewriteā the 3 lines to fix an obvious bug?
Hi all, I have a few questions and comments:
This has been merged, FYI. (post was in Oct 2021, merged in Dec 2021)
Your post is from April 2022. What is the fix that was rejected that you are mentioning? Are you talking about a different issue because #28219 has been merged awhile ago (Dec 9, 2021)
Could you explain to me what exactly your issues might be with our current CLA? Iāll note our current CLA does not include assignment of copyright. Are there other provisions in the CLA that would cause friction? Is filling out the CLA itself too onerous? (Iāll mention that weāve moved to a DocuSign, so filling out the CLA is as simple as submitting one web form: https://openedx.org/cla ) Is the two-page CLA too difficult to read or understand?
Please note Iām legitimately curious here. I run the Core Contributor program and spend a lot of time thinking about issues to contribute, how to get more contributions, and how to ease developer burden. I am happy to do investigation on what a DCO is, and its legal ramifications, if thereās sufficient compelling reason to do so. Please do share your thoughts.
1 Like
I finally gave in and forked edx-platform to get this fix on Lilac, so now I can possibly make the export-all modification a try (because I have like 40 libraries needing exporting/importing)ā¦ except the baseline export doesnāt seem to be working quite right for me even after I apply what was in the merge.
If I do:
tutor local run cms ./manage.py cms export_content_library library-v1:Org+LibName ./all_libs
I get
CommandError: Output path "./all_libs/" not found.
Error: Command failed with status 1: docker-compose -f /home/tutor/.local/share/tutor/env/local/docker-compose.yml -f /home/tutor/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm cms ./manage.py cms export_content_library library-v1:Org+LibName ./all_libs/
despite the fact that I definitely created ./all_libs. And if I leave off the ./all_libs, it seems to export, saying
Library "library-v1:Org+LibName" exported to "./library-v1+Org+LibName.tar.gz"
But then I donāt see any file.
So I suspect itās reading from / writing to some other filesystem location. But if I do tutor local run lms bash I also donāt see the file. So any ideas where itās reading/writing from/to?
(I feel like this is probably definitely a dockerization issue, since I just tested and found tutor local run cms ./manage.py cms export_all_courses ./all_course/ fails for the same path-not-found reasonsā¦)