Export OLX Course Format – Management Command Failing

Zachary_Trabookis · May 3, 2022, 9:47pm

Trying to export a course from a CMS management command results in failure. The file that the output mentions doesn’t exist does exist in Files & Uploads. Any advice on what’s going wrong. The course content is all published out so we’re not in draft state.

ubuntu@hawthorn-app:/edx/app/edxapp/edx-platform$ sudo -u edxapp /edx/bin/python.edxapp ./manage.py cms --settings=aws export_olx course-v1:PHOTONICS+IPSL+DEVELOPMENT --output ~/course.tar.gz
2022-05-03 17:42:26,296 INFO 26758 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
2022-05-03 17:42:26,969 WARNING 26758 [py.warnings] __init__.py:1555 - /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py:1145: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
  self, resource_name

Traceback (most recent call last):
  File "./manage.py", line 121, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 60, in handle
    export_course_to_tarfile(course_key, filename)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 78, in export_course_to_tarfile
    course_dir = export_course_to_directory(course_key, tmp_dir)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 98, in export_course_to_directory
    export_course_to_xml(store, None, course.id, root_dir, course_dir)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_exporter.py", line 346, in export_course_to_xml
    CourseExportManager(modulestore, contentstore, course_key, root_dir, course_dir).export()
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_exporter.py", line 169, in export
    courselike.add_xml_to_node(root)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/course_module.py", line 1103, in definition_to_xml
    xml_object = super(CourseDescriptor, self).definition_to_xml(resource_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/vertical_block.py", line 160, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/video_module/video_module.py", line 717, in definition_to_xml
    course_id=unicode(self.runtime.course_id.for_branch(None))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 864, in export_to_xml
    return create_transcripts_xml(video_id, video_el, resource_fs, static_dir)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 924, in create_transcripts_xml
    static_dir=combine(u'course', static_dir)  # File system should not start from /draft directory.
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 889, in create_transcript_file
    create_file_in_fs(transcript_content, transcript_filename, resource_fs, static_dir)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/utils.py", line 198, in create_file_in_fs
    with file_system.open(combine(static_dir, file_name), 'wb') as f:
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/osfs.py", line 386, in open
    **options
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/error_tools.py", line 82, in __exit__
    traceback
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/osfs.py", line 386, in open
    **options
fs.errors.ResourceNotFound: resource 'course/static/6451bc96-08fe-4e07-85a3-bcf360b5e48f-en.srt' not found

cc: @dave @jmbowman

dave · May 3, 2022, 10:20pm

So I think what this means is that there is an edx-val VideoTranscript associated with this video. That model stores transcripts as a file field, meaning it’s backed by some django-storages backing and the MySQL field points to it. At some point, that transcript data was written out. Now it’s trying to copy that data back out into a new file to be included in your export.

I’m not clear from the trace whether it’s failing because it can’t find the transcript data to read from, or it doesn’t have permissions (or isn’t properly configured) to write the transcript file as part of the export process. But you might want to start by looking at that angle.

If you really just need to take a hammer to it, you might be able to delete the relevant VideoTranscript model using the Django admin. I’ll caution that I have not tried that before though, and I don’t know if it will cause more issues.

Zachary_Trabookis · May 4, 2022, 1:09pm

Thanks @dave. For now I’m removing the Video ID set within each video component and republishing the content. That seems to remove that fs.errors.ResourceNotFound error when I perform and management command export_olx. I’ll let you know how this goes.

Zachary_Trabookis · May 4, 2022, 1:30pm

@dave I was able to export the course now successfully by just removing all Video ID values from all videos throughout the course.

dave · May 4, 2022, 1:31pm

Ah right, because removing the Video ID removes the link between the VideoBlock in Modulestore and edx-val. Glad it got you over the issue, but I wonder how that happened…

Zachary_Trabookis · May 4, 2022, 1:32pm

@dave I imported a course that someone gave us from another Open edX. Do you think those Video ID values came in with the import?

sambapete · May 4, 2022, 3:12pm

Is it possible the course you imported was originally a course that was on edx.org even if it came from another Open edX instance? I don’t remember ever entering a value in courses on our own Open edX instance. I could be wrong though…

Zachary_Trabookis · May 4, 2022, 4:37pm

@sambapete Try either performing the following actions then look at the new course video component and you’ll see the Video ID being set.

(Option 1)

Perform a course rerun

Update: I just tested this option and it didn’t set the Video ID in the new course run, so it is most likely done with Option 2 below.

I just tested this and it did not update the Video ID so I must have done this with Option 2 below.

(Option 2)

Create a new course
Export existing course with content then import the content into new course.

Zachary_Trabookis · May 4, 2022, 4:53pm

@sambapete The Video ID gets set when I perform a Upload New Transcript using an .srt transcript file within the video component. I think once that is done the Course Rerun or Export / Import Course for new course creation mentioned above applies.

sambapete · May 4, 2022, 5:42pm

That might be the case.

We do not use course reruns. We simply recreate a new course and import the contents of the previous course into the new course.

You are right, it seems to apply to course that do have a transcript.

sambapete · May 4, 2022, 5:45pm

Which in itself is interesting because it states

“If you were assigned a Video ID by edX for the video to play in this component, enter the ID here. In this case, do not enter values in the Default Video URL, the Video File URLs, and the YouTube ID fields. If you were not assigned a Video ID, enter values in those other fields and ignore this field”

edX never assigned a Video ID for our own course runs running on our own Open edX instance. Maybe the help under the field should be rewritten?

Zachary_Trabookis · May 4, 2022, 5:51pm

@sambapete Yeah I think that documentation is for reuse of the video component in another course. We’ve always just set the Default Video URL and failover video and have ignore the Video ID for the most part until now with Course Export failing.

rose · June 28, 2022, 9:01am

pls, sir How to export all courses? I tried this command and shown this error? pls advice me.
$ ./manage.py cms export_all_courses /your/output/path

edxapp@Edx-VM-DB:~/edx-platform$ ./manage.py cms export_all_courses /root/
WARNING:py.warnings:/edx/app/edxapp/edx-platform/lms/djangoapps/courseware/init.py:5: DeprecationWarning: Importing ‘lms.djangoapps.courseware’ as ‘courseware’ is no longer supported
warnings.warn(“Importing ‘lms.djangoapps.courseware’ as ‘courseware’ is no longer supported”, DeprecationWarning)

2022-06-28 04:58:55,847 WARNING 20300 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2022-06-28 04:58:55,847 WARNING 20300 [enterprise.utils] utils.py:51 - cannot import name EnterpriseCustomerUser
2022-06-28 04:59:01,116 WARNING 20300 [py.warnings] init.py:1564 - /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/init.py:1145: DeprecationWarning: Use of … or absolute path in a resource path is not allowed and will raise exceptions in a future release.
self, resource_name

Traceback (most recent call last):
File “./manage.py”, line 123, in
execute_from_command_line([sys.argv[0]] + django_args)
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line
utility.execute()
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/init.py”, line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py”, line 283, in run_from_argv
self.execute(*args, **cmd_options)
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py”, line 330, in execute
output = self.handle(*args, **options)
File “/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_all_courses.py”, line 27, in handle
courses, failed_export_courses = export_courses_to_output_path(options[‘output_path’])
File “/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_all_courses.py”, line 45, in export_courses_to_output_path
courses = module_store.get_courses()
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py”, line 89, in inner
retval = func(field_decorator=strip_key_collection, *args, **kwargs)
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mixed.py”, line 317, in get_courses
for course in store.get_courses(**kwargs):
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split_draft.py”, line 99, in get_courses
return super(DraftVersioningModuleStore, self).get_courses(ModuleStoreEnum.BranchName.draft, **kwargs)
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/mongodb_proxy.py”, line 53, in wrapper
return func(*args, **kwargs)
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py”, line 1021, in get_courses
return self._get_structures_for_branch_and_locator(branch, self._create_course_locator, **kwargs)
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py”, line 981, in _get_structures_for_branch_and_locator
structures_list = self._load_items(envelope, [root], depth=0, **kwargs)
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py”, line 822, in _load_items
runtime = self.create_runtime(course_entry, lazy)
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/split.py”, line 3455, in create_runtime
services=services,
File “/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py”, line 64, in init
root.makedirs_p() # create directory if it doesn’t exist
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/path.py”, line 1220, in makedirs_p
self.makedirs(mode)
File “/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/path.py”, line 1213, in makedirs
os.makedirs(self, mode)
File “/edx/app/edxapp/venvs/edxapp/lib/python2.7/os.py”, line 150, in makedirs
makedirs(head, mode)
File “/edx/app/edxapp/venvs/edxapp/lib/python2.7/os.py”, line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: ‘/edx/var/edxapp/data/testing_course/OPRD-MSO-190202’
edxapp@Edx-VM-DB:~/edx-platform$

system · September 26, 2022, 9:01am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Permission denied in export all courses/libraries Site Operations Help	1	247	September 27, 2022
Error when trying to export a course Site Operations Help	0	796	August 10, 2020
There has been an error with your export Site Operators	4	1383	December 11, 2020
Any ideas how to get to work manage.py? Site Operations Help	4	1566	November 4, 2020
Uh oh, we are having some server issues Site Operations Help	7	473	November 9, 2021

Export OLX Course Format – Management Command Failing

Related Topics