Export OLX Course Format – Management Command Failing

Trying to export a course from a CMS management command results in failure. The file that the output mentions doesn’t exist does exist in Files & Uploads. Any advice on what’s going wrong. The course content is all published out so we’re not in draft state.

ubuntu@hawthorn-app:/edx/app/edxapp/edx-platform$ sudo -u edxapp /edx/bin/python.edxapp ./manage.py cms --settings=aws export_olx course-v1:PHOTONICS+IPSL+DEVELOPMENT --output ~/course.tar.gz
2022-05-03 17:42:26,296 INFO 26758 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
2022-05-03 17:42:26,969 WARNING 26758 [py.warnings] __init__.py:1555 - /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py:1145: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
  self, resource_name

Traceback (most recent call last):
  File "./manage.py", line 121, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 60, in handle
    export_course_to_tarfile(course_key, filename)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 78, in export_course_to_tarfile
    course_dir = export_course_to_directory(course_key, tmp_dir)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/export_olx.py", line 98, in export_course_to_directory
    export_course_to_xml(store, None, course.id, root_dir, course_dir)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_exporter.py", line 346, in export_course_to_xml
    CourseExportManager(modulestore, contentstore, course_key, root_dir, course_dir).export()
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/xml_exporter.py", line 169, in export
    courselike.add_xml_to_node(root)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/course_module.py", line 1103, in definition_to_xml
    xml_object = super(CourseDescriptor, self).definition_to_xml(resource_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 674, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/vertical_block.py", line 160, in definition_to_xml
    self.runtime.add_block_as_child_node(child, xml_object)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1520, in add_block_as_child_node
    block.add_xml_to_node(child)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 602, in add_xml_to_node
    super(XmlParserMixin, self).add_xml_to_node(node)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1114, in add_xml_to_node
    xml_string = self.export_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 591, in export_to_xml
    super(XmlDescriptor, self).add_xml_to_node(node)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 454, in add_xml_to_node
    xml_object = self.definition_to_xml(self.runtime.export_fs)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/video_module/video_module.py", line 717, in definition_to_xml
    course_id=unicode(self.runtime.course_id.for_branch(None))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 864, in export_to_xml
    return create_transcripts_xml(video_id, video_el, resource_fs, static_dir)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 924, in create_transcripts_xml
    static_dir=combine(u'course', static_dir)  # File system should not start from /draft directory.
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/api.py", line 889, in create_transcript_file
    create_file_in_fs(transcript_content, transcript_filename, resource_fs, static_dir)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/edxval/utils.py", line 198, in create_file_in_fs
    with file_system.open(combine(static_dir, file_name), 'wb') as f:
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/osfs.py", line 386, in open
    **options
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/error_tools.py", line 82, in __exit__
    traceback
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/fs/osfs.py", line 386, in open
    **options
fs.errors.ResourceNotFound: resource 'course/static/6451bc96-08fe-4e07-85a3-bcf360b5e48f-en.srt' not found

cc: @dave @jmbowman

So I think what this means is that there is an edx-val VideoTranscript associated with this video. That model stores transcripts as a file field, meaning it’s backed by some django-storages backing and the MySQL field points to it. At some point, that transcript data was written out. Now it’s trying to copy that data back out into a new file to be included in your export.

I’m not clear from the trace whether it’s failing because it can’t find the transcript data to read from, or it doesn’t have permissions (or isn’t properly configured) to write the transcript file as part of the export process. But you might want to start by looking at that angle.

If you really just need to take a hammer to it, you might be able to delete the relevant VideoTranscript model using the Django admin. I’ll caution that I have not tried that before though, and I don’t know if it will cause more issues.

Thanks @dave. For now I’m removing the Video ID set within each video component and republishing the content. That seems to remove that fs.errors.ResourceNotFound error when I perform and management command export_olx. I’ll let you know how this goes.

@dave I was able to export the course now successfully by just removing all Video ID values from all videos throughout the course.

Ah right, because removing the Video ID removes the link between the VideoBlock in Modulestore and edx-val. Glad it got you over the issue, but I wonder how that happened… :slightly_frowning_face:

@dave I imported a course that someone gave us from another Open edX. Do you think those Video ID values came in with the import?

Is it possible the course you imported was originally a course that was on edx.org even if it came from another Open edX instance? I don’t remember ever entering a value in courses on our own Open edX instance. I could be wrong though…

@sambapete Try either performing the following actions then look at the new course video component and you’ll see the Video ID being set.

(Option 1)

  • Perform a course rerun

Update: I just tested this option and it didn’t set the Video ID in the new course run, so it is most likely done with Option 2 below.

I just tested this and it did not update the Video ID so I must have done this with Option 2 below.

(Option 2)

  • Create a new course
  • Export existing course with content then import the content into new course.

@sambapete The Video ID gets set when I perform a Upload New Transcript using an .srt transcript file within the video component. I think once that is done the Course Rerun or Export / Import Course for new course creation mentioned above applies.

That might be the case.

We do not use course reruns. We simply recreate a new course and import the contents of the previous course into the new course.

You are right, it seems to apply to course that do have a transcript.

1 Like

Which in itself is interesting because it states

“If you were assigned a Video ID by edX for the video to play in this component, enter the ID here. In this case, do not enter values in the Default Video URL, the Video File URLs, and the YouTube ID fields. If you were not assigned a Video ID, enter values in those other fields and ignore this field”

edX never assigned a Video ID for our own course runs running on our own Open edX instance. Maybe the help under the field should be rewritten?

@sambapete Yeah I think that documentation is for reuse of the video component in another course. We’ve always just set the Default Video URL and failover video and have ignore the Video ID for the most part until now with Course Export failing.