Course import issue from ironwood to juniper

Zubair · December 11, 2020, 10:08am

I’m trying to import a course that has multiple videos and is hosted on an ironwood server to a newly deployed juniper server. While importing it throws the following errors in the console.
Following is the console log output

TypeError: Unicode-objects must be encoded before hashing
[2020-12-11 09:29:02,847: INFO/Worker-1] VAL: Video created with id [54454916-47e9-4769-8b41-06062d0b7e8c] and status [external]
[2020-12-11 09:29:02,860: ERROR/Worker-1] [VAL] Transcript save failed to storage for video_id "54454916-47e9-4769-8b41-06062d0b7e8c" language code "en"
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/models.py", line 489, in create
    video_transcript.transcript.save(file_name, transcript_content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)

Zubair · December 11, 2020, 4:33pm

@regis can you please look into this and tell me what is happening? I have tried updating botcore version but same error occurs. I saw you comment here, I’m getting similar errors in the logs

[2020-12-11 09:29:05,993: WARNING/Worker-1] XBlock <class 'xblock.internal.DiscussionXBlockWithMixins'> does not contain field url_name
[2020-12-11 09:29:05,994: INFO/Worker-1] Exception 'str' object has no attribute '__traceback__' when trying to load definition xml for block <DiscussionXBlockWithMixins @ECEE discussion_target='1. 1A page1', source_file=None, tags=[], xqa_key=None, discussion_category='Unit 1: The integral', user_partitions=[], matlab_api_key=None, static_asset_path='', format=None, use_latex_compiler=False, due=None, edxnotes_visibility=True, max_attempts=None, self_paced=False, days_early_for_beta=None, discussion_id='discussion_psA_1-tab1', sort_key=None, start=datetime.datetime(2030, 1, 1, 0, 0, tzinfo=<UTC>), video_auto_advance=False, chrome=None, group_access={}, showanswer='finished', show_reset_button=False, in_entrance_exam=False, name=None, video_speed_optimizations=True, graceperiod=None, default_tab=None, hide_from_toc=False, show_correctness='always', graded=False, video_bumper={}, display_name='1. 1A page1', course_edit_method='Studio', xml_attributes={}, parent=None, edxnotes=False, visible_to_staff_only=False, rerandomize='never', giturl=None> - assuming XBlock export format

regis · December 11, 2020, 5:17pm

Indeed I have investigated this issue before, not there, but here.

The solution is provided in this answer. tl;dr: you must apply this patch to edx-platform.

Zubair · December 14, 2020, 5:23am

Bundle of thanks for posting the solution, got stuck on it & couldn’t find any solutions anywhere so far. Thanks again @regis .

Zubair · December 14, 2020, 7:51am

@regis strange but I’m facing same issue after applying the suggested solution, I don’t get this part of the answer though
Apply it to Dockerfile and rebuild image:
RUN curl https://github.com/nguyennk92/edx-platform/commit/e70cf63180b3acd5e19032235595c880cb35bc20.patch 9 | git apply -

regis · December 14, 2020, 9:45am

The instructions in the above-mentioned post apply to Tutor; if you run the native installation, you should patch edx-platform and restart the lms service.

Zubair · December 14, 2020, 9:46am

Already did that but still getting the error…

regis · December 14, 2020, 10:40am

Did you also restart the cms service? (you should)

Zubair · December 14, 2020, 10:44am

@regis Yes I did restart all the services. I even updated edxval version to 1.4.2 but still getting this error.
It throws the following error and the course shows empty verticles for units that contained videos.

TypeError: Unicode-objects must be encoded before hashing
[2020-12-11 09:29:02,735: ERROR/Worker-1] Unable to load child when parsing Vertical. Continuing...
Traceback (most recent call last):
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/vertical_block.py", line 169, in definition_from_xml
    child_block = system.process_xml(etree.tostring(child, encoding='unicode'))
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/xml.py", line 173, in process_xml
    id_manager,
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1714, in xblock_from_node
    block = block_class.parse_xml(node, self, keys, id_generator)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/xml_module.py", line 569, in parse_xml
    return super(XmlParserMixin, cls).parse_xml(node, runtime, keys, id_generator)  # pylint: disable=bad-super-call
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1156, in parse_xml
    block = cls.from_xml(xml, runtime, id_generator)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/video_module/video_module.py", line 673, in from_xml
    getattr(id_generator, 'target_course_id', None)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/video_module/video_module.py", line 1023, in import_video_info_into_val
    course_id=course_id
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/api.py", line 1113, in import_from_xml
    create_transcript_objects(xml, edx_video_id, resource_fs, static_dir, external_transcripts)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/api.py", line 1232, in create_transcript_objects
    static_dir=static_dir
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/api.py", line 1176, in import_transcript_from_fs
    provider=provider
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/api.py", line 350, in create_video_transcript
    transcript_serializer.save(content=content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/rest_framework/serializers.py", line 214, in save
    self.instance = self.create(validated_data)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/serializers.py", line 94, in create
    return VideoTranscript.create(**validated_data)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/edxval/models.py", line 489, in create
    video_transcript.transcript.save(file_name, transcript_content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/db/models/fields/files.py", line 87, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/client.py", line 596, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.5/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)

regis · December 14, 2020, 11:54am

Indeed, it appears that the issue still occurs in Koa. There might be a deeper issue with edxval. I re-opened my PR here: https://github.com/edx/edx-platform/pull/24800#issuecomment-744389775

Zubair · December 14, 2020, 11:57am

I was just reading the comments on that PR, let’s see.

Zubair · December 15, 2020, 6:43am

@regis I checked edxval release versions and version 1.4.3 is the one that fixed S3 bucket upload issue and it works for me

regis · December 15, 2020, 7:04am

@Zubair can you please comment on the PR? This is an issue that should be fixed in Koa.

Zubair · December 15, 2020, 9:44am

Added the comment and tagged you there.

Topic		Replies	Views
Juniper upgrade, getting error while restoring the data Site Operations Help juniper	2	554	October 27, 2020
Migrations error with master: No JSON object could be decoded Site Operations Help how-to	3	1134	March 16, 2020
Automated database migration from Ironwood to Juniper Site Operations Help	3	1022	December 8, 2022
Database Migration course_overviews `courseoverviewtab` Foreign Key Constraint Issue Site Operations Help juniper	4	689	July 14, 2022
There has been an error with your export Site Operators	4	1427	December 11, 2020

Course import issue from ironwood to juniper

Related topics