Exception when submitting a Custom response Python problem

It looks like the custom response OLX with Python code problem breaks upon submission when codejail service is enabled. I’m posting it here to bring it into the notice of the community and see what they think about this.

Steps to reproduce:

  • Create a custom response problem with a script like
  <script type="text/python" system_path="python_lib">
# Python code here e.g. 

def hint_fn(answer_ids, student_answers, new_cmap, old_cmap):

    <hintgroup hintfn="hint_fn"/>

  • Publish changes, And try to test the submissions with any user(admin, staff, or any), NOTE: Make sure that you have enabled codejail service to run the custom Python code.
  • You can check the error stacktrace at the end of the post

Probable Cause?
I think I was able to get to the root of this error and here are some of the findings:

  1. The error is produced because when we try to run the custom Python code through codejail, we build a dictionary followed by a JSON to send as a payload to the request. The code breaks when we try to convert our dictionary to JSON.
  2. Why point# 1 is happening? We are trying to convert a dictionary to JSON which has a whole binary/bytes python_lib package in it(See the example dict below) which json.dumps() fails to convert to JSON and it breaks.

What i think, we do with Custom Response Problems?

For every custom Python code response problem, we run the custom script through codejail or local service either when we load an xBlock in the browser or we submit a response.

Possible Solution?
We should remove any binaries from globals_dict as well before we try json.dumps() on it. Just like how we are doing it here.

I’m not sure how many things can break with this fix, So just wanted to hear from commuity.

An example of how the data looks with a Binary in it is:

'globals_dict': {'seed': 1, 'anonymous_student_id': '98845cd477b76f4339f46600264a3e83'}, 'python_path': ['python_lib.zip'], 'limit_overrides_context': 'course_id', 'slug': 'course_slug', 'unsafely': False, 'extra_files': [('python_lib.zip', b'PK\x03\x04\n\x00\x00\x00\x00\x00k^]O\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0b\x00\x10\x00grader_lib/UX\x0c\x00\x12`\xb8]\xfa_\xb8]\xf5\x01\x14\x00PK\x03\x04\x14\x00\x08\x00\x08\x00=c\xf8J\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1c\x00\x10\x00grader_lib/MultiFunction.pycUX\x0c\x00SZ\xb8]\x95\x1fvY\xf5\x01\x14\x00\xddY_s\xdb\xc6\x11_\x10$ER\x94-\xdb\x92l\xd5\xb1\x85\xd8n\xcb\xb6\x8a\xe5\xa93\x93\x99\x8c\xed$\x8d\xc7}p\x93\x07\xa8\x8d\xc7rU\x0cD\x1cI\xc8 @\x03\'\x0b\xecP\x0f\x1d\xe5\xa1\x1f\xa1\xed\xf4\xad\x0f\xfd\x0c}\xee\xa7\xe8[_\xfa\t\xfa\x01\xda\xdd\xbd;\xf0\x8f\x18\xc5I\x98t\xa6\x94\x08-\x0e\x87\xbd\xfd\xf3\xbb\xdd\xbd\x95\xfd\xef\x95\xc6\xbfn\xbe~\x...

Stack trace of the actual error:

Staff debug info: Traceback (most recent call last): File "/openedx/edx-platform/xmodule/capa/responsetypes.py", line 2297, in execute_check_function safe_exec.safe_exec( File "/usr/local/lib/python3.8/contextlib.py", line 75, in inner return func(*args, **kwds) File "/openedx/edx-platform/xmodule/capa/safe_exec/safe_exec.py", line 159, in safe_exec emsg, exception = get_remote_exec(data) File "/openedx/edx-platform/xmodule/capa/safe_exec/remote_exec.py", line 48, in get_remote_exec return remote_exec_function(*args, **kwargs) File "/openedx/edx-platform/xmodule/capa/safe_exec/remote_exec.py", line 69, in send_safe_exec_request_v0 payload = json.dumps(data) File "/usr/local/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type bytes is not JSON serializable During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/openedx/edx-platform/xmodule/capa_block.py", line 1777, in submit_problem correct_map = self.lcp.grade_answers(answers) File "/openedx/edx-platform/xmodule/capa/capa_problem.py", line 436, in grade_answers new_cmap = self.get_grade_from_current_answers(answers) File "/openedx/edx-platform/xmodule/capa/capa_problem.py", line 494, in get_grade_from_current_answers results = responder.evaluate_answers(self.student_answers, oldcmap) File "/openedx/edx-platform/xmodule/capa/responsetypes.py", line 314, in evaluate_answers new_cmap = self.get_score(student_answers) File "/openedx/edx-platform/xmodule/capa/responsetypes.py", line 2267, in get_score self.execute_check_function(idset, submission) File "/openedx/edx-platform/xmodule/capa/responsetypes.py", line 2308, in execute_check_function self._handle_exec_exception(err) File "/openedx/edx-platform/xmodule/capa/responsetypes.py", line 2523, in _handle_exec_exception raise ResponseError(text_type(err), traceback_obj) xmodule.capa.responsetypes.ResponseError: ('Object of type bytes is not JSON serializable', )

I was looking at https://github.com/openedx/edx-platform/pull/27795 PR that originally added the options for running the codejail optionally.

Would really appreciate it if someone with experience in codejail could respond. @mgmdi or @eric.herrera any chance you can take a look at this and let us know what you think about the problem? I’ve tried a reasonable amount of details, But definitely let me know if you would need more info from my side.

1 Like

Hello @arslanashraf, thanks for your patience. I couldn’t replicate your error with the script you gave us. Can you put a script example that has enough information to be able to reproduce the error?

Another question, Do you mean Problem > Advanced > Blank Advanced Problem, with custom response OLX? Or is there something I’m missing?

On the other hand, I opened an issue around this to have your problem on our radar: ["BUG"] Exception when submitting a Custom response Python problem · Issue #40 · eduNEXT/tutor-contrib-codejail · GitHub

I’ll be attentive to your answers.

1 Like

@mafermazu Thanks for looking at this and taking the appropriate actions to keep this on priority.

So, Actually, we were seeing this in one of the courses and I’m not sure I could share that exact problem code with you so I tried reproducing this with a locally created course in Studio and finally I was able to do so. Also, I think I missed one point in the detail in my original post but now I have all the steps. Please take a look at the steps below:

So here are the details:

  1. Create Problem > Advanced > Blank Advanced Problem with: (Note: This is a regular drag and drop inline problem XML with just the addition of hint_fn)

    Sample Drag and Drop code (Click Here)
    <script type="text/python" system_path="python_lib">
    def hint_fn(answer_ids, student_answers, new_cmap, old_cmap):
          <h3>Drag and Drop with Outline</h3>
          <p>Label the hydrogen atoms connected with the left carbon atom.</p>
          <drag_and_drop_input img="https://studio.edx.org/c4x/edX/DemoX/asset/ethglycol.jpg" target_outline="true" one_per_target="true" no_labels="true" label_bg_color="rgb(222, 139, 238)">
              <draggable id="1" label="Hydrogen" />
              <draggable id="2" label="Hydrogen" />
              <target id="t1_o" x="10" y="67" w="100" h="100"/>
              <target id="t2" x="133" y="3" w="70" h="70"/>
              <target id="t3" x="2" y="384" w="70" h="70"/>
              <target id="t4" x="95" y="386" w="70" h="70"/>
              <target id="t5_c" x="94" y="293" w="91" h="91"/>
              <target id="t6_c" x="328" y="294" w="91" h="91"/>
              <target id="t7" x="393" y="463" w="70" h="70"/>
              <target id="t8" x="344" y="214" w="70" h="70"/>
              <target id="t9_o" x="445" y="162" w="100" h="100"/>
              <target id="t10" x="591" y="132" w="70" h="70"/>
        <answer type="loncapa/python">
    correct_answer = [{
      'draggables': ['1', '2'],
      'targets': ['t2', 't3', 't4' ],
    if draganddrop.grade(submission[0], correct_answer):
      correct = ['correct']
      correct = ['incorrect']
        <hintgroup hintfn="hint_fn"/>
  2. Add this python_lib.zip to your course assets
    python_lib.zip (345.9 KB)

I noticed that the actual problem is because of this additional python_lib.zip file that we have in our assets and the platform loads this file because of <script type="text/python" system_path="python_lib">. But when it does, It tries to convert this whole library into JSON for code jail payload and fails to do so. Here is the relevant edX code that breaks if a library is there in the assets of a course.

I think we should approach this in ways:

  • We should fix this issue upstream e.g. if there is a lib in the assets we don’t try to convert it into a JSON.
  • If adding a zip library in assets is not the right thing we should add it somewhere in the docs.

Let me know if you are able to reproduce this now or need more details.

1 Like

Sorry for the late response, @arslanashraf.

I tried to replicate your problem, but I couldn’t. It works for me D:

Screenshot of the block of the sample you gave me.

I will tell you step-by-step what I did:

  • I used tutor v16.1.4
    • Note: that tutor version uses the branch “open-release/palm.3” of edx-platform
  • I install tutor-contrib-codejail v16.0.0 as the instructions said.
  • I created a course with nothing special and added your python_lib as an asset.
  • I create a Black Advanced Problem with your example, and it works :see_no_evil:

I obtained the result I showed you in the screenshot following those steps.

I don’t know if you are using a different version of tutor or codejail, but your sample works with the latest version.

:pushpin: A thing that can help you, and it’s associated with your approaches, is that when we test codejail we import a particular course that uses a python_lib too. In that course they don’t use the system_path variable in the script, they only use type=“text/python” and works.

I suggest you try to test your example with the latest versions or inside a course in Studio, select Tools > Import, and then use this course to see if you can see the problems they had in their units. I think this could help because they use a python_lib, and they can use the packages in that zip, and you can see how they use it.

Please let me know if you can make it work. :raised_hands:

@mafermazu Thanks a lot for trying this and and all the feedback. I think I understand why it worked for you, That’s because the problem wasn’t with the custom response XML, it was related to a library asset (python_lib.zip) we had in our course in the studio. I’m adding details about it below. But I really appreciate you trying it for us!

We fixed the issue in our own way and I’m going to mention the steps we performed during that. While looking more into it these two solutions came out:

Solution 1: (Override checkfn() function in Custom problem)

What we did was, override the checkfn in the custom OLX problem so that it won’t go through all the steps in the backend. Instead, It would run the simple check logic through the custom response problem itself. This works for us since we didn’t need the check login from the backend.

Solution 2: (Remove python_lib.zip from assets)

This is the main problem and this is why you were not seeing this problem locally while you tried. So, When you have python_lib.zip file in your course assets and you try the OLX that I shared you will see the original error. Removing that asset file also fixes the issue.

When there is python_lib.zip file in the assets, the check_fn function tries to load it and convert it into JSON that will later on be sent to codejail. And it actually fails in the process of converting the python_lib.zip to JSON.

This is the code that breaks.

1 Like

I’m glad you could see what it was @arslanashraf :raised_hands:
Please remember to mark your last post as the solution :smiley: