Tutorial: Attaching Visual Studio Code to Devstack LMS Container

Development
1

Hey there!

The move to using Docker for the Open edX® devstack brought many improvements to our workflow, but it isn’t obvious how to use a key developer tool with the Docker-based devstack: a python debugger.

With recent changes to Visual Studio Code, it’s now possible for VS Code users to get Intellisense and debugging working with your containerized devstack LMS. This will give you as-you-type auto-completion of most python code, near-real-time pylint linting using edX®’s linting rules, and the ability to step through edxapp’s execution and pause and inspect variables while it runs.

Given the complexity of the setup, we at OpenCraft decided to write a tutorial on how to set this up and share it with the development community.

You can find it in this blog post.

Cheers!

10 Likes
2

Hello. Thank you for your post. I was trying to run in debug mode but the error occurs. And I understand that problem with path or env but couldn’t clarify where.
Could you please help with advice)
ERROR:stevedore.extension:Could not load 'completion': No module named apps ERROR:stevedore.extension:No module named apps Traceback (most recent call last): File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins verify_requirements, File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/stevedore/extension.py", line 184, in _load_one_plugin plugin = ep.resolve() File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) ImportError: No module named apps

3

Hey @RuslanSkiraRaccoonga, sorry I missed you question here.

This is an issue I did not see when using these.

I’d try running paver install_python_prereqs inside the container to make sure all dependencies are present.

4

@viadanna Thanks for this post. I got debugging working as you explain in those directions. Really appreciate it. Do you have any advice for PyCharm debugging with devstack_docker? Here is my progress on this.

One concern I had was that the /edx/app/edxapp/edx-platform/lms/env/private.py update of the following produced the following error when I did a make stop.all then make dev.up and Django never started until I commented out these lines.

import ptvsd
ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)

Error

WARNING:py.warnings:/edx/app/edxapp/edx-platform/lms/envs/private.py:9: DeprecationWarning: 'redirect_output' setting via enable_attach will be deprecated in the future versions of the debugger. This can be set using redirectOutput in Launch config in VS Code, using Tee output option in Visual Studio, or debugOptions configuration for any client.
  ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)

2020-07-07 21:57:27,956 WARNING 1101 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2020-07-07 21:57:27,957 WARNING 1101 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
Traceback (most recent call last):
  File "/edx/app/edxapp/edx-platform/manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super(Command, self).execute(*args, **options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 100, in handle
    self.run(**options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 109, in run
    autoreload.main(self.inner_run, None, options)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 341, in main
    reloader(wrapped_main_func, args, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 312, in python_reloader
    exit_code = restart_with_reloader()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 298, in restart_with_reloader
    exit_code = subprocess.call(args, env=new_environ)
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
    self.pid = os.fork()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 528, in new_fork
    _on_forked_process()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 50, in _on_forked_process
    pydevd.settrace_forked()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2427, in settrace_forked
    patch_multiprocessing=True,
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2179, in settrace
    wait_for_ready_to_run,
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 2230, in _locked_settrace
    debugger.connect(host, port)  # Note: connect can raise error.
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_vendored/pydevd/pydevd.py", line 1060, in connect
    s = start_client(host, port)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/pydevd_hooks.py", line 136, in _start_client
    return start_client(daemon, h, p)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_remote.py", line 62, in <lambda>
    start_client=(lambda daemon, h, port: start_daemon()),
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/_remote.py", line 50, in start_daemon
    _, next_session = daemon.start_server(addr=(host, port))
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/daemon.py", line 158, in start_server
    with self.started():
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/daemon.py", line 110, in started
    self.start()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/ptvsd/daemon.py", line 145, in start
    raise RuntimeError('already started')
RuntimeError: already started
5

@viadanna Seems like this is an issue with python 2.7. I’m currently working with the open-release/ironwood.master branch and I think Open edX wasn’t upgraded to python 3 until after that release. For now I have to comment out that ptvsd.enable_attach command in the ./lms/envs/private.py file then when the containers are launched successfully I uncomment it and save. Django is restarted and then I can attach to the process through VS Code debugging. A little hickup but how often do you really stop and start the containers.

Ref: https://github.com/microsoft/ptvsd/issues/1835

6

@viadanna This seems like a potential fix whenever the LMS restarts after a save to say private.py and it autoreloads. I added this to the top of /edx/app/edxapp/edx-platform/lms/envs/private.py file. Found the solution here https://blog.hipolabs.com/remote-debugging-with-vscode-docker-and-pico-fde11f0e5f1c. After the LMS restarts you still need to remember to issue a VS Code > Run > Start Debugging to get it to attach for debugging in that workspace. Seems like this works fine. Let me know what you think.

############## Remote Container Debugging for LMS with VS Code #########################
# https://opencraft.com/blog/tutorial-attaching-visual-studio-code-to-devstack-lms-container/
# https://blog.hipolabs.com/remote-debugging-with-vscode-docker-and-pico-fde11f0e5f1c
# Make sure the LMS enables debugging via ptvsd. 

import os
import subprocess

ENABLE_DEBUGGER_PARAM = "python"
def start_ptvsd_debugger():
    parent_pid = os.getppid()
    cmd = "ps aux | grep %s | awk '{print $2}'" % ENABLE_DEBUGGER_PARAM
    ps = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    pids = ps.communicate()[0].split('\n')

    if str(parent_pid) in pids:
        print('Starting ptvsd debugger')
        import ptvsd
        ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)

start_ptvsd_debugger()
7

Thanks for the heads-up @Zachary_Trabookis

Are you able to get the debugger to stop on breakpoints? They are ignored when using the attach method for me, so I’m working towards launching the LMS and Studio from VSCode. I have the following launch configuration for a remote SSH that I’m trying to convert for the devstack:

{
            "name": "LMS",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/manage.py",
            "args": [
                "lms",
                "runserver",
                "--settings",
                "devstack",
                "--traceback",
                "--pythonpath=.",
                "--noreload",
                "0.0.0.0:8000"
            ],
            "django": true
        }
8

@viadanna Yes I’m able to get it stop on breakpoints. I’m using the launch configuration below like your article mentions. I noticed that if I didn’t have the workspace launched then the debugger wouldn’t work since it was looking for a launch configuration at /edx/app/edxapp/.vscode/ which I never setup.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Remote Attach to LMS Container",
            "type": "python",
            "request": "attach",
            "port": 5678,
            "host": "localhost",
            "django": true, // allows you to set breakpoints in Django template files.
            "pathMappings": [
                {
                    "localRoot": "/edx/app/edxapp/edx-platform",
                    "remoteRoot": "/edx/app/edxapp/edx-platform"
                }
            ]
        }
    ]
}

Here are the steps that I use to start debugging:

  • Startup all services make dev.up
  • Shift + Command + P then Remote Containers: Attach to Running Containers ... then select the LMS service. A new VS Code window opens
  • Within new VS Code window I select File > Open Workspace then select the LMS containers /edx/app/edxapp/edx-platform/.vscode/edxapp-remote.code-workspace file.
  • Select Run > Start Debugging to attach the ptvsd debugger

One thing I noticed was that ./edx-platform/node_modules directory had all of its contents removed if I left the debugger running on a page for a long time and then came back to continue the execution. Even after restarting the LMS service it doesn’t show back. I have to actually perform a make stop.all then make dev.up to get it back. Any ideas on what’s going on with that?

9

@viadanna I don’t think the fix to check to see if a LMS service python process is running before executing the ptvsd.enable_attach command is working at the moment. Let me know if it consistently works for you or not. It seems to work only after the initially load of make dev.up without calling start_ptvsd_debugger(). Any additional saves to the private.py file it does seem to work.

10

@viadanna I had an issue running ptvsd in Ironwood and Juniper releases with the code that I defined Tutorial: Attaching Visual Studio Code to Devstack LMS Container and found success with the following configuration in my ./edx-platform/lms/envs/private.py file. Before the LMS will completely start it hangs on you to start the VS Code > Run > Start Debugging command at the ptvsd.wait_for_attach() statement.

# https://vinta.ws/code/tag/debug/page/2
try:
    import socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.close()
    import ptvsd
    # ptvsd.enable_attach('my_secret', address=('0.0.0.0', 3000))
    ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)
    print('ptvsd is started')
    ptvsd.wait_for_attach()
    print('debugger is attached')
except OSError as exc:
    print(exc)

I also updated my ./edx-platform/.vscode/launch.json file to include the following option. Not sure if that’s necessary though.

{
    "configurations": [
        {
            ...
            "subProcess": true
        }
    ]
}

Anyway let me know if that works for you or not and if you’ve tried other ways to debug using ptvsd. Anything helps with visual IDE debugging.

11

@viadanna So it seems like to get this to work consistently I have to comment this out and wait for the Django server to launch. Then uncomment and save the file waiting for the django.utils.autoreload to happen. For the ptvsd install I’ve include a check for that too. After every autoreload I have to make sure to say VS Code > Run > Start Debugging since the Django app hangs at ptvsd.wait_for_attach() as expected.

try:
    import socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.close()

    # Check to see that `ptvsd` gets installed.
    app_name = 'ptvsd'
    if importlib.util.find_spec(app_name) is None:
        try:
            __import__(app_name)
        except ImportError:
            print('could not import ptvsd')

    import ptvsd
    # ptvsd.enable_attach('my_secret', address=('0.0.0.0', 3000))
    ptvsd.enable_attach(address=('localhost', 5678), redirect_output=True)
    print('ptvsd is started')
    ptvsd.wait_for_attach()
    print('debugger is attached')
except OSError as exc:
    print(exc)
12

Hey @Zachary_Trabookis, glad someone else is invested in this :slight_smile:

I’ll experiment with your setup and report back. For now, I’ve got breakpoints working by using the Remote - Container extension and the following launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "LMS",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/edx-platform/manage.py",
            "args": [
                "lms",
                "--settings",
                "devstack_docker",
                "runserver",
                "0.0.0.0:18000",
                "--noreload"
            ],
            "envFile": "${workspaceFolder}/edxapp_env",
            "pythonPath": "${workspaceFolder}/venvs/edxapp/bin/python",
            "django": true
        }
    ]
}

This launch configuration will launch the LMS development server, so I had to modify devstack/docker-compose.yml to replace the runserver command with a sleep for the LMS/Studio containers (which makes it harder to run those).

It’s very hacky so I refrained from posting it here, but it’s been working so far.

13

Hi @viadanna thanks for your post (https://opencraft.com/blog/tutorial-attaching-visual-studio-code-to-devstack-lms-container/)

It helped me to setup the vscode with remote containers for devstack(https://github.com/edx/devstack#edx-devstack-), however when I start to debug (vscode -> run -> start debugging) i get an error as connection refused

image
image1365×701 93.6 KB

The container python version is 3.5.2 while in the tutorial it is showcased in the screenshot with ver 2.7 , is there some configuration that i need to update so that it works.

14

Hey @camel11, nice to meet you!

The original guide was for the Hawthorn version of the platform, that still ran on Python 2.7.

It seems the ptvsd isn’t being loaded, can you try moving the code from private.py to devstack_docker.py ?

There’s been some issues as you can see from this discussion, so you might want to try one of the other solutions mentioned as well.

Let me know if you have further questions, cheers!

15

Hi @viadanna, nice to meet you too.

I tried to edit the devstack_docker.py and ran the code and I still face the same error.

Is there any easy way to set up the code for modification else it is very difficult only to set up the development environment.

16

Hey @camel11

Can you check the LMS has been correctly started, is running, and PTVSD is loaded? We’ve been facing PTVSD attach issues, with the LMS failing to load, on non-Windows setups.

Note that these changes are related to enabling debugging capabilities (breakpoints, step debugger) on VSCode. You can still properly modify code and run the containers with the platform without this setup.

Cheers!

17

Hi @viadanna

I tried the above configurations with the modifications as mentioned.

When I start the debugger I get the following error given below. Any modification in the configuration if you could suggest would be greatly helpful

File “/edx/app/edxapp/edx-platform/lms/envs/production.py”, line 66, in
CONFIG_FILE = get_env_setting(‘LMS_CFG’)
File “/edx/app/edxapp/edx-platform/lms/envs/production.py”, line 47, in get_env_setting
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the LMS_CFG env variable

18

Hey @camel11

It’s missing the environment from /edx/app/edxapp/edxapp_env can you check properly sourced?

Cheers!

20

HI @viadanna

I checked the location /edx/app/edxapp/edxapp_env it had ansible configuration and LMS_CFG entry with the location of yaml file pointing correctly. But when I hit debug LMS_CFG env is not setup is notified by the debugger.

21

Hi @viadanna

Thanks for your inputs.

I tried running the command in /edx/app/edxapp source edxapp_env after that, the debugger ran fine without any errors stating the following

/edx/app/edxapp/edx-platform/manage.py lms --settings devstack_docker runserver 0.0.0.0:18000 --noreload

1 Like