How do edX developers run edx-platform unit tests?

Development
6

Hey all,

A lot of useful information. Thanks, @Jeff_Chaves @kmccormick @Dawoud_Sheraz
I want to add my +1 to the sad story :slight_smile: of realizing the paver strange behavior and I agree with the intention to move toward the pytest-based way.

Notes for the documentation - actually testing docs is good enough to start:

  • it describes the to way to select test using pytest -k option
  • describes the way to do parallel tests execution by --processes=2 option
  • re-run failed tests by --failed option

Despite this I see ways to improve:

  1. Fix the lack of propper tests separation by subsystems - now we have to run tests for cms like:
pytest cms/djangoapps/__init__.py cms/djangoapps/api cms/djangoapps/cms_user_tasks cms/djangoapps/contentstore cms/djangoapps/course_creators cms/djangoapps/maintenance cms/djangoapps/models cms/djangoapps/pipeline_js cms/djangoapps/xblock_config common/djangoapps/course_action_state common/djangoapps/course_modes common/djangoapps/database_fixups common/djangoapps/edxmako common/djangoapps/entitlements common/djangoapps/pipeline_mako common/djangoapps/static_replace common/djangoapps/status common/djangoapps/student common/djangoapps/terrain common/djangoapps/third_party_auth common/djangoapps/track common/djangoapps/util common/djangoapps/xblock_django openedx/core/djangoapps/__init__.py openedx/core/djangoapps/ace_common openedx/core/djangoapps/api_admin openedx/core/djangoapps/auth_exchange openedx/core/djangoapps/bookmarks openedx/core/djangoapps/cache_toolbox openedx/core/djangoapps/catalog openedx/core/djangoapps/ccxcon openedx/core/djangoapps/certificates openedx/core/djangoapps/commerce openedx/core/djangoapps/common_initialization openedx/core/djangoapps/common_views openedx/core/djangoapps/config_model_utils openedx/core/djangoapps/content openedx/core/djangoapps/content_libraries openedx/core/djangoapps/contentserver openedx/core/djangoapps/cors_csrf openedx/core/djangoapps/course_date_signals openedx/core/djangoapps/course_groups openedx/core/djangoapps/coursegraph openedx/core/djangoapps/courseware_api openedx/core/djangoapps/crawlers openedx/core/djangoapps/credentials openedx/core/djangoapps/credit openedx/core/djangoapps/dark_lang openedx/core/djangoapps/debug openedx/core/djangoapps/django_comment_common openedx/core/djangoapps/embargo openedx/core/djangoapps/enrollments openedx/core/djangoapps/external_user_ids openedx/core/djangoapps/geoinfo openedx/core/djangoapps/header_control openedx/core/djangoapps/heartbeat openedx/core/djangoapps/lang_pref openedx/core/djangoapps/models openedx/core/djangoapps/monkey_patch openedx/core/djangoapps/oauth_dispatch openedx/core/djangoapps/olx_rest_api openedx/core/djangoapps/password_policy openedx/core/djangoapps/plugin_api openedx/core/djangoapps/plugins openedx/core/djangoapps/profile_images openedx/core/djangoapps/programs openedx/core/djangoapps/safe_sessions openedx/core/djangoapps/schedules openedx/core/djangoapps/self_paced openedx/core/djangoapps/service_status openedx/core/djangoapps/session_inactivity_timeout openedx/core/djangoapps/signals openedx/core/djangoapps/site_configuration openedx/core/djangoapps/system_wide_roles openedx/core/djangoapps/theming openedx/core/djangoapps/user_api openedx/core/djangoapps/user_authn openedx/core/djangoapps/util openedx/core/djangoapps/verified_track_content openedx/core/djangoapps/video_config openedx/core/djangoapps/video_pipeline openedx/core/djangoapps/waffle_utils openedx/core/djangoapps/xblock openedx/core/djangoapps/xmodule_django openedx/core/djangoapps/zendesk_proxy openedx/tests/__init__.py openedx/tests/completion_integration openedx/tests/settings.py openedx/tests/util openedx/tests/xblock_integration openedx/core/lib/__init__.py openedx/core/lib/api openedx/core/lib/blockstore_api openedx/core/lib/cache_utils.py openedx/core/lib/celery openedx/core/lib/command_utils.py openedx/core/lib/course_tabs.py openedx/core/lib/courses.py openedx/core/lib/derived.py openedx/core/lib/django_courseware_routers.py openedx/core/lib/django_require openedx/core/lib/django_test_client_utils.py openedx/core/lib/dynamic_partitions_generators.py openedx/core/lib/edx_api_utils.py openedx/core/lib/edx_six.py openedx/core/lib/exceptions.py openedx/core/lib/extract_tar.py openedx/core/lib/gating openedx/core/lib/grade_utils.py openedx/core/lib/graph_traversals.py openedx/core/lib/hash_utils.py openedx/core/lib/html_to_text.py openedx/core/lib/json_utils.py openedx/core/lib/license openedx/core/lib/log_utils.py openedx/core/lib/logsettings.py openedx/core/lib/mail_utils.py openedx/core/lib/mobile_utils.py openedx/core/lib/plugins.py openedx/core/lib/request_utils.py openedx/core/lib/rooted_paths.py openedx/core/lib/session_serializers.py openedx/core/lib/teams_config.py openedx/core/lib/tempdir.py openedx/core/lib/tests openedx/core/lib/time_zone_utils.py openedx/core/lib/url_utils.py openedx/core/lib/x_forwarded_for openedx/core/lib/xblock_builtin openedx/core/lib/xblock_pipeline openedx/core/lib/xblock_utils cms/lib/__init__.py cms/lib/xblock

Using markers it can be simplified to:

pytest -m cms

  1. We can marks heavy/slow and fast tests to be able to run fast tests as often as possible (--fasttest is not for this but for skipping collectstatic - compiling Sass to CSS) and slow tests (actually full test suite) for major PRs. This also can be useful in CI optimization work (e.g. to implement the fail early strategy to not block a pipeline executor).

  2. Return back to the BDD style for our Acceptance tests (or at least add more human-readable output). I’m not sure why some old BDD tests were rewritten into default unittests and BDD scenarios were moved into docstrings.
    Current Karma tests (karma-bdd) is a great example of how the behavior style can help developers and testers to understand system behavior:

 FileUpload
    ✓ is unfinished by default
    ✓ is not uploading by default
    ✓ is valid by default
    ✓ is valid for text files by default
    ✓ is valid for PNG files by default
    ✓ can accept a file type when explicitly set
    ✓ can accept a file format when explicitly set
    ✓ can accept multiple file types
    ✓ can accept multiple file formats
    fileTypes
      ✓ returns a list of the uploaders file types
    formatValidTypes
      ✓ returns a map of formatted file types and extensions
      ✓ does not format with only one mime type
  1. There is a possibility to improve a developer experience by utilizing docker features to avoid unnecessary setup steps (like a DB setup or dependency installation). For the developer work probably it is not so critical as for CI flow which can be optimized.

To push all these ideas forward I’ve created the very initial roadmap issue (it’s named as a CI improvement issue but I believe it should cover developer day-to-day experience too) https://openedx.atlassian.net/browse/OEROADMAP-38.
Also, it should be described in more detail with exact proposals - will add some.

@regis great thanks to starting raising all these questions :metal: