Running Blockstore in Juniper

Site Operators Site Operations Help
1

Hi all!

We have recently started testing Blockstore in Juniper. To deploy it we used the blockstore role from edx-configuration, and for the initial setup we applied the following variables:

BLOCKSTORE_VERSION
BLOCKSTORE_DATABASE_PASSWORD
BLOCKSTORE_DATABASE_HOST
BLOCKSTORE_SECRET_KEY

One thing that isn’t quite clear is how we can set SOCIAL_AUTH_EDX_OAUTH2_SECRET from edx-configuration. Other services have INSIGHTS_SOCIAL_AUTH_EDX_OAUTH2_SECRET or ECOMMERCE_SOCIAL_AUTH_EDX_OAUTH2_SECRET, but there’s no BLOCKSTORE_SOCIAL_AUTH_EDX_OAUTH2_SECRET. Is the idea to add SOCIAL_AUTH_EDX_OAUTH2_SECRET to the BLOCKSTORE_SERVICE_CONFIG_OVERRIDES dictionary?

Moving forward, we would like to know what is the actual, proper way in Juniper to include and edit content libraries stored in Blockstore?

Do we need to deploy ramshackle into the edxapp venv to be able to edit Blockstore-backed content libraries from Studio, or is there a different recommended way of editing those now?

Is it correct, that in order to be able to use these content libraries (once they’ve been created), all that the LMS needs to be configured with is EDXAPP_BLOCKSTORE_API_URL (and the blockstore-sso-key OAuth2 credentials), or are there more settings to be added to edxapp so that the LMS and CMS can both use Blockstore?

3

Hi Maari, and welcome to the board!

You’re correct in that currently, the only way to edit libraries is via ramshackle. And yes, it does need to be installed in the edxapp venv. There is, however, work in progress to create a more fully-featured, Studio-integrated effort. It’s not ready, yet, but you can keep tabs on it via BD-14.

As for having the LMS actually use blockstore libraries, I’m not familiar with how edx.org deploys blockstore, or even if they use edx-configuration for it. (Maybe @dave or @braden can enlighten you, here.) What I would suggest is for you to deploy it in conjunction with the Docker devstack, and try it out there. If you’re running vanilla master, all you have to do is check out the blockstore repo and run make easyserver. By looking at the configuration in both repos, you should be able to figure out what’s needed to connect the two together in production.

As for whether this will also work with Juniper, your mileage may vary. As far as I know, it should, as Juniper is a recent enough cut of master, but I haven’t tried doing so, yet.

Finally, to be perfectly honest, while there are projects (such as LabXchange) that use Blockstore-backed content via LMS API calls, I’m not entirely sure that a regular modulestore-backed course on the LMS itself can. @braden?

4

As for having the LMS actually use blockstore libraries, I’m not familiar with how edx.org deploys blockstore, or even if they use edx-configuration for it.

It looks to me like the blockstore role in edx-configuration is currently borked in a subtle way:

  • blockstore inherits from the edx_django_service role via a meta dependency, and in its dependency declaration it sets a bunch of variables for that service.
  • One of those variables is edx_django_service_default_db_conn_max_age, which it sets as follows:
edx_django_service_default_db_conn_max_age: '{{ BLOCKSTORE_DATABASE_CONN_MAX_AGE }}'

For comparison, see also:

edx_django_service_default_db_atomic_requests: true

Note how the former value is quoted, which it must be, due to a quirk in the YAML parser that Ansible uses. The latter is not.

When edx_django_service (which itself depends on edx_service) creates the edx_service_config dictionary and then renders the blockstore.yml file using the | to_nice_yaml filter in edx_service/templates/config.yml.j2, it apparently treats the quoted value as a string, whereas the unquoted value is correctly interpreted as a boolean. And what you end up getting in blockstore.yml is:

DATABASES: 
    default: 
        ATOMIC_REQUESTS: true
        CONN_MAX_AGE: '60'

Unfortunately Django doesn’t accept a string for CONN_MAX_AGE. Any API requests to a thus-configured blockstore end up in an HTTP 500, with this exception in the blockstore-stdout.log:

2020-07-07 12:05:05,673 ERROR 1564 [django.request] exception.py:135 - Internal Server Error: /api/v1/
Traceback (most recent call last):
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 29, in inner
    with self._recreate_cm():
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/db/transaction.py", line 158, in __enter__
    if not connection.get_autocommit():
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/db/backends/base/base.py", line 385, in get_autocommit
    self.ensure_connection()
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/edx/app/blockstore/venvs/blockstore/lib/python3.5/site-packages/django/db/backends/base/base.py", line 184, in connect
    self.close_at = None if max_age is None else time.time() + max_age

Is there a workaround for this?

Maybe that’s another question for @braden, as I understand he’s the original author of that role.

5

Welcome @mrtmm, I’m glad to see your interest in Blockstore.

Yes, exactly, use BLOCKSTORE_SERVICE_CONFIG_OVERRIDES.

Note that you don’t really need to set SOCIAL_AUTH_EDX_OAUTH2_SECRET for production instances, unless you need/want to log in to the Blockstore Django admin directly. The LMS/Studio communicate with Blockstore using APIs that are authenticated by an API token (BLOCKSTORE_API_AUTH_TOKEN), and don’t require use of OAUTH2 nor SSO.

In Juniper there are only two ways to edit content in (Blockstore-based) content libraries: using the REST API (this is how we do the editing functionality on www.labxchange.org for example), or using Ramshackle. The APIs are listed here and here.

Unfortunately Juniper doesn’t provide any mechanism to include content from Blockstore-based content libraries into a course. That’s something that’s currently in development, and you could try it out by cherry-picking this open PR for the BD-14 “Source from Library” XBlock onto your Juniper branch. However, Juniper does allow learners (registered and optionally anonymous/not-logged-in) to view and learn from content libraries directly, without even needing to pull the content into a course. Ramshackle includes an example of that, and it’s what we use on www.labxchange.org too.

The only settings you need are BLOCKSTORE_API_URL and BLOCKSTORE_API_AUTH_TOKEN. Here is a fully working example of ansible vars for deploying Blockstore, taken from one of our Open edX instances used for continuous integration. The first two blocks are all configuring the blockstore service itself; only the third and final section is affecting the LMS/Studio.

# Enable blockstore: ############################################

# Run blockstore, and expose it publicly at 'blockstore.openedx-example.opencraft.hosting'
SANDBOX_ENABLE_BLOCKSTORE: true
BLOCKSTORE_NGINX_HOSTNAME: 'blockstore.*'
BLOCKSTORE_NGINX_PORT: 80
BLOCKSTORE_SSL_NGINX_PORT: 443
BLOCKSTORE_SECRET_KEY: secretvalue2here
BLOCKSTORE_DATABASE_HOST: mysql-server.opencraft.hosting
BLOCKSTORE_DATABASE_USER: blockstore_user
BLOCKSTORE_DATABASE_PASSWORD: secretvalue3here
BLOCKSTORE_DEFAULT_DB_NAME: openedx_opencraft_hosting_blockstore

# Use S3 for blockstore data:
BLOCKSTORE_SERVICE_CONFIG_OVERRIDES:
    DEFAULT_FILE_STORAGE: storages.backends.s3boto3.S3Boto3Storage
    AWS_ACCESS_KEY_ID: AKIAWABCDEFGHIJKLMNOPQRS
    AWS_SECRET_ACCESS_KEY: secretvalue4here
    AWS_STORAGE_BUCKET_NAME: blockstore-bucket

# Configure LMS/Studio to access Blockstore:
EDXAPP_BLOCKSTORE_API_URL: http://localhost:8250/api/v1/
EDXAPP_LMS_ENV_EXTRA:
    BLOCKSTORE_API_AUTH_TOKEN: secretvalue1here
    # ^ Corresponds to the token in the django admin at https://blockstore.openedx-example.opencraft.hosting/admin/authtoken/token/
EDXAPP_CMS_ENV_EXTRA:
    BLOCKSTORE_API_AUTH_TOKEN: secretvalue1here

edx.org prod uses the blockstore role that’s in edx/configuration, or at least they did last time I checked.

I answered this above; basically you need [BD-14]"Source from Library" XBlock by mavidser · Pull Request #24385 · openedx/edx-platform · GitHub which hasn’t merged yet.

Thanks for the debugging! I’m not sure what the workaround is, but I believe that role is being used for edx.org’s deployment, and we’re using it for some deployments. That variable appears to work similarly for several other services, so perhaps only specific ansible versions or something else in your environment is causing that bug?

6

Hmmm. Our use case for Blockstore was that we hoped we would be able to include essentially any content from a library in randomized content. In essence, what’s described here, but without the limitations of this, that is, only ‘html’, ‘problem’, ‘video’ being allowed in a library.

Would BD-14 allow us to do that?

If it does not, would Ramshackle?

Ugh! Looks like you may be right; thanks for the pointer. We were indeed running with the Ansible release that requirements.txt mandated for the Ironwood branch of edx-configuration, and neglected to upgrade Ansible on the deployment node. We’ll give 2.7 a shot. Thanks again!

7

The way that randomized content works in modulestore is a hack: it copies all of the library blocks into the course and then for each user, it randomly hides all but one of them. There are a number of inefficiencies and problems resulting from this approach. That’s why for the ongoing Blockstore Content Libraries work, we haven’t looked at implementing randomized content into modulestore just yet (instead we’re focusing on non-randomized content). Once we have Blockstore-based courses, then we’ll be able to randomly assign content from Blockstore content libraries in a way that’s much cleaner and more efficient.

That said, some other XBlocks actually work OK with the current modulestore libraries + randomization, but are not allowed because they haven’t been tested (and some blocks are known to have issues). If you want to test individual block types and enable them by modifying the code, you can, as described in my answers on Drag and drop problem type not available in content library. Just be aware that you might hit bugs, depending on what sort of APIs the XBlock in questions uses.

8

I agree, but correct me if I’m wrong if I deduce from the above that it is the more prudent approach to forgo a mixed-mode approach for now, and transition to Blockstore once both courses and libraries are available?

It would appear to me that adopting a mixed-mode approach now would entail transitioning to a couple of tools that are explicitly designed as stop-gaps: Ramshackle says it “is not intended to evolve into an end-user tool, but to be replaced by one;” the BD-14 PR description states that “once we get courses implemented in Blockstore as well I expect we won’t use this block at all, so I haven’t designed it to be forwards compatible in that sense”.

So that would mean planning two transitions: one to adopt Blockstore now (with the tools that are available today), and then moving to Blockstore-based courses and moving to a new toolkit when it’s available. That strikes me as somewhat less efficient for downstream operators than to stick with the present contentstore/modulestore and monolithic LMS/CMS, and plan their transition to Blockstore once it supports courses and a “final” set of authoring tools is available. Would you concur with that assessment?

Don’t get me wrong, I love the concept behind Blockstore and we’d prefer to migrate off of the old bits rather sooner than later; we’re just trying to assess what’s the best time to plan the transition. :slight_smile:

9

We know that lots of people are excited about the new possibilities that Blockstore unlocks, and that fully implementing Blockstore courses will take a long time, so we are planning a “fully supported” mixed mode; it’s just not quite there yet. The BD-14 project is currently building Blockstore library authoring into Studio, replacing Ramshackle as expected. You’ll be able to test that out this fall on master/devstacks and use it for library authoring in the Koa release.

Thinking about this a bit more, and re-reading my last answer, I realize that we should also try to get content randomization from Blockstore libraries into modulestore courses working this summer. Because then everyone can move all their library authoring/content into blockstore-based libraries, and they’ll be backwards-compatible with modulestore courses as well as forwards-compatible with future blockstore-based courses. We haven’t scheduled that (randomization) work just yet so I’m not promising we will, but it would make sense.

So my recommendation for the transition is:

  • Transition all library authoring to Blockstore this fall (if you track master) or with Koa (if you use named releases)
  • Transition course authoring to Blockstore when it becomes available
  • Invest developer time into helping with Blockstore functionality where possible

P.S. As far as I know, Blockstore libraries will offer all the same features as modulestore libraries (and much more) with one exception: a little-known and rarely used ability in modulestore libraries allows course authors to override some XBlock settings when using a library block in their course, and those overrides get preserved even when updating the block with a newer version from the library. Blockstore libraries probably won’t support that (unless we hear that that’s a big problem for anyone).

10

That’s helpful advice, thanks Braden! Appreciate the background info, too.

11

Thanks everyone! Lots of useful information here, much appreciated! :slight_smile:

12

FYI, I have opened a PR at https://github.com/edx/blockstore/pull/76 to add this information into the Blockstore README, which I should have done long ago.

2 Likes