TC_AUTHOR_60: STUDIO: CONTENT LIBRARIES: Library Sourced Content

[DeanJayMathew]

Summary of Issue:

In Studio we have two options to add components from a Content Library, namely (1) Library Sourced Content and (2) Randomized Content Block:

I can create Libraries in Studio Home fine. From the two options mentioned above the "Randomized Content Block" works fine, but I get a error when trying the other option "Library Sourced Content":

Could not fetch library
SyntaxError: Unexpected token '<', " <!doctype "... is not valid JSON

Steps to reproduce:

1. Add a Content Library in Studio Home.
2. Click "Library Sourced Component" at the unit level to see the two options:

3. Try to add a Library Sourced Component and click Edit to see the error. It also says an XBlock needs to be configured for this but I'm not sure which XBlock:

Expected Outcome: The Library Sourced Component should work.

Actual Outcome: It gives an error as described: Could not fetch library SyntaxError: Unexpected token '<', " <!doctype "... is not valid JSON

[DeanJayMathew]

label: olive testing

[DeanJayMathew]

label: release blocker

[mariajgrimaldi]

I tested this in my local environment using tutor olive and got a similar error. I'll attach the error from the CMS logs:

cms_1                        | 2022-11-29 11:43:29,228 ERROR 24 [django.request] [user None] [ip None] log.py:224 - Internal Server Error: /api/libraries/v2/
cms_1                        | Traceback (most recent call last):
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
cms_1                        |     response = get_response(request)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
cms_1                        |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
cms_1                        |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
cms_1                        |     return func(*args, **kwds)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
cms_1                        |     return view_func(*args, **kwargs)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
cms_1                        |     return self.dispatch(request, *args, **kwargs)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
cms_1                        |     response = self.handle_exception(exc)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
cms_1                        |     self.raise_uncaught_exception(exc)
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
cms_1                        |     raise exc
cms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
cms_1                        |     response = handler(request, *args, **kwargs)
cms_1                        |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
cms_1                        |     return func(*args, **kwds)
cms_1                        |   File "/openedx/edx-platform/openedx/core/djangoapps/content_libraries/views.py", line 180, in get
cms_1                        |     result = api.get_metadata_from_index(paginated_qs)
cms_1                        |   File "/openedx/edx-platform/openedx/core/djangoapps/content_libraries/api.py", line 313, in get_metadata_from_index
cms_1                        |     bundles = get_bundles(uuids=uuids, text_search=text_search)
cms_1                        |   File "/openedx/edx-platform/openedx/core/lib/blockstore_api/methods.py", line 47, in wrapper
cms_1                        |     return func(*args, **kwargs)
cms_1                        |   File "/openedx/edx-platform/openedx/core/lib/blockstore_api/methods.py", line 206, in get_bundles
cms_1                        |     response = api_request('get', version_url)
cms_1                        |   File "/openedx/edx-platform/openedx/core/lib/blockstore_api/methods.py", line 62, in api_request
cms_1                        |     raise ImproperlyConfigured("Cannot use Blockstore unless BLOCKSTORE_API_AUTH_TOKEN is set.")
cms_1                        | django.core.exceptions.ImproperlyConfigured: Cannot use Blockstore unless BLOCKSTORE_API_AUTH_TOKEN is set.

I'm unfamiliar with blockstore so I don't know how to configure this. Can we confirm we're getting the same error in the testing instance?

@regisb: can you help us check the logs? thanks!

[arbrandes]

@DeanJayMathew, I think the Library Sourced block is exclusive for use with Blockstore. I believe we can let this one pass, as I don't think anybody intends to use Blockstore in production, specifically not with Tutor.

[DeanJayMathew]

@arbrandes @mariajgrimaldi

Well noted, with thanks!

If it's not going to be used, should we not remove the "Library Sourced Content" from the Open edX® standard distribution? Is there a formal procedure to allow/block these rarely-used features from making their way into the standard distribution?

[regisb]

I can confirm that I'm seeing a similar error on the backend:

tutor_local-cms-1                        | 2022-11-29 17:42:05,431 ERROR 17 [django.request] [user 8] [ip 77.195.23.18] log.py:224 - Internal Server Error: /api/libraries/v2/
tutor_local-cms-1                        | Traceback (most recent call last):
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
tutor_local-cms-1                        |     response = get_response(request)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
tutor_local-cms-1                        |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
tutor_local-cms-1                        |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
tutor_local-cms-1                        |     return func(*args, **kwds)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
tutor_local-cms-1                        |     return view_func(*args, **kwargs)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
tutor_local-cms-1                        |     return self.dispatch(request, *args, **kwargs)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
tutor_local-cms-1                        |     response = self.handle_exception(exc)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
tutor_local-cms-1                        |     self.raise_uncaught_exception(exc)
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
tutor_local-cms-1                        |     raise exc
tutor_local-cms-1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
tutor_local-cms-1                        |     response = handler(request, *args, **kwargs)
tutor_local-cms-1                        |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
tutor_local-cms-1                        |     return func(*args, **kwds)
tutor_local-cms-1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/content_libraries/views.py", line 180, in get
tutor_local-cms-1                        |     result = api.get_metadata_from_index(paginated_qs)
tutor_local-cms-1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/content_libraries/api.py", line 313, in get_metadata_from_index
tutor_local-cms-1                        |     bundles = get_bundles(uuids=uuids, text_search=text_search)
tutor_local-cms-1                        |   File "/openedx/edx-platform/./openedx/core/lib/blockstore_api/methods.py", line 47, in wrapper
tutor_local-cms-1                        |     return func(*args, **kwargs)
tutor_local-cms-1                        |   File "/openedx/edx-platform/./openedx/core/lib/blockstore_api/methods.py", line 205, in get_bundles
tutor_local-cms-1                        |     version_url = api_url('bundles') + '?' + urlencode(query_params)
tutor_local-cms-1                        |   File "/openedx/edx-platform/./openedx/core/lib/blockstore_api/methods.py", line 53, in api_url
tutor_local-cms-1                        |     raise ImproperlyConfigured('BLOCKSTORE_API_URL must be set and should end with /api/v1/')
tutor_local-cms-1                        | django.core.exceptions.ImproperlyConfigured: BLOCKSTORE_API_URL must be set and should end with /api/v1/

I can verify that the setting is None:

$ tutor local exec cms ./manage.py cms shell
>>> from django.conf import settings
>>> print(settings.BLOCKSTORE_API_URL)
None

I have no idea why we are going through this particular piece of code when blockstore is not enabled. I'm taking the liberty to ping @bradenmacdonald, as the designated blockstore expert :)

[bradenmacdonald]

Using Library Sourced Block requires that you have Blockstore enabled. Blockstore now ships by default with Open edX so using it is merely a matter of setting the correct configuration, but it is not enabled out of the box. I am not sure why Library Source Content is enabled out of the box.

I think the correct fix is one of these two things:

• When you click the green "Library Content" button, you should only see "Library Sourced Content" if Blockstore/Libraries v2 are enabled. Or:
• When you add a Library Sourced Content block to the course, where it currently says "No XBlock has been configured", it should instead say "Blockstore is not enabled on this Open edX instance, so Library Sourced Content is unavailable."

I would prefer the latter approach as it's more clear and also makes admins aware of the feature that they could enable if they wanted to.

[bradenmacdonald]

BTW here is where to add that check and there is already an example of how to display the error message.

[regisb]

Blockstore now ships by default with Open edX so using it is merely a matter of setting the correct configuration, but it is not enabled out of the box.

If blockstore is stable and provides useful features, then I think that we should configure Tutor to actually use it. What do you think? Is it worth it? If I understand correctly, correctly configuring Blockstore is just a matter of setting BLOCKSTORE_USE_BLOCKSTORE_APP_API = True. Right?

[regisb]

It looks like it's working! I'm a bit moved, it's the first time I see blockstore in action in Tutor :) All I had to do was to set BLOCKSTORE_USE_BLOCKSTORE_APP_API = True and create the proper "blockstore" entry in CACHES.

What do you think, should we enable blockstore by default in Tutor? I think we should, if possible.

[DeanJayMathew]

Wow! Let me know if we should create a Test Case for Olive and get it through formal testing.

[regisb]

Let me know if we should create a Test Case for Olive and get it through formal testing.

I guess? we should? I don't know because I have no idea what features are included in Blockstore...

[arbrandes]

I don't consider myself a blockstore expert, but I have used it a lot in different contexts, and here's my take.

If blockstore is stable and provides useful features

Blockstore is pretty stable and provides very useful features, but only if you can use it: it's just a storage backend. As far as I know, the only way to do so is via the Library Sourced block we're discussing, and it hasn't seen activity for some time. The last feature-meaningful commit was over 2 years ago., and even then we knew that it was just a placeholder for a better solution down the line.

And (again, AFAIK) the only way to create content for the Library Sourced block is via frontend-app-library-authoring, and on that I can speak with some confidence: it's nowhere near being ready for production use.

correctly configuring Blockstore is just a matter of setting BLOCKSTORE_USE_BLOCKSTORE_APP_API = True

Blockstore was designed to use object storage. Last I checked, the configuration of the store was separate. We'd have to check that this is the case, and then make it work with Minio, or expose corresponding Tutor config variables to the user.

[bradenmacdonald]

@arbrandes

As far as I know, the only way to do so is via the Library Sourced block

There is a PR to make it work with the (existing) "Randomized" Library Content Block too, but it has been waiting for review since August: openedx/edx-platform#30895

Yes, frontend-app-library-authoring is extremely immature. The visual editor is mostly not usable and could be a source of frustration to users who expect it to work. That said, the OLX authoring workflow works pretty well for advanced users.

Blockstore was designed to use object storage. Last I checked, the configuration of the store was separate. We'd have to check that this is the case, and then make it work with Minio, or expose corresponding Tutor config variables to the user.

Like the rest of edx-platform, blockstore will use django file storage by default, and you can change it to use S3/MinIO if you want. As you point out, right now that would mean Tutor would have to set an additional blockstore-specific config var to make it use MinIO.

[kdmccormick]

My 2c:

"Library Sourced XBlock" only works with V2 libraries (the ones made by frontend-app-library-authoring, backed by blockstore). Blockstore is fine and stable, but V2 libraries are not. They cannot be authored using an instance running the default Tutor configuration.

I think it'd be very confusing to have Library Sourced XBlock shown by default, when the underlying content it is exposing (V2 library blocks) cannot be authored by default.

[kdmccormick]

Relevant: openedx/edx-platform#30803

[regisb]

@arbrandes For the record, and contrary to what you said in the BTR meetup today, this is a new issue in Olive: the "library content" component button did not appear in Nutmeg, as you can see here: https://studio.demo.openedx.overhang.io/container/block-v1:edX+DemoX+Demo_Course+type@vertical+block@vertical_0270f6de40fc

[kdmccormick]

@regisb I think was the expected behavior in Nutmeg: https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/exercises_tools/randomized_content_blocks.html#enable-content-libraries . Before Olive, one needed to go into course Advanced Settings and add library_content as an advanced component type.

The "Library Content" button is a new thing in Olive ✨

[kdmccormick]

@connorhaugh will look into this.

[regisb]

I commented here: openedx/edx-platform#30803 (comment)

tl;dr: we want to keep the "Randomized Content" but remove the "Library Source Content" xblock, right? Then I pushed a change to the Tutor settings to do just that. If this is an acceptable solution, then we can close this issue.

[arbrandes]

Works for me! @DeanJayMathew, can you confirm?

[connorhaugh]

Ahh that works for me as well I submitted a PR to fix it on the olive branch, but Let's not be redundant. I missed this issue thread. I'll close that PR. openedx/edx-platform#31405

[kdmccormick]

tl;dr: we want to keep the "Randomized Content" but remove the "Library Source Content" xblock, right? Then I pushed a change to the Tutor settings to do just that. If this is an acceptable solution, then we can close this issue.

could you link me to your change?

If we're in a rush then that works fine, but if we have the time to fix this upstream in edx-platform then IMO that'd be the best outcome.

[kdmccormick]

@regisb ^

[regisb]

@kdmccormick it's really the same patch as the one by @connorhaugh: https://github.com/overhangio/tutor/pull/734/files#diff-fc126cedbc6f2e010b4ccf860376e79778855985dab737cee6ff5b15f5bb0473

[kdmccormick]

@connorhaugh would you be able to make this change at the source instead? That is, could you remove library_sourced from settings.LIBRARY_BLOCK_TYPES in common/cms/envs.py ?

I think the strategy of "fix edx-platform's bad defaults at the Tutor layer" is OK as a stop-gap, but if we want edx-platform itself to be a good piece of software, then we need to get in the habit of fixing configuration issues at the source.

[DeanJayMathew]

Yes @arbrandes the following solution proposed by @regisb would resolve the issue from my perspective.

I commented here: openedx/edx-platform#30803 (comment)

tl;dr: we want to keep the "Randomized Content" but remove the "Library Source Content" xblock, right? Then I pushed a change to the Tutor settings to do just that. If this is an acceptable solution, then we can close this issue.

[kdmccormick]

@connorhaugh has merged a fix to master and now has an Olive backport available for review: openedx/edx-platform#31424

[arbrandes]

Another release blocker bites the dust! Thanks everybody!