Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add import taxonomy endpoint #33663

Conversation

rpenido
Copy link
Contributor

@rpenido rpenido commented Nov 6, 2023

Description

Add a rest api endpoint to import taxonomies

Supporting Information

Testing instructions

  • Please ensure that the tests cover the expected behavior of the view

Private-ref: FAL-3532

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Nov 6, 2023
@openedx-webhooks
Copy link

openedx-webhooks commented Nov 6, 2023

Thanks for the pull request, @rpenido! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

  • supporting documentation
  • Open edX discussion forum threads
  • timeline information ("this must be merged by XX date", and why that is)
  • partner information ("this is a course on edx.org")
  • any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

@rpenido rpenido changed the title feat: add import taxonomy endpoint (edx-platform) feat: add import taxonomy endpoint Nov 6, 2023
@open-craft-grove
Copy link

Sandbox deployment started.

1 similar comment
@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

Copy link
Contributor

@pomegranited pomegranited left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My apologies @rpenido , I neglected to send this review! My comments are mainly about aligning this to your work on openedx/openedx-learning#112 (review).

Comment on lines 70 to 106
@action(detail=False, url_path="import", methods=["post"])
def create_import(self, request: Request, **_kwargs) -> HttpResponse:
"""
Creates a new taxonomy and imports the tags from the uploaded file.
"""
perm = "oel_tagging.import_taxonomy"
if not request.user.has_perm(perm):
raise PermissionDenied("You do not have permission to import taxonomies")

body = TaxonomyImportNewBodySerializer(data=request.data)
body.is_valid(raise_exception=True)

taxonomy_name = body.validated_data["taxonomy_name"]
taxonomy_description = body.validated_data["taxonomy_description"]
file = body.validated_data["file"].file
parser_format = body.validated_data["parser_format"]

# ToDo: This code is temporary
# In the future, the orgs parameter will be defined in the request body from the frontend
# See: https://github.com/openedx/modular-learning/issues/116
if oel_tagging_rules.is_taxonomy_admin(request.user):
orgs = None
else:
orgs = get_admin_orgs(request.user)

taxonomy = create_taxonomy(taxonomy_name, taxonomy_description, orgs=orgs)
try:
import_success = import_tags(taxonomy, file, parser_format)

if import_success:
return HttpResponse(status=200)
else:
import_error = get_last_import_log(taxonomy)
taxonomy.delete()
return HttpResponseBadRequest(import_error)
except ValueError as e:
return HttpResponseBadRequest(e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the parent TaxonomyView returns the created taxonomy, we can do this with less duplicated code:

Suggested change
@action(detail=False, url_path="import", methods=["post"])
def create_import(self, request: Request, **_kwargs) -> HttpResponse:
"""
Creates a new taxonomy and imports the tags from the uploaded file.
"""
perm = "oel_tagging.import_taxonomy"
if not request.user.has_perm(perm):
raise PermissionDenied("You do not have permission to import taxonomies")
body = TaxonomyImportNewBodySerializer(data=request.data)
body.is_valid(raise_exception=True)
taxonomy_name = body.validated_data["taxonomy_name"]
taxonomy_description = body.validated_data["taxonomy_description"]
file = body.validated_data["file"].file
parser_format = body.validated_data["parser_format"]
# ToDo: This code is temporary
# In the future, the orgs parameter will be defined in the request body from the frontend
# See: https://github.com/openedx/modular-learning/issues/116
if oel_tagging_rules.is_taxonomy_admin(request.user):
orgs = None
else:
orgs = get_admin_orgs(request.user)
taxonomy = create_taxonomy(taxonomy_name, taxonomy_description, orgs=orgs)
try:
import_success = import_tags(taxonomy, file, parser_format)
if import_success:
return HttpResponse(status=200)
else:
import_error = get_last_import_log(taxonomy)
taxonomy.delete()
return HttpResponseBadRequest(import_error)
except ValueError as e:
return HttpResponseBadRequest(e)
@action(detail=False, url_path="import", methods=["post"])
def create_import(self, request: Request, **kwargs) -> HttpResponse:
"""
Creates a new taxonomy with the given orgs and imports the tags from the uploaded file.
"""
response = super().create_import(request, **kwargs)
# If creation was successful, set the orgs for the new taxonomy
if status.is_success(response.status_code):
# ToDo: This code is temporary
# In the future, the orgs parameter will be defined in the request body from the frontend
# See: https://github.com/openedx/modular-learning/issues/116
if oel_tagging_rules.is_taxonomy_admin(request.user):
orgs = None
else:
orgs = get_admin_orgs(request.user)
taxonomy = get_taxonomy(response.data['id']) # or wherever it is
set_taxonomy_orgs(taxonomy, all_orgs=False, orgs=orgs)
# might need to re-serialize the response to include the orgs?
return response

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this one @pomegranited.

We are preventing some code duplication in the View but duplicating the logic of the edx-platform/create_taxonomy.

It will be a pain if our create_taxonomy gets complicated. Or we will get in trouble if we change some behavior from the oel_tagging/create_taxonomy.

Do you think it is ok to leave the duplicated create_import?

Copy link
Contributor

@pomegranited pomegranited Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rpenido I don't think we should be writing/maintaining more code than we need..

@bradenmacdonald , could we get a 2nd opinion here?

Copy link
Contributor

@bradenmacdonald bradenmacdonald Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like @pomegranited's suggestion; it re-uses more code. If we're going to have a separate "create_import" function at all, we should aim to keep it as small as possible, just calling the other create and import functions.

The content_tagging version of create_taxonomy is only two lines of code so it's fine to duplicate it here. Just make sure to add comments explaining everything and any concerns about that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the discussion! 😄

Done here: 1b65728

Comment on lines 107 to 142

@action(detail=True, url_path="tags/import", methods=["put"])
def update_import(self, request: Request, **_kwargs) -> HttpResponse:
"""
Creates a new taxonomy and imports the tags from the uploaded file.
"""
perm = "oel_tagging.import_taxonomy"
if not request.user.has_perm(perm):
raise PermissionDenied("You do not have permission to import taxonomies")

body = TaxonomyImportBodySerializer(data=request.data)
body.is_valid(raise_exception=True)

file = body.validated_data["file"].file
parser_format = body.validated_data["parser_format"]

# ToDo: This code is temporary
# In the future, the orgs parameter will be defined in the request body from the frontend
# See: https://github.com/openedx/modular-learning/issues/116
if oel_tagging_rules.is_taxonomy_admin(request.user):
orgs = None
else:
orgs = get_admin_orgs(request.user)

taxonomy = self.get_object()
try:
import_success = import_tags(taxonomy, file, parser_format)

if import_success:
return HttpResponse(status=200)
else:
import_error = get_last_import_log(taxonomy)
return HttpResponseBadRequest(import_error)
except ValueError as e:
return HttpResponseBadRequest(e)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this whole method can go away? I think the parent TaxonomyView now handles all that's needed here.

Suggested change
@action(detail=True, url_path="tags/import", methods=["put"])
def update_import(self, request: Request, **_kwargs) -> HttpResponse:
"""
Creates a new taxonomy and imports the tags from the uploaded file.
"""
perm = "oel_tagging.import_taxonomy"
if not request.user.has_perm(perm):
raise PermissionDenied("You do not have permission to import taxonomies")
body = TaxonomyImportBodySerializer(data=request.data)
body.is_valid(raise_exception=True)
file = body.validated_data["file"].file
parser_format = body.validated_data["parser_format"]
# ToDo: This code is temporary
# In the future, the orgs parameter will be defined in the request body from the frontend
# See: https://github.com/openedx/modular-learning/issues/116
if oel_tagging_rules.is_taxonomy_admin(request.user):
orgs = None
else:
orgs = get_admin_orgs(request.user)
taxonomy = self.get_object()
try:
import_success = import_tags(taxonomy, file, parser_format)
if import_success:
return HttpResponse(status=200)
else:
import_error = get_last_import_log(taxonomy)
return HttpResponseBadRequest(import_error)
except ValueError as e:
return HttpResponseBadRequest(e)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done ea67de1!

openedx/core/djangoapps/content_tagging/rules.py Outdated Show resolved Hide resolved
@rpenido rpenido force-pushed the rpenido/fal-3532-import-taxonomy-bare-bones branch from 484af49 to 1b65728 Compare November 10, 2023 12:42
@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

1 similar comment
@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

@rpenido
Copy link
Contributor Author

rpenido commented Nov 10, 2023

Hi @bradenmacdonald!

This is ready for a final review and merge.

I added 3c1a9d0 to fix the constraint update in our lint upstream (openedx/edx-lint#376) that was preventing the tests from passing.

If you checkout master and run make compile-requirements you can see the following diff:

diff --git a/requirements/common_constraints.txt b/requirements/common_constraints.txt
index d1be149855..53dff4a22d 100644
--- a/requirements/common_constraints.txt
+++ b/requirements/common_constraints.txt
@@ -24,7 +24,3 @@ Django<4.0
 elasticsearch<7.14.0
 
 # django-simple-history>3.0.0 adds indexing and causes a lot of migrations to be affected
-
-# tox>4.0.0 isn't yet compatible with many tox plugins, causing CI failures in almost all repos.
-# Details can be found in this discussion: https://github.com/tox-dev/tox/discussions/1810
-tox<4.0.0
diff --git a/requirements/edx/development.txt b/requirements/edx/development.txt
index 59637f095e..6b3f653618 100644
--- a/requirements/edx/development.txt
+++ b/requirements/edx/development.txt
@@ -2037,7 +2037,6 @@ tomlkit==0.12.1
     #   snowflake-connector-python
 tox==3.28.0
     # via
-    #   -c requirements/edx/../common_constraints.txt
     #   -r requirements/edx/testing.txt
     #   tox-battery
 tox-battery==0.6.2
diff --git a/requirements/edx/testing.txt b/requirements/edx/testing.txt
index 1fbf98b980..8f62ae7b7f 100644
--- a/requirements/edx/testing.txt
+++ b/requirements/edx/testing.txt
@@ -1502,7 +1502,6 @@ tomlkit==0.12.1
     #   snowflake-connector-python
 tox==3.28.0
     # via
-    #   -c requirements/edx/../common_constraints.txt
     #   -r requirements/edx/testing.in
     #   tox-battery
 tox-battery==0.6.2

This seems to be handled here:

@rpenido rpenido marked this pull request as ready for review November 10, 2023 16:40
@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox deployment started.

@open-craft-grove
Copy link

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

Copy link
Contributor

@bradenmacdonald bradenmacdonald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I can merge on Monday.

@bradenmacdonald
Copy link
Contributor

@rpenido I was going to merge this now but it will need a rebase first (or merge master into it).

@bradenmacdonald bradenmacdonald merged commit 034d632 into openedx:master Nov 14, 2023
64 checks passed
@bradenmacdonald bradenmacdonald deleted the rpenido/fal-3532-import-taxonomy-bare-bones branch November 14, 2023 19:12
@openedx-webhooks
Copy link

@rpenido 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

@edx-pipeline-bot
Copy link
Contributor

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

@edx-pipeline-bot
Copy link
Contributor

2U Release Notice: This PR has been deployed to the edX production environment.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

7 similar comments
@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

@open-craft-grove
Copy link

Sandbox update request received. Deployment will start soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
open-source-contribution PR author is not from Axim or 2U
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

6 participants