Shareableviz extended support (#1749)

* refactor router to accept new deployer inputs (#1739) Signed-off-by: ravi-kumar-pilla <[email protected]> * Refactor Shareableviz CLI (#1740) * refactor router to accept new deployer inputs Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli for shareableviz deploy Signed-off-by: ravi-kumar-pilla <[email protected]> * merge router change Signed-off-by: ravi-kumar-pilla <[email protected]> * PR comments fix Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> * Kedro Viz Static Website hosting on Azure (#1708) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * resolve conflicts Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments Signed-off-by: ravi-kumar-pilla <[email protected]> * add back cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove duplicate pytest parameter Signed-off-by: ravi-kumar-pilla <[email protected]> * remove fsspec upper bound Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Kedro Viz Static Website Hosting on GCP (#1711) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for gcp Signed-off-by: ravi-kumar-pilla <[email protected]> * fix gcp pytest coverage Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * revert file permission change --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Sharable Kedro-Viz UI for AWS, GCP and Azure (#1718) * shareable viz with multiple platform UI * test fix * Relative path fix for user entered url * Test updated * Test updated * UI alignment and endpoint help text added. Signed-off-by: <> * Add Kedro-Viz documentation URLs and update modal content Signed-off-by: <> * Code review suggestions added Signed-off-by: <> * Refactor form field names in ShareableUrlModal component Signed-off-by: <> * Add cursor style to input field in shareable-url-modal.js Signed-off-by: <> * Add cursor style to input field in shareable-url-modal.js Signed-off-by: <> * Variable names updated. Signed-off-by: <> * add exception handlers Signed-off-by: ravi-kumar-pilla <[email protected]> * Disclaimer note added and shareable-url-modal component refactored by moving major jsx to other file Signed-off-by: <> * Wrapped the prop methods with useCallback Signed-off-by: <> * info text updated. Signed-off-by: <> * Test fixed Signed-off-by: <> * test fix and rearranging jsx Signed-off-by: <> * Cypress test fix Signed-off-by: <> * button text updated Signed-off-by: <> * Keeping button text as Publish for a link seetings Signed-off-by: <> --------- Signed-off-by: <> Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: ravi-kumar-pilla <[email protected]> * Kedro-viz doc link added in disclaimer note. Signed-off-by: <> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Signed-off-by: <> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]>
kedro-org · Feb 26, 2024 · af08f9c · af08f9c
1 parent 4ed8b76
commit af08f9c
Show file tree

Hide file tree

Showing 47 changed files with 1,510 additions and 672 deletions.
diff --git a/RELEASE.md b/RELEASE.md
@@ -5,6 +5,11 @@ Please follow the established format:
 - Use present tense (e.g. 'Add new feature')
 - Include the ID number for the related PR (or PRs) in parentheses
 -->
+# Upcoming Release
+
+## Major features and improvements
+
+- Extend support for 'Publishing and sharing Kedro-viz' to Azure and GCP. (#1708, #1711)
 
 # Upcoming release
 

diff --git a/cypress/fixtures/mock/deploySuccessResponse.json b/cypress/fixtures/mock/deploySuccessResponse.json
@@ -1,4 +1,4 @@
 {
-  "message": "Website deployed on S3",
-  "url": "http://myBucketName.s3-website.us-east-1.amazonaws.com"
+  "message": "Website deployed on AWS",
+  "url": "http://www.example.com"
 }
diff --git a/cypress/tests/ui/flowchart/shareable-urls.cy.js b/cypress/tests/ui/flowchart/shareable-urls.cy.js
@@ -10,6 +10,7 @@ describe('Shareable URLs', () => {
     // Action
     cy.reload();
     cy.get('.pipeline-menu-button--deploy').click({ force: true });
+    cy.get('[data-test="disclaimerButton"]').click({ force: true });
 
     // Assert after action
     cy.get('.shareable-url-modal .modal__wrapper').contains(
@@ -28,6 +29,7 @@ describe('Shareable URLs', () => {
     // Action
     cy.reload();
     cy.get('.pipeline-menu-button--deploy').click({ force: true });
+    cy.get('[data-test="disclaimerButton"]').click({ force: true });
 
     // Assert after action
     cy.get('.shareable-url-modal .modal__wrapper').contains(
@@ -38,6 +40,7 @@ describe('Shareable URLs', () => {
   it('verifies that shareable url modal closes on close button click #TC-54', () => {
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains('Cancel')
       .click();
@@ -49,65 +52,73 @@ describe('Shareable URLs', () => {
     );
   });
 
-  it('verifies that users can click on region dropdown and see all region options #TC-55', () => {
-    const regionCount = 30;
+  it('verifies that users can click on platform dropdown and see all platform options #TC-55', () => {
+    const platformCount = 3;
 
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
 
     // Assert after action
     cy.get('.shareable-url-modal .menu-option').should(
       'have.length',
-      regionCount
+      platformCount
     );
   });
 
-  it('verifies that publish button should be disabled when region is not selected and bucket name is empty #TC-56', () => {
-    const selectedRegion = 'Select a region';
+  it('verifies that publish button should be disabled when platform is not selected and bucket name & endpoint name are empty #TC-56', () => {
+    const selectedPlatform = 'Select a hosting platform';
     const primaryButtonNodeText = 'Publish';
 
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
 
     // Assert after action
     cy.get(
       '.shareable-url-modal [data-test=kedro-pipeline-selector] .dropdown__label span'
-    ).contains(selectedRegion);
-    cy.get('.shareable-url-modal textarea').should('have.value', '');
+    ).contains(selectedPlatform);
+    cy.get('.shareable-url-modal input').should('have.value', '');
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains(primaryButtonNodeText)
       .should('be.disabled');
   });
 
-  it('verifies that publish button should be disabled when a bucket region is selected and bucket name is empty #TC-57', () => {
+  it('verifies that publish button should be disabled when a platform is selected and bucket name is empty #TC-57', () => {
     const primaryButtonNodeText = 'Publish';
 
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
     cy.get('.shareable-url-modal .dropdown__options section div')
       .first()
       .click();
 
     // Assert after action
-    cy.get('.shareable-url-modal textarea').should('have.value', '');
+    cy.get('.shareable-url-modal input').should('have.value', '');
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains(primaryButtonNodeText)
       .should('be.disabled');
   });
 
-  it('verifies that publish button should be enabled when region is selected and bucket name is not empty #TC-58', () => {
+  it('verifies that publish button should be enabled when platform is selected and bucket name & endpoint name are not empty #TC-58', () => {
+    const endpointName = 'http://www.example.com';
     const bucketName = 'myBucketName';
     const primaryButtonNodeText = 'Publish';
 
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
     cy.get('.shareable-url-modal .dropdown__options section div')
       .first()
       .click();
-    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="bucket_name"]').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="endpoint_name"]').type(
+      endpointName
+    );
 
     // Assert after action
     cy.get('.shareable-url-modal__button-wrapper button')
@@ -116,17 +127,20 @@ describe('Shareable URLs', () => {
   });
 
   it('verifies that error message appears with wrong inputs on publish button click #TC-59', () => {
+    const endpointName = 'http://www.example.com';
     const bucketName = 'myBucketName';
     const primaryButtonNodeText = 'Publish';
-    const errorButtonNodeText = 'Go back';
-
     // Action
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
     cy.get('.shareable-url-modal .dropdown__options section div')
       .first()
       .click();
-    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="bucket_name"]').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="endpoint_name"]').type(
+      endpointName
+    );
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains(primaryButtonNodeText)
       .click();
@@ -135,11 +149,11 @@ describe('Shareable URLs', () => {
     cy.get('.shareable-url-modal .modal__wrapper').contains(
       'Something went wrong. Please try again later.'
     );
-    cy.get('.shareable-url-modal__error button').contains(errorButtonNodeText);
   });
 
   it('verifies that AWS link is generated with correct inputs on publish button click #TC-60', () => {
     const bucketName = 'myBucketName';
+    const endpointName = 'http://www.example.com';
     const primaryButtonNodeText = 'Publish';
 
     // Intercept the network request to mock with a fixture
@@ -152,11 +166,15 @@ describe('Shareable URLs', () => {
     // Action
     cy.reload();
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
     cy.get('.shareable-url-modal .dropdown__options section div')
       .first()
       .click();
-    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="bucket_name"]').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="endpoint_name"]').type(
+      endpointName
+    );
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains(primaryButtonNodeText)
       .click();
@@ -172,8 +190,9 @@ describe('Shareable URLs', () => {
 
   it('verifies that AWS link is generated with correct inputs on Republish button click #TC-61', () => {
     const bucketName = 'myBucketName';
+    const endpointName = 'http://www.example.com';
     const primaryButtonNodeText = 'Publish';
-    const primaryButtonNodeTextVariant = 'Republish';
+    const primaryButtonNodeTextVariant = 'Publish';
     const secondaryButtonNodeText = 'Link Settings';
 
     // Intercept the network request to mock with a fixture
@@ -186,11 +205,15 @@ describe('Shareable URLs', () => {
     // Action
     cy.reload();
     cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('[data-test="disclaimerButton"]').click();
     cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
     cy.get('.shareable-url-modal .dropdown__options section div')
       .first()
       .click();
-    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="bucket_name"]').type(bucketName);
+    cy.get('.shareable-url-modal [data-test="endpoint_name"]').type(
+      endpointName
+    );
     cy.get('.shareable-url-modal__button-wrapper button')
       .contains(primaryButtonNodeText)
       .click();

diff --git a/package/kedro_viz/api/rest/requests.py b/package/kedro_viz/api/rest/requests.py
@@ -1,9 +1,11 @@
 """`kedro_viz.api.rest.requests` defines REST request types."""
+
 from pydantic import BaseModel
 
 
-class S3DeployerConfiguration(BaseModel):
-    """Credentials for S3 Deployer."""
+class DeployerConfiguration(BaseModel):
+    """Credentials for Deployers."""
 
-    region: str
+    platform: str
+    endpoint: str
     bucket_name: str
diff --git a/package/kedro_viz/api/rest/responses.py b/package/kedro_viz/api/rest/responses.py
@@ -5,12 +5,10 @@
 import logging
 from typing import Any, Dict, List, Optional, Union
 
-import fsspec
 import orjson
 import packaging
 from fastapi.encoders import jsonable_encoder
 from fastapi.responses import JSONResponse, ORJSONResponse
-from kedro.io.core import get_protocol_and_path
 from pydantic import BaseModel, ConfigDict
 
 from kedro_viz.api.rest.utils import get_package_version
@@ -443,22 +441,19 @@ def save_api_pipeline_response_to_fs(pipelines_path: str, remote_fs: Any):
             raise exc
 
 
-def save_api_responses_to_fs(api_dir: str):
+def save_api_responses_to_fs(path: str, remote_fs: Any):
     """Saves all Kedro Viz API responses to a directory."""
     try:
-        protocol, path = get_protocol_and_path(str(api_dir))
-        remote_fs = fsspec.filesystem(protocol)
-
         logger.debug(
             """Saving/Uploading api files to %s""",
-            api_dir,
+            path,
         )
 
         main_path = f"{path}/api/main"
         nodes_path = f"{path}/api/nodes"
         pipelines_path = f"{path}/api/pipelines"
 
-        if protocol == "file":
+        if "file" in remote_fs.protocol:
             remote_fs.makedirs(path, exist_ok=True)
             remote_fs.makedirs(nodes_path, exist_ok=True)
             remote_fs.makedirs(pipelines_path, exist_ok=True)

diff --git a/package/kedro_viz/api/rest/router.py b/package/kedro_viz/api/rest/router.py
@@ -1,12 +1,13 @@
 """`kedro_viz.api.rest.router` defines REST routes and handling logic."""
+
 # pylint: disable=missing-function-docstring, broad-exception-caught
 import logging
 
 from fastapi import APIRouter
 from fastapi.responses import JSONResponse
 
-from kedro_viz.api.rest.requests import S3DeployerConfiguration
-from kedro_viz.integrations.deployment.s3_deployer import S3Deployer
+from kedro_viz.api.rest.requests import DeployerConfiguration
+from kedro_viz.integrations.deployment.deployer_factory import DeployerFactory
 
 from .responses import (
     APIErrorMessage,
@@ -19,6 +20,11 @@
     get_selected_pipeline_response,
 )
 
+try:
+    from azure.core.exceptions import ServiceRequestError
+except ImportError:  # pragma: no cover
+    ServiceRequestError = None  # type: ignore
+
 logger = logging.getLogger(__name__)
 
 router = APIRouter(
@@ -50,17 +56,33 @@ async def get_single_pipeline_data(registered_pipeline_id: str):
 
 
 @router.post("/deploy")
-async def deploy_kedro_viz(input_values: S3DeployerConfiguration):
+async def deploy_kedro_viz(input_values: DeployerConfiguration):
     try:
-        deployer = S3Deployer(input_values.region, input_values.bucket_name)
-        url = deployer.deploy_and_get_url()
-        response = {"message": "Website deployed on S3", "url": url}
+        deployer = DeployerFactory.create_deployer(
+            input_values.platform, input_values.endpoint, input_values.bucket_name
+        )
+        deployer.deploy()
+        response = {
+            "message": "Website deployed on "
+            f"{input_values.platform and input_values.platform.upper()}",
+            "url": input_values.endpoint,
+        }
         return JSONResponse(status_code=200, content=response)
     except PermissionError as exc:  # pragma: no cover
         logger.exception("Permission error in deploying Kedro Viz : %s ", exc)
         return JSONResponse(
             status_code=401, content={"message": "Please provide valid credentials"}
         )
+    except (
+        # pylint: disable=catching-non-exception
+        (FileNotFoundError, ServiceRequestError)
+        if ServiceRequestError is not None
+        else FileNotFoundError
+    ) as exc:  # pragma: no cover
+        logger.exception("FileNotFoundError while deploying Kedro Viz : %s ", exc)
+        return JSONResponse(
+            status_code=400, content={"message": "The specified bucket does not exist"}
+        )
     except Exception as exc:  # pragma: no cover
         logger.exception("Deploying Kedro Viz failed: %s ", exc)
         return JSONResponse(status_code=500, content={"message": f"{exc}"})

diff --git a/package/kedro_viz/constants.py b/package/kedro_viz/constants.py
@@ -1,4 +1,5 @@
 """`kedro_viz.constants` defines constants to be used throughout the application."""
+
 import kedro
 from packaging.version import parse
 
@@ -9,37 +10,6 @@
 DEFAULT_HOST = "127.0.0.1"
 DEFAULT_PORT = 4141
 
-VIZ_DEPLOY_TIME_LIMIT = 60
+VIZ_DEPLOY_TIME_LIMIT = 300
 
-AWS_REGIONS = [
-    "us-east-2",
-    "us-east-1",
-    "us-west-1",
-    "us-west-2",
-    "af-south-1",
-    "ap-east-1",
-    "ap-south-2",
-    "ap-southeast-3",
-    "ap-southeast-4",
-    "ap-south-1",
-    "ap-northeast-3",
-    "ap-northeast-2",
-    "ap-southeast-1",
-    "ap-southeast-2",
-    "ap-northeast-1",
-    "ca-central-1",
-    "cn-north-1",
-    "cn-northwest-1",
-    "eu-central-1",
-    "eu-west-1",
-    "eu-west-2",
-    "eu-south-1",
-    "eu-west-3",
-    "eu-north-1",
-    "eu-south-2",
-    "eu-central-2",
-    "sa-east-1",
-    "me-south-1",
-    "me-central-1",
-    "il-central-1",
-]
+SHAREABLEVIZ_SUPPORTED_PLATFORMS = ["aws", "azure", "gcp"]