Make Kedro-Viz shareable via a hosted URL (#1487)

* nodes working Signed-off-by: Rashida Kanchwala <[email protected]> * added pipeline Signed-off-by: Rashida Kanchwala <[email protected]> * fix path Signed-off-by: Rashida Kanchwala <[email protected]> * fix bug and lint Signed-off-by: Rashida Kanchwala <[email protected]> * fix lint Signed-off-by: Rashida Kanchwala <[email protected]> * Add deploy button, modal, and form fields Signed-off-by: Tynan DeBold <[email protected]> * Fix failing tests Signed-off-by: Tynan DeBold <[email protected]> * Fix failing tests Signed-off-by: Tynan DeBold <[email protected]> * Update single file to S3 via file input Signed-off-by: Tynan DeBold <[email protected]> * Update test case Signed-off-by: Tynan DeBold <[email protected]> * Remove package lock changes Signed-off-by: Tynan DeBold <[email protected]> * Hide exp tracking and deploy button when not running Viz locally; remove WS/subscription code Signed-off-by: Tynan DeBold <[email protected]> * Add the beginnings of a /deploy REST endpoint Signed-off-by: Tynan DeBold <[email protected]> * Trying to send credentials to the backend Signed-off-by: Tynan DeBold <[email protected]> * update responses * test * Update UI to remove unneeded fields Signed-off-by: Tynan DeBold <[email protected]> * Fix failing text Signed-off-by: Tynan DeBold <[email protected]> * Workable Prototype * fix rebase * fixing stuff * Successfully returning a 200 from /api/deploy endpoint Signed-off-by: Tynan DeBold <[email protected]> * Remove aws sdk package Signed-off-by: Tynan DeBold <[email protected]> * backend fixes * remove map files when uploading * fix issue on html folder * UI updates for loading, success, error states; resetting modal Signed-off-by: Tynan DeBold <[email protected]> * fix lowerbound on fsspec * refactored to latest * some refactor * fix lint - WIP * refactor work * fix lint * remove old code * further refactor * add error handling and debugging * fix based on review * modify upload static files logic * refactor upload api with latest fsspec * fix unit tests_1 * revert os logic to pathlib * fix static folder issue * fix format and lint errors Signed-off-by: ravi-kumar-pilla <[email protected]> * add unit tests for shareable viz s3deployer Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytests for responses module Signed-off-by: ravi-kumar-pilla <[email protected]> * add s3fs as dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * add temporary no cover for apps Signed-off-by: ravi-kumar-pilla <[email protected]> * update lower reqs * update fsspec * check kedro latest version as 18.0 in e2e tests Signed-off-by: ravi-kumar-pilla <[email protected]> * update fsspec and s3fs requirements to support earliest kedro version Signed-off-by: ravi-kumar-pilla <[email protected]> * add timestamp file for deploy Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for timestamp route Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint and format errors Signed-off-by: ravi-kumar-pilla <[email protected]> * fix server changes and test e2e scenarios Signed-off-by: ravi-kumar-pilla <[email protected]> * try to catch versionConflicterror * Style updates; sending updated args to the backend Signed-off-by: Tynan DeBold <[email protected]> * Re-deploy flow Signed-off-by: Tynan DeBold <[email protected]> * add route /api/project-metadata to provide package version info Signed-off-by: ravi-kumar-pilla <[email protected]> * remove frontend build for backend unit tests Signed-off-by: ravi-kumar-pilla <[email protected]> * remove s3fs requirement to test Signed-off-by: ravi-kumar-pilla <[email protected]> * add s3fs without specific version Signed-off-by: ravi-kumar-pilla <[email protected]> * adjust requirements and add pytests for project metadata Signed-off-by: ravi-kumar-pilla <[email protected]> * test open s3fs requirement Signed-off-by: ravi-kumar-pilla <[email protected]> * test open s3fs requirement Signed-off-by: ravi-kumar-pilla <[email protected]> * test open s3fs requirement Signed-off-by: ravi-kumar-pilla <[email protected]> * Move share viz button to new location; add download icon Signed-off-by: Tynan DeBold <[email protected]> * Update tests Signed-off-by: Tynan DeBold <[email protected]> * Add shareable timestamp component Signed-off-by: Tynan DeBold <[email protected]> * Update timestamp fetch Signed-off-by: Tynan DeBold <[email protected]> * add version info and modify route name from /api/timestamp to /api/deploy-viz-metadata Signed-off-by: ravi-kumar-pilla <[email protected]> * Rename file; add version to timestamp Signed-off-by: Tynan DeBold <[email protected]> * Add back correct isRunningLocally check Signed-off-by: Tynan DeBold <[email protected]> * Remove console.log and update endpoint url Signed-off-by: Tynan DeBold <[email protected]> * add pytests for updated api Signed-off-by: ravi-kumar-pilla <[email protected]> * undo all new requirements * undo all fsspec changes * added s3fs as dependency * fix unit tests * clean up tests * lint * fix lint * fix test and lint and compatibility response * add packaging * packaging reqs * Consume compatibility endpoint; use Dropdown for bucket regions Signed-off-by: Tynan DeBold <[email protected]> * Update failing test Signed-off-by: Tynan DeBold <[email protected]> * Run format-fix Signed-off-by: Tynan DeBold <[email protected]> * Remove unused import Signed-off-by: Tynan DeBold <[email protected]> * fix lint and router link * Update API endpoint Signed-off-by: Tynan DeBold <[email protected]> * Add try/catch; remove console.log Signed-off-by: Tynan DeBold <[email protected]> * Update e2e test Signed-off-by: Tynan DeBold <[email protected]> * Update e2e test, again Signed-off-by: Tynan DeBold <[email protected]> * Drodown component update Signed-off-by: Tynan DeBold <[email protected]> * fix api endpoint * fixes based on reviews * fixes based on reviews * changes based on reviews * fix lint * shareable URL modal UI Fixes (#1537) * shareable url modal UI fix * Build error fix * shareable URL modal form validation fix * fixes based on review * updated cli help definition * update filpath to directory * update filpath to directory * Add the first of the Cypress tests Signed-off-by: Tynan DeBold <[email protected]> * add s3 protocol in the backend * leftover from merge fix * Changing language from deploy to publish Signed-off-by: Tynan DeBold <[email protected]> * Update wording again; return a better error message to the FE; handle the error; move s3 bucket regions to config file Signed-off-by: Tynan DeBold <[email protected]> * Add go back button to error modal stategs Signed-off-by: Tynan DeBold <[email protected]> * Update tests; use fstrings; remove unused CSS Signed-off-by: Tynan DeBold <[email protected]> * E2E cypress test for Shareable URL modal (#1543) * E2E cypress test added for shareable * Renaming from "Deploy" to "Publish" * Code review changes added * update pip fsspec * Update UI with docs links; update feature hits Signed-off-by: Tynan DeBold <[email protected]> * Update feature hints for when Viz is deployed Signed-off-by: Tynan DeBold <[email protected]> * Update release notes Signed-off-by: Tynan DeBold <[email protected]> * Update release notes again Signed-off-by: Tynan DeBold <[email protected]> * Add tracking to create_api_app_from_file Signed-off-by: Tynan DeBold <[email protected]> * fix e2e tests * Revert changes to create_api_app_from_file function Signed-off-by: Tynan DeBold <[email protected]> * Fix for telemetry in sharaeable viz (#1551) * fix telemetry for sharaeable viz Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint issue Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Signed-off-by: Tynan DeBold <[email protected]> Co-authored-by: Tynan DeBold <[email protected]> * Design QA changes Signed-off-by: Tynan DeBold <[email protected]> * Update release file Signed-off-by: Tynan DeBold <[email protected]> * PR review fixes Signed-off-by: Tynan DeBold <[email protected]> * Add back graphql subscription (removal moved to another PR) Signed-off-by: Tynan DeBold <[email protected]> * Update cypress test Signed-off-by: Tynan DeBold <[email protected]> --------- Signed-off-by: Rashida Kanchwala <[email protected]> Signed-off-by: Tynan DeBold <[email protected]> Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Rashida Kanchwala <[email protected]> Co-authored-by: Rashida Kanchwala <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> Co-authored-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]>
kedro-org · Oct 10, 2023 · a185d5a · a185d5a
1 parent cb07ab3
commit a185d5a
Show file tree

Hide file tree

Showing 38 changed files with 1,067 additions and 76 deletions.
diff --git a/.gitignore b/.gitignore
@@ -21,7 +21,7 @@ kedro-viz.tgz
 # e2e testing
 cypress/videos
 cypress/screenshots
-cypress/downloads/ 
+cypress/downloads/
 
 # production
 build/
@@ -65,4 +65,3 @@ coverage.xml
 
 # Kedro
 *.log
-
diff --git a/RELEASE.md b/RELEASE.md
@@ -8,17 +8,16 @@ Please follow the established format:
 
 # Release 6.6.0
 
-## Bug fixes and other changes
-
-- Fix for Kedro Viz Connection Error. (#1507)
-- Fix display of modular pipeline nodes that are associated with tags. (#1542)
-- Remove GraphQL subscription. (#1554)
+## Major features and improvements
 
-# Release 6.5.1
+- Make Kedro-Viz shareable via a hosted URL. (#1487)
 
 ## Bug fixes and other changes
 
 - Updated dependencies to ensure compatibility with Vite and Next.js environments; combine CSS into a single file when used as a React component. (#1510)
+- Fix for Kedro Viz Connection Error. (#1507)
+- Fix display of modular pipeline nodes that are associated with tags. (#1542)
+- Remove GraphQL subscription. (#1554)
 
 # Release 6.5.0
 

diff --git a/cypress/fixtures/mock/deploySuccessResponse.json b/cypress/fixtures/mock/deploySuccessResponse.json
@@ -0,0 +1,4 @@
+{
+  "message": "Website deployed on S3",
+  "url": "http://myBucketName.s3-website.us-east-1.amazonaws.com"
+}
diff --git a/cypress/fixtures/mock/package-compatibilities-compatible.json b/cypress/fixtures/mock/package-compatibilities-compatible.json
@@ -0,0 +1,5 @@
+{
+  "package_name": "fsspec",
+  "package_version": "2023.9.1",
+  "is_compatible": true
+}
diff --git a/cypress/fixtures/mock/package-compatibilities-incompatible.json b/cypress/fixtures/mock/package-compatibilities-incompatible.json
@@ -0,0 +1,5 @@
+{
+  "package_name": "fsspec",
+  "package_version": "2023.8.1",
+  "is_compatible": false
+}
diff --git a/cypress/tests/ui/flowchart/menu.cy.js b/cypress/tests/ui/flowchart/menu.cy.js
@@ -3,10 +3,12 @@
 import { prettifyName } from '../../../../src/utils';
 
 describe('Flowchart Menu', () => {
-  it('verifies that users can select a section of the flowchart, through the drop down. #TC-16', () => {
+  it('verifies that users can select a section of the flowchart through the drop down. #TC-16', () => {
     // Alias
     cy.intercept('GET', '/api/pipelines/*').as('pipelineRequest');
-    cy.get(':nth-child(2) > .menu-option__content > span').as('menuOption');
+    cy.get('.pipeline-list :nth-child(2) > .menu-option__content > span').as(
+      'menuOption'
+    );
 
     let menuOptionValue;
 
@@ -17,7 +19,7 @@ describe('Flowchart Menu', () => {
       });
 
     // Action
-    cy.get('[data-test="kedro-pipeline-selector"]').click();
+    cy.get('.pipeline-list [data-test="kedro-pipeline-selector"]').click();
     cy.get('@menuOption').click({ force: true });
 
     // Assert after action

diff --git a/cypress/tests/ui/flowchart/shareable-urls.cy.js b/cypress/tests/ui/flowchart/shareable-urls.cy.js
@@ -0,0 +1,217 @@
+describe('Shareable URLs', () => {
+  it('verifies that users can open the Deploy Kedro-Viz modal. #TC-52', () => {
+    // Intercept the network request to mock with a fixture
+    cy.__interceptRest__(
+      '/api/package-compatibilities',
+      'GET',
+      '/mock/package-compatibilities-compatible.json'
+    );
+
+    // Action
+    cy.reload();
+    cy.get('.pipeline-menu-button--deploy').click({ force: true });
+
+    // Assert after action
+    cy.get('.shareable-url-modal .modal__wrapper').contains(
+      `Publish and Share Kedro-Viz`
+    );
+  });
+
+  it("shows an incompatible message given the user's fsspec package version is outdated. #TC-53", () => {
+    // Intercept the network request to mock with a fixture
+    cy.__interceptRest__(
+      '/api/package-compatibilities',
+      'GET',
+      '/mock/package-compatibilities-incompatible.json'
+    );
+
+    // Action
+    cy.reload();
+    cy.get('.pipeline-menu-button--deploy').click({ force: true });
+
+    // Assert after action
+    cy.get('.shareable-url-modal .modal__wrapper').contains(
+      `Publishing Kedro-Viz is only supported with fsspec>=2023.9.0. You are currently on version 2023.8.1.`
+    );
+  });
+
+  it('verifies that shareable url modal closes on close button click #TC-54', () => {
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains('Cancel')
+      .click();
+
+    // Assert after action
+    cy.get('.modal.shareable-url-modal').should(
+      'not.have.class',
+      'modal--visible'
+    );
+  });
+
+  it('verifies that users can click on region dropdown and see all region options #TC-55', () => {
+    const regionCount = 30;
+
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+
+    // Assert after action
+    cy.get('.shareable-url-modal .menu-option').should(
+      'have.length',
+      regionCount
+    );
+  });
+
+  it('verifies that publish button should be disabled when region is not selected and bucket name is empty #TC-56', () => {
+    const selectedRegion = 'Select a region';
+    const primaryButtonNodeText = 'Publish';
+
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+
+    // Assert after action
+    cy.get(
+      '.shareable-url-modal [data-test=kedro-pipeline-selector] .dropdown__label span'
+    ).contains(selectedRegion);
+    cy.get('.shareable-url-modal textarea').should('have.value', '');
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .should('be.disabled');
+  });
+
+  it('verifies that publish button should be disabled when a bucket region is selected and bucket name is empty #TC-57', () => {
+    const primaryButtonNodeText = 'Publish';
+
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+    cy.get('.shareable-url-modal .dropdown__options section div')
+      .first()
+      .click();
+
+    // Assert after action
+    cy.get('.shareable-url-modal textarea').should('have.value', '');
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .should('be.disabled');
+  });
+
+  it('verifies that publish button should be enabled when region is selected and bucket name is not empty #TC-58', () => {
+    const bucketName = 'myBucketName';
+    const primaryButtonNodeText = 'Publish';
+
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+    cy.get('.shareable-url-modal .dropdown__options section div')
+      .first()
+      .click();
+    cy.get('.shareable-url-modal textarea').type(bucketName);
+
+    // Assert after action
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .should('be.enabled');
+  });
+
+  it('verifies that error message appears with wrong inputs on publish button click #TC-59', () => {
+    const bucketName = 'myBucketName';
+    const primaryButtonNodeText = 'Publish';
+    const errorButtonNodeText = 'Go back';
+
+    // Action
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+    cy.get('.shareable-url-modal .dropdown__options section div')
+      .first()
+      .click();
+    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .click();
+
+    // Assert after action
+    cy.get('.shareable-url-modal .modal__wrapper').contains(
+      'Something went wrong. Please try again later.'
+    );
+    cy.get('.shareable-url-modal__error button').contains(errorButtonNodeText);
+  });
+
+  it('verifies that AWS link is generated with correct inputs on publish button click #TC-60', () => {
+    const bucketName = 'myBucketName';
+    const primaryButtonNodeText = 'Publish';
+
+    // Intercept the network request to mock with a fixture
+    cy.__interceptRest__(
+      '/api/deploy',
+      'POST',
+      '/mock/deploySuccessResponse.json'
+    ).as('publishRequest');
+
+    // Action
+    cy.reload();
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+    cy.get('.shareable-url-modal .dropdown__options section div')
+      .first()
+      .click();
+    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .click();
+
+    // Wait for the POST request to complete and check the mocked response
+    cy.wait('@publishRequest').then((interception) => {
+      // Assert after action
+      cy.get('.shareable-url-modal__result-url').contains(
+        interception.response.body.url
+      );
+    });
+  });
+
+  it('verifies that AWS link is generated with correct inputs on Republish button click #TC-61', () => {
+    const bucketName = 'myBucketName';
+    const primaryButtonNodeText = 'Publish';
+    const primaryButtonNodeTextVariant = 'Republish';
+    const secondaryButtonNodeText = 'Link Settings';
+
+    // Intercept the network request to mock with a fixture
+    cy.__interceptRest__(
+      '/api/deploy',
+      'POST',
+      '/mock/deploySuccessResponse.json'
+    ).as('publishRequest');
+
+    // Action
+    cy.reload();
+    cy.get('.pipeline-menu-button--deploy').click();
+    cy.get('.shareable-url-modal [data-test=kedro-pipeline-selector]').click();
+    cy.get('.shareable-url-modal .dropdown__options section div')
+      .first()
+      .click();
+    cy.get('.shareable-url-modal textarea').type(bucketName);
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeText)
+      .click();
+
+    // Wait for the POST request to complete
+    cy.wait('@publishRequest');
+
+    // Action
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(secondaryButtonNodeText)
+      .click();
+    cy.get('.shareable-url-modal__button-wrapper button')
+      .contains(primaryButtonNodeTextVariant)
+      .click();
+
+    // Wait for the POST request to complete and check the mocked response
+    cy.wait('@publishRequest').then((interception) => {
+      // Assert after action
+      cy.get('.shareable-url-modal__result-url').contains(
+        interception.response.body.url
+      );
+    });
+  });
+});
diff --git a/package/features/environment.py b/package/features/environment.py
@@ -46,8 +46,8 @@ def before_scenario(context, scenario):
 
     if (
         kedro_version
-        and kedro_version <= Version.parse("0.18.0")
-        and sys.version_info >= (3, 9)
+        and kedro_version <= Version.parse("0.18.12")
+        and sys.version_info >= (3, 11)
     ):
         print(
             (

diff --git a/package/features/viz.feature b/package/features/viz.feature
@@ -3,7 +3,7 @@ Feature: Viz plugin in new project
         Given I have prepared a config file with example code
 
     Scenario: Execute viz with the earliest Kedro version that it supports 
-        Given I have installed kedro version "0.17.5"
+        Given I have installed kedro version "0.18.2"
         And I have run a non-interactive kedro new with pandas-iris starter
         And I have installed the project's requirements
         When I execute the kedro viz command

diff --git a/package/kedro_viz/api/rest/router.py b/package/kedro_viz/api/rest/router.py
@@ -63,9 +63,7 @@ async def deploy_kedro_viz(input_values: S3DeployerConfiguration):
         )
     except Exception as exc:  # pragma: no cover
         logger.exception("Deploying Kedro Viz failed: %s ", exc)
-        return JSONResponse(
-            status_code=500, content={"message": "Failed to deploy Kedro Viz"}
-        )
+        return JSONResponse(status_code=500, content={"message": f"{exc}"})
 
 
 @router.get(

diff --git a/package/kedro_viz/integrations/deployment/s3_deployer.py b/package/kedro_viz/integrations/deployment/s3_deployer.py
@@ -3,14 +3,17 @@
 
 import json
 import logging
+import tempfile
 from datetime import datetime
 from pathlib import Path
 
 import fsspec
+from jinja2 import Environment, FileSystemLoader
 from semver import VersionInfo
 
 from kedro_viz import __version__
 from kedro_viz.api.rest.responses import save_api_responses_to_fs
+from kedro_viz.integrations.kedro import telemetry as kedro_telemetry
 
 _HTML_DIR = Path(__file__).parent.parent.parent.absolute() / "html"
 _METADATA_PATH = "api/deploy-viz-metadata"
@@ -48,11 +51,41 @@ def _upload_api_responses(self):
         """Upload API responses to S3."""
         save_api_responses_to_fs(self._bucket_path)
 
+    def _ingest_heap_analytics(self):
+        """Ingest heap analytics to index file in the build folder."""
+        project_path = Path.cwd().absolute()
+        heap_app_id = kedro_telemetry.get_heap_app_id(project_path)
+        heap_user_identity = kedro_telemetry.get_heap_identity()
+        should_add_telemetry = bool(heap_app_id) and bool(heap_user_identity)
+        html_content = (_HTML_DIR / "index.html").read_text(encoding="utf-8")
+        injected_head_content = []
+
+        env = Environment(loader=FileSystemLoader(_HTML_DIR))
+
+        if should_add_telemetry:
+            logger.debug("Ingesting heap analytics.")
+            telemetry_content = env.get_template("telemetry.html").render(
+                heap_app_id=heap_app_id, heap_user_identity=heap_user_identity
+            )
+            injected_head_content.append(telemetry_content)
+
+        injected_head_content.append("</head>")
+        html_content = html_content.replace("</head>", "\n".join(injected_head_content))
+
+        with tempfile.TemporaryDirectory() as temp_dir:
+            temp_file_path = f"{temp_dir}/index.html"
+
+            with open(temp_file_path, "w", encoding="utf-8") as temp_index_file:
+                temp_index_file.write(html_content)
+
+            self._remote_fs.put(temp_file_path, f"{self._bucket_path}/")
+
     def _upload_static_files(self, html_dir: Path):
         """Upload static HTML files to S3."""
         logger.debug("Uploading static html files to %s.", self._bucket_path)
         try:
             self._remote_fs.put(f"{str(html_dir)}/*", self._bucket_path, recursive=True)
+            self._ingest_heap_analytics()
         except Exception as exc:  # pragma: no cover
             logger.exception("Upload failed: %s ", exc)
             raise exc

diff --git a/package/requirements.txt b/package/requirements.txt
@@ -3,7 +3,7 @@ packaging~=23.0
 kedro>=0.17.5
 ipython>=7.0.0, <9.0
 fastapi>=0.73.0, <0.96.0
-fsspec>=2021.4, <2024.1 
+fsspec[s3]>=2021.4, <2024.1 
 aiofiles==22.1.0
 uvicorn[standard]~=0.22.0
 watchgod~=0.8.2