Add solution builder script (#9)

* Add solution builder script * Update ib_cicd/ib_helpers.py Co-authored-by: Ben Hope <[email protected]> * Update README.md Co-authored-by: Ben Hope <[email protected]> * Update README.md Co-authored-by: Ben Hope <[email protected]> * Update ib_cicd/migration_helpers.py Co-authored-by: Ben Hope <[email protected]> * Add SB script * Add pagination for list dir function * Add unit tests for promote sb solution * Linting * Update templates with new build process * Update CI/CD templates to use build * Update for unit tests * Updates from PR comments --------- Co-authored-by: Ben Hope <[email protected]>
instabase · Jul 12, 2024 · 17a49d7 · 17a49d7
1 parent cfda9c1
commit 17a49d7
Show file tree

Hide file tree

Showing 12 changed files with 611 additions and 60 deletions.
diff --git a/README.md b/README.md
@@ -4,7 +4,9 @@ CI/CD solution to migrate a local solution to a target instabase environment.
 
 The solution uses Instabase APIs and CI tooling to promote solutions between environments. Template CI pipelines for GitHub Actions and Azure DevOps are provided and should be updated to suit customer's needs.
 
-### Python Script
+### Python Scripts
+
+#### Filesystem projects: promote_solution.py
 
 The Python `promote_solution.py` script calls Instabase APIs and can execute the following steps when the corresponding flags are passed in:
 
@@ -35,6 +37,28 @@ The Python `promote_solution.py` script calls Instabase APIs and can execute the
 - `--marketplace`
   - Publishes solution to Marketplace. If not used then Deployed Solutions will be used
 
+#### Solution builder projects: promote_sb_solution.py
+
+##### **NOTE: Solution builder projects supported from 23.07**
+
+- `--compile_source_solution`
+  - Compiles solution in source environment in path specified in `SOURCE_SOLUTION_DIR` environment variable.
+- `--deploy_source_solution`
+  - Publishes the latest version of the solution in the source environment to Deployed Solutions
+- `--promote_solution_to_target`
+  - Uploads to the solution to the target environment
+- `--deploy_target_solution`
+  - Publishes the latest version of the solution in the target environment to Deployed Solutions
+- `--upload_dependencies`
+  - Uploads and publishes the solution dependencies to the target environment based on dependencies listed in `package.json`
+- `--download_ibsolution`
+  - Downloads the `.ibsolution` file to the local filesystem
+- `--set_github_actions_env_var`
+  - Sets `PACKAGE_VERSION` environment variable to be used in later steps in the GitHub Actions pipleline
+- `--set_azure_devops_env_var`
+  - Sets `PACKAGE_VERSION` environment to be used in later steps in the Azure DevOps pipeline
+- `--remote_flow`
+  - Performs series of above steps for a remote workflow, `--compile_source_solution`, `--set_azure_devops_env_var`, `--download_ibsolution`, and `--set_github_actions_env_var` are not included in these so need to be used too
 
 ### GitHub Actions Workflows
 
@@ -50,6 +74,8 @@ If you're not using CI tooling and running the script locally you will need to s
 
 ### Configure Repo
 
+#### Filesystem project configuration
+
 1. Set the following git secrets in Settings > Secrets and Variables > Actions:
     - **SOURCE_IB_API_TOKEN** API token for source environment. Ideally both tokens should belong to a service account
     - **TARGET_IB_API_TOKEN** API token for target environment
@@ -62,9 +88,25 @@ If you're not using CI tooling and running the script locally you will need to s
     - **SOURCE_COMPILED_SOLUTIONS_PATH** Path to where `.ibsolution` files are generated. This will usually be one level higher than the `SOURCE_SOLUTION_DIR`
     - **TARGET_IB_HOST** e.g. "https://instabase.com"
     - **TARGET_IB_PATH** Path in the target IB environment where the solution will get uploaded
-3. Copy `cicd` folder into repository and move one of the `.yml` workflow files into a `.GitHub/workflows` folder
+3. Copy `ib_cicd` folder into repository and move one of the `.yml` template workflow files into a `.GitHub/workflows` folder
+
+#### Solution builder project configuration
+
+1. Set the following git secrets in Settings > Secrets and Variables > Actions:
+    - **SOURCE_IB_API_TOKEN** API token for source environment. Ideally both tokens should belong to a service account
+    - **TARGET_IB_API_TOKEN** API token for target environment
+2. Set git variables in the same place:
+    - **SOURCE_IB_HOST** e.g. "https://solution-eng.aws.sandbox.instabase.com"
+    - **SOURCE_WORKING_DIR** Path in source IB environment's filesystem to use for downloading dependencies and storing ibsolution files
+    - **TARGET_IB_HOST** e.g. "https://instabase.com"
+    - **TARGET_IB_PATH** Path in the target IB environment where the solution will get uploaded
+    - **SOLUTION_BUILDER_NAME** Name of solution builder project
+    - **FLOW_NAME** Name of flow to promote
+    - **WORKSPACE_DRIVE_PATH** Path in source IB environment's filesystem to the drive where the solution builder project is stored, e.g. hannahroiter/ci-cd/fs/Instabase Drive
+    - **DEPENDENCIES** String representation of list of model and marketplace package dependencies in the format `model_name==0.0.1,package_name==0.4.5,second_package==0.0.1`
+3. Copy `ib_cicd` folder into repository and move one of the `.yml` template workflow files into a `.GitHub/workflows` folder
 
-### Local Workflow
+### Local Workflow - filesystem projects only
 
 1. Develop in IB environment.
 2. Once happy with changes checkout a new git branch and copy the solution directory into codebase.
@@ -84,6 +126,8 @@ If you're not using CI tooling and running the script locally you will need to s
 
 ### Remote Workflow
 
+#### Filesystem projects
+
 1. Develop in IB environment.
 2. If in a development environment, bump up the version in `package.json` and ensure that `package.json` and `icon.png` files are in the top level of the directory
 3. If using Deployed solutions, ensure the `package.json` file includes fields `owner` set to `IB_DEPLOYED` and `visibility` set to `PUBLIC`
@@ -94,7 +138,17 @@ If you're not using CI tooling and running the script locally you will need to s
    4. Upload the `.ibsolution` file to the release artifacts
 5. This will compile the solution in the source environment, promote it to the target environment, and store the `.ibsolution` file in the GitHub release
 
-### Sample package.json
+#### Solution builder projectts
+
+1. Develop in IB environment.
+2. Manually run the Github actions workflow which will:
+   1. Run the `promote_sb_solution.py` python script with `--remote_flow` parameters, include `--compile_source_solution` parameter if moving a solution builder project from DEV -> UAT.
+   2. Commit and push the `.ibsolution` file to the repository
+   3. Create a GitHub release with tag equal to the version of the promoted solution
+   4. Upload the `.ibsolution` file to the release artifacts
+3. This will compile the solution in the source environment, promote it to the target environment, and store the `.ibsolution` file in the GitHub release
+
+### Sample package.json (filesystem projects only)
 
 ```{
   "name": "Form W-2",

diff --git a/ib_cicd/ib_helpers.py b/ib_cicd/ib_helpers.py
@@ -5,6 +5,7 @@
 from urllib.parse import quote
 import time
 import logging
+import pathlib
 
 
 def __get_file_api_root(ib_host, api_version="v2", add_files_suffix=True):
@@ -199,7 +200,14 @@ def unzip_files(ib_host, api_token, zip_path, destination_path=None):
     return resp
 
 
-def compile_solution(ib_host, api_token, solution_path, relative_flow_path):
+def compile_solution(
+    ib_host,
+    api_token,
+    solution_path,
+    relative_flow_path=None,
+    solution_builder=False,
+    solution_version=None,
+):
     """
     Compiles a flow
 
@@ -208,24 +216,39 @@ def compile_solution(ib_host, api_token, solution_path, relative_flow_path):
     :param solution_path: (string) path to root folder of solution
                               (e.g. ganan.prabaharan/testing/fs/Instabase Drive/testing_solution)
     :param relative_flow_path: relative path of flow from solution_path (e.g. testing_flow.ibflow)
-                               full flow path is {solutionPath}/{relative_flow_path}
+                               full flow path is {solutionPath}/{relative_flow_path} (used for filesystem projects only)
+    :param solution_builder: (bool) if the solution to be compiled is a solution builder project
+    :param solution_version: (string) version of compiled solution (used for solution builder projects only)
     :return: Response object
     """
     # TODO: API docs issue
     path_encoded = quote(solution_path)
 
     url = os.path.join(*[ib_host, "api/v1", "flow_binary", "compile", path_encoded])
-    bin_path = relative_flow_path.replace(".ibflow", ".ibflowbin")
+
+    if solution_builder:
+        p = pathlib.Path(solution_path)
+        bin_path = os.path.join(
+            *p.parts[:-1], "builds", f"{solution_version}.ibflowbin"
+        )
+        flow_project_root = os.path.join(*p.parts[:7])
+        flow_path = os.path.join(*p.parts[7:])
+    else:
+        bin_path = relative_flow_path.replace(".ibflow", ".ibflowbin")
+        bin_path = os.path.join(solution_path, bin_path)
+        flow_project_root = os.path.join(
+            solution_path, *relative_flow_path.split("/")[:-1]
+        )
+        flow_path = relative_flow_path.split("/")[-1]
+
     headers = {"Authorization": "Bearer {0}".format(api_token)}
     data = json.dumps(
         {
             "binary_type": "Single Flow",
-            "flow_project_root": os.path.join(
-                solution_path, *relative_flow_path.split("/")[:-1]
-            ),
-            "predefined_binary_path": os.path.join(solution_path, bin_path),
+            "flow_project_root": flow_project_root,
+            "predefined_binary_path": bin_path,
             "settings": {
-                "flow_file": relative_flow_path.split("/")[-1],
+                "flow_file": flow_path,
                 "is_flow_v3": True,
             },
         }
@@ -382,6 +405,43 @@ def check_job_status(ib_host, job_id, job_type, api_token):
     return resp
 
 
+def list_directory(ib_host, folder, api_token):
+    """
+    Lists a directory on the IB filesystem and returns full paths
+
+    :param ib_host: (string) IB host url (e.g. https://www.instabase.com)
+    :param folder: (string) path to folder to list
+    :param api_token: (string) api token for IB environment
+    :return: (list) List of paths in directory
+    """
+    file_api_root = __get_file_api_root(ib_host)
+    url = os.path.join(file_api_root, folder)
+
+    headers = {"Authorization": "Bearer {0}".format(api_token)}
+
+    paths = []
+    has_more = None
+    start_token = None
+
+    while has_more is not False:
+        params = {"expect-node-type": "folder", "start-token": start_token}
+        resp = requests.get(url, headers=headers, params=params)
+
+        # Verify request is successful
+        content = json.loads(resp.content)
+        if resp.status_code != 200 or (
+            "status" in content and content["status"] == "ERROR"
+        ):
+            raise Exception(f"Error checking job status: {resp.content}")
+
+        nodes = content["nodes"]
+        paths += [node["full_path"] for node in nodes]
+
+        has_more = content["has_more"]
+        start_token = content["next_page_token"]
+    return paths
+
+
 def wait_until_job_finishes(ib_host, job_id, job_type, api_token):
     """
     Helper function to continuously wait until a job finishes (uses job status api to determine this)
@@ -460,6 +520,6 @@ def deploy_solution(ib_host, api_token, ibsolution_path):
         job_id = json.loads(resp.content)["job_id"]
         logging.info(f"Solution deployed with job ID {job_id}")
     except:
-        logging.info(f"Solution publish status exception: {resp.content}")
+        logging.warning(f"Solution publish status exception: {resp.content}")
 
     return resp
diff --git a/ib_cicd/migration_helpers.py b/ib_cicd/migration_helpers.py
@@ -18,6 +18,7 @@
     get_file_metadata,
     create_folder_if_it_does_not_exists,
     wait_until_job_finishes,
+    publish_to_marketplace,
 )
 
 
@@ -391,3 +392,16 @@ def download_dependencies_from_dev_and_upload_to_prod(
         upload_paths.append(uploaded_path)
 
     return upload_paths
+
+
+def publish_dependencies(uploaded_ibsolutions, ib_host, api_token):
+    """
+    Publish list of dependencies to marketplace
+    :param uploaded_ibsolutions: (list) list of paths to ibsolution dependency paths on the IB filesystem
+    :param ib_host: (string) IB host url (e.g. https://www.instabase.com)
+    :param api_token: (string) api token for IB environment
+    :return: None
+    """
+    for ib_solution_path in uploaded_ibsolutions:
+        publish_resp = publish_to_marketplace(ib_host, api_token, ib_solution_path)
+        logging.info("Publish response for %s: %s", ib_solution_path, publish_resp)