From bc8995965c0ae520029ad412e70a3c140cce0ef6 Mon Sep 17 00:00:00 2001 From: Michael Chin Date: Mon, 13 Mar 2023 17:00:57 -0700 Subject: [PATCH] Add Neptune Workbench CFN template (#442) Co-authored-by: Michael Chin --- additional-databases/sagemaker/README.md | 9 +- .../neptune-notebook-cloudformation/README.md | 58 ++++++++ .../neptune-workbench-stack.yaml | 130 ++++++++++++++++++ .../sagemaker-notebook-lifecycle/README.md | 8 ++ .../install-graph-notebook-lc-cn.sh | 0 .../install-graph-notebook-lc.sh | 0 6 files changed, 200 insertions(+), 5 deletions(-) create mode 100644 additional-databases/sagemaker/neptune-notebook-cloudformation/README.md create mode 100644 additional-databases/sagemaker/neptune-notebook-cloudformation/neptune-workbench-stack.yaml create mode 100644 additional-databases/sagemaker/sagemaker-notebook-lifecycle/README.md rename additional-databases/sagemaker/{ => sagemaker-notebook-lifecycle}/install-graph-notebook-lc-cn.sh (100%) rename additional-databases/sagemaker/{ => sagemaker-notebook-lifecycle}/install-graph-notebook-lc.sh (100%) diff --git a/additional-databases/sagemaker/README.md b/additional-databases/sagemaker/README.md index e149bc13..0b9290d3 100644 --- a/additional-databases/sagemaker/README.md +++ b/additional-databases/sagemaker/README.md @@ -1,8 +1,7 @@ -## Launching graph-notebook using Amazon SageMaker -You can easily launch instances of graph-notebook on Amazon SageMaker by using a lifecycle configuration. To learn more about lifecycle configurations and how to create one, see [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html). +## Using graph-notebook on Amazon Sagemaker -Use the sample lifecycle configuration in this folder, [`install-graph-notebook-lc.sh`](install-graph-notebook-lc.sh) ([`install-graph-notebook-lc-cn.sh`](install-graph-notebook-lc-cn.sh) if using `cn-north-1` or `cn-northwest-1` region) or create your own shell script. +Amazon Sagemaker Notebooks provide an easy and effective solution for hosting, configuring, and running `graph-notebook` against a graph database. These notebooks also serve as the base platform for [Neptune Workbench](https://docs.aws.amazon.com/neptune/latest/userguide/graph-notebooks.html). -After you create a lifecycle configuration on SageMaker, you can create new notebook instances by specifying a saved lifecycle configuration: +If you would like to manually deploy a Neptune Workbench instance via AWS CloudFormation, please see the instructions in the [`neptune-notebook-cloudformation`](https://github.com/aws/graph-notebook/blob/main/additional-databases/sagemaker/neptune-notebook-cloudformation) folder. -![create-a-notebook](/././images/Create-Notebook-Instance.png) +For non-Neptune use cases, you can follow the instructions in the [`sagemaker-notebook-lifecycle`](https://github.com/aws/graph-notebook/blob/main/additional-databases/sagemaker/sagemaker-notebook-lifecycle) folder. \ No newline at end of file diff --git a/additional-databases/sagemaker/neptune-notebook-cloudformation/README.md b/additional-databases/sagemaker/neptune-notebook-cloudformation/README.md new file mode 100644 index 00000000..89e7b0a9 --- /dev/null +++ b/additional-databases/sagemaker/neptune-notebook-cloudformation/README.md @@ -0,0 +1,58 @@ +## Launching graph-notebook as Amazon Neptune Workbench via AWS CloudFormation + +The AWS CloudFormation template in this folder, [`neptune-workbench-stack.yaml`](neptune-workbench-stack.yaml), deploys Amazon Neptune workbench notebooks as resources, and includes the base 'Getting Started' notebooks. The workbench lets you work with your Amazon Neptune cluster using Jupyter notebooks hosted by Amazon SageMaker. You are billed for workbench resources through Amazon SageMaker, separately from your Neptune billing. + +### Parameter details +#### Minimum permissions for the SageMakerNotebookRole +This is the ARN for the AWS IAM role that the notebook instance will assume. Make sure that this role has at least the following minimum permissions within its service role policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::aws-neptune-notebook", + "arn:aws:s3:::aws-neptune-notebook/*" + ] + }, + { + "Effect": "Allow", + "Action": "neptune-db:connect", + "Resource": [ + "your-cluster-arn/*" + ] + } + ] +} +``` + +The role should also establish the following trust relationship: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "sagemaker.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] +} +``` + +#### How to populate the 'Cluster' value within the AWS Console for Amazon Neptune Notebooks +Add the following tags manually to the notebook instance. + +| Key | Value | +| ------------- |-------------| +| **aws-neptune-cluster-id** | Amazon Neptune database cluster ID (found under *DB cluster id* under *Configuration* of the selected cluster in the AWS console) | +| **aws-neptune-resource-id** | Amazon Neptune cluster resource ID (found under *Resource id* under *Configuration* of the selected cluster in the AWS console) | diff --git a/additional-databases/sagemaker/neptune-notebook-cloudformation/neptune-workbench-stack.yaml b/additional-databases/sagemaker/neptune-notebook-cloudformation/neptune-workbench-stack.yaml new file mode 100644 index 00000000..f1848446 --- /dev/null +++ b/additional-databases/sagemaker/neptune-notebook-cloudformation/neptune-workbench-stack.yaml @@ -0,0 +1,130 @@ +AWSTemplateFormatVersion: '2010-09-09' + +Description: A template to deploy Neptune Notebooks using CloudFormation resources. + +Parameters: + NotebookInstanceType: + Description: The notebook instance type. + Type: String + Default: ml.t2.medium + AllowedValues: + - ml.t2.medium + - ml.t2.large + - ml.t2.xlarge + - ml.t2.2xlarge + - ml.t3.2xlarge + - ml.t3.large + - ml.t3.medium + - ml.t3.xlarge + - ml.m4.xlarge + - ml.m4.2xlarge + - ml.m4.4xlarge + - ml.m4.10xlarge + - ml.m4.16xlarge + - ml.m5.12xlarge + - ml.m5.24xlarge + - ml.m5.2xlarge + - ml.m5.4xlarge + - ml.m5.xlarge + - ml.p2.16xlarge + - ml.p2.8xlarge + - ml.p2.xlarge + - ml.p3.16xlarge + - ml.p3.2xlarge + - ml.p3.8xlarge + - ml.c4.2xlarge + - ml.c4.4xlarge + - ml.c4.8xlarge + - ml.c4.xlarge + - ml.c5.18xlarge + - ml.c5.2xlarge + - ml.c5.4xlarge + - ml.c5.9xlarge + - ml.c5.xlarge + - ml.c5d.18xlarge + - ml.c5d.2xlarge + - ml.c5d.4xlarge + - ml.c5d.9xlarge + - ml.c5d.xlarge + ConstraintDescription: Must be a valid SageMaker instance type. + + NeptuneClusterEndpoint: + Description: The cluster endpoint of an existing Neptune cluster. + Type: String + + NeptuneClusterPort: + Description: 'OPTIONAL: The Port of an existing Neptune cluster (default 8182).' + Type: String + Default: '8182' + + NeptuneClusterSecurityGroups: + Description: The VPC security group IDs. The security groups must be for the same VPC as specified in the subnet. + Type: List + + NeptuneClusterSubnetId: + Description: The ID of the subnet in a VPC to which you would like to have a connectivity from your ML compute instance. + Type: AWS::EC2::Subnet::Id + + SageMakerNotebookRole: + Description: The ARN for the IAM role that the notebook instance will assume. + Type: String + AllowedPattern: ^arn:aws[a-z\-]*:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@\-_/]+$ + + SageMakerNotebookName: + Description: The name of the Neptune notebook. + Type: String + +Resources: + NeptuneNotebookInstance: + Type: AWS::SageMaker::NotebookInstance + Properties: + NotebookInstanceName: !Join + - '' + - - 'aws-neptune-' + - !Ref SageMakerNotebookName + InstanceType: + Ref: NotebookInstanceType + SubnetId: + Ref: NeptuneClusterSubnetId + SecurityGroupIds: + Ref: NeptuneClusterSecurityGroups + RoleArn: + Ref: SageMakerNotebookRole + LifecycleConfigName: + Fn::GetAtt: + - NeptuneNotebookInstanceLifecycleConfig + - NotebookInstanceLifecycleConfigName + + NeptuneNotebookInstanceLifecycleConfig: + Type: AWS::SageMaker::NotebookInstanceLifecycleConfig + Properties: + OnStart: + - Content: + Fn::Base64: + Fn::Join: + - '' + - - "#!/bin/bash\n" + - sudo -u ec2-user -i << 'EOF' + - "\n" + - echo 'export GRAPH_NOTEBOOK_AUTH_MODE= + - "DEFAULT' >> ~/.bashrc\n" + - echo 'export GRAPH_NOTEBOOK_HOST= + - !Ref NeptuneClusterEndpoint + - "' >> ~/.bashrc\n" + - echo 'export GRAPH_NOTEBOOK_PORT= + - !Ref NeptuneClusterPort + - "' >> ~/.bashrc\n" + - echo 'export NEPTUNE_LOAD_FROM_S3_ROLE_ARN= + - "' >> ~/.bashrc\n" + - echo 'export AWS_REGION= + - !Ref AWS::Region + - "' >> ~/.bashrc\n" + - aws s3 cp s3://aws-neptune-notebook/graph_notebook.tar.gz /tmp/graph_notebook.tar.gz + - "\n" + - rm -rf /tmp/graph_notebook + - "\n" + - tar -zxvf /tmp/graph_notebook.tar.gz -C /tmp + - "\n" + - /tmp/graph_notebook/install.sh + - "\n" + - EOF diff --git a/additional-databases/sagemaker/sagemaker-notebook-lifecycle/README.md b/additional-databases/sagemaker/sagemaker-notebook-lifecycle/README.md new file mode 100644 index 00000000..8eec63ed --- /dev/null +++ b/additional-databases/sagemaker/sagemaker-notebook-lifecycle/README.md @@ -0,0 +1,8 @@ +## Launching graph-notebook on Amazon SageMaker using a lifecycle +You can easily configure graph-notebook to run on an Amazon SageMaker Notebook instance by using a lifecycle configuration. To learn more about lifecycle configurations and how to create one, see [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html). + +Use the sample lifecycle configuration in this folder, [`install-graph-notebook-lc.sh`](install-graph-notebook-lc.sh) ([`install-graph-notebook-lc-cn.sh`](install-graph-notebook-lc-cn.sh) if using `cn-north-1` or `cn-northwest-1` region) or create your own shell script. + +After you create a lifecycle configuration on SageMaker, you can create new notebook instances by specifying a saved lifecycle configuration: + +![create-a-notebook](/images/Create-Notebook-Instance.png) diff --git a/additional-databases/sagemaker/install-graph-notebook-lc-cn.sh b/additional-databases/sagemaker/sagemaker-notebook-lifecycle/install-graph-notebook-lc-cn.sh similarity index 100% rename from additional-databases/sagemaker/install-graph-notebook-lc-cn.sh rename to additional-databases/sagemaker/sagemaker-notebook-lifecycle/install-graph-notebook-lc-cn.sh diff --git a/additional-databases/sagemaker/install-graph-notebook-lc.sh b/additional-databases/sagemaker/sagemaker-notebook-lifecycle/install-graph-notebook-lc.sh similarity index 100% rename from additional-databases/sagemaker/install-graph-notebook-lc.sh rename to additional-databases/sagemaker/sagemaker-notebook-lifecycle/install-graph-notebook-lc.sh