Generative fill demo for re'Invent2023 (#4477)

* add lab7-hosting-controlnet-models-on-sagemaker * remove stable-diffusion-webui-dreambooth-training.ipynb * update for controlnet extension upgrade * update Dockerfile.training and Dockerfile.inference and cleanup * update train.py and serve * update README.md * fix issues with subprocess and update Dockerfile for training and inference * cleanup * cleanup * cleanup * cleanup * remove training part * remove train.sh * cleanup * switch to use djl-inference:0.21.0-fastertransformer5.3.0-cu117 * revise the Dockerfile.inference to login the public ECR * update Dockerfile * update ipynb for shell script * revise shell script in ipynb * revise for Docker build * update ipynb * revise ipynb * update ipynb * update ipynb * cleanup * cleanup * remove --no-half --precision full * add docker comments * add docker comment * switch to docker testing * switch to docker * switch to docker * switch to docker * switch to docker * switch to docker * switch to docker * add CI build test information * change seed/subseed for Lora character * add disclaimers upon to legal requirement * add missing file - build_and_push.sh * change kernel * test * test * test * test * test * upgrade webui to v1.5.1 and pytorch to 2.0.1 * cleanup * minor cleanup * minor cleanup * update to v1.6.0 * cleanup * update for generative fill * update for generative fill * cleanup * cleanup && add badges * update the notebook to use different text prompt * cleanup * revise for docker build on sagemaker notebook and sagemaker studio * reformat for notebook * update README * cleanup * cleanup * add libtcmalloc-minimal4 in Dockerfile * clear all outputs * cleanup * update README --------- Co-authored-by: atqy <[email protected]> Co-authored-by: atqy <[email protected]>
aws · Nov 22, 2023 · 4abe8c5 · 4abe8c5
1 parent 720198b
commit 4abe8c5
Show file tree

Hide file tree

Showing 7 changed files with 2,182 additions and 922 deletions.
diff --git a/...nerativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/Dockerfile.inference b/...nerativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/Dockerfile.inference
@@ -1,23 +1,28 @@
 ARG REGION
 FROM 763104351884.dkr.ecr.$REGION.amazonaws.com/pytorch-inference:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker
 
-RUN apt update && apt install wget git libgl1-mesa-glx libcairo2-dev -y
+RUN apt update && apt install wget git libgl1-mesa-glx libcairo2-dev libtcmalloc-minimal4 -y
 RUN rm -rf /opt/conda/lib/python3.10/site-packages/opencv_python-4.6.0.dist-info/
 RUN pip install opencv-python
-RUN pip install install -U -I --no-deps xformers==0.0.20
+RUN pip install xformers==0.0.22.post7 --extra-index-url https://download.pytorch.org/whl/cu118
 RUN pip install boto3 huggingface_hub triton deepspeed accelerate
 
 RUN mkdir -p /opt/ml/code
 WORKDIR /opt/ml/code
 
-ADD https://api.github.com/repos/xieyongliang/stable-diffusion-webui/git/refs/heads/api-v1.5.1 /opt/ml/webui.version.json
-RUN git clone https://github.com/xieyongliang/stable-diffusion-webui.git /opt/ml/code -b api-v1.5.1
+ADD https://api.github.com/repos/xieyongliang/stable-diffusion-webui/git/refs/heads/api /opt/ml/webui.version.json
+RUN git clone https://github.com/xieyongliang/stable-diffusion-webui.git /opt/ml/code -b api
 
 ADD https://api.github.com/repos/Mikubill/sd-webui-controlnet/git/refs/heads/main /opt/ml/controlnet.version.json
 RUN git clone https://github.com/Mikubill/sd-webui-controlnet.git /opt/ml/code/extensions/sd-webui-controlnet
 
+ADD https://api.github.com/repos/xieyongliang/inpaint-anything/git/refs/heads/main /opt/ml/inpaint-anything.version.json
+RUN git clone https://github.com/xieyongliang/inpaint-anything.git /opt/ml/code/extensions/inpaint-anything
+
 RUN python launch.py -f --exit --skip-torch-cuda-test
-RUN pip install -U pytorch_lightning==1.6.5 pydantic==1.10.11
+RUN pip install -U pytorch_lightning==1.6.5 pydantic==1.10.11 httpx==0.24.1
+RUN pip install -U torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118
+
 RUN rm requirements.txt
 
 COPY serve /opt/ml/code

diff --git a/...enerativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/README.md b/...enerativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/README.md
@@ -1,7 +1,89 @@
 # stable-diffusion-webui-api
 
-This repo shows how to use stable-diffusion-webui to build Amazon SageMaker Docker image for inference and training job and how to generate images based on the combination with Stable-diffusion models and ControlNet / Lora models.
+## Overview
 
-You will find 2 Jupyter Notebooks: 1 for sync ControlNet / Lora inference, and 1 for async ControlNet / Lora inference.
+In this notebook, we will explore how to build generative fill application and host Stable Diffusion/ ControlNet / segment anything models on SageMaker asynchronous endpoint using DLC container.
 
-Note: Amazon Web Services has no control or authority over the third-party generative AI service referenced in this Workshop, and does not make any representations or warranties that the third-party generative AI service is secure, virus-free, operational, or compatible with your production environment and standards. You are responsible for making your own independent assessment of the content provided in this Workshop, and take measures to ensure that you comply with your own specific quality control practices and standards, and the local rules, laws, regulations, licenses and terms of use that apply to you, your content, and the third-party generative AI service referenced in this Workshop. The content of this Workshop: (a) is for informational purposes only, (b) represents current Amazon Web Services product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from Beijing Sinnet Technology Co., Ltd. (“Sinnet”), Ningxia Western Cloud Data Technology Co., Ltd. (“NWCD”), Amazon Connect Technology Services (Beijing) Co., Ltd. (“Amazon”), or their respective affiliates, suppliers or licensors.  Amazon Web Services’ content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied.  The responsibilities and liabilities of Sinnet, NWCD or Amazon to their respective customers are controlled by the applicable customer agreements. 
+You will find 2 Jupyter Notebooks: 1 for running with Amazon SageMaker Studio and 1 for running with Amazon SageMaker Notebook.
+
+## IAM role recommendations
+
+1) Running with Amazon SageMaker Studio
+
+* Permissions policies
+
+    ```
+    AmazonSageMakerFullAccess
+    AmazonEC2ContainerRegistryFullAccess
+    AWSCodeBuildAdminAccess
+    IAMFullAccess
+    ```
+
+* Trusted entities
+
+    ```
+    {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Principal": {
+                    "Service": [
+                        "sagemaker.amazonaws.com",
+                        "codebuild.amazonaws.com"
+                    ]
+                },
+                "Action": "sts:AssumeRole"
+            }
+        ]
+    }
+    ```
+
+* Tested image, kernel, and instance:
+    ```
+    image: Pytorch 2.0.1 Python 3.10 CPU Optimized
+    kernel: Python 3
+    instance: ml.m5.4xlarge
+    
+    ```
+
+2) Running with Amazon SageMaker Notebook
+
+* Permission Policies
+
+    ```
+    AmazonSageMakerFullAccess
+    AmazonEC2ContainerRegistryFullAccess
+    ```
+
+* Trusted entities
+
+    ```
+    {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Principal": {
+                    "Service": [
+                        "ecs.amazonaws.com",
+                        "sagemaker.amazonaws.com"
+                    ]
+                },
+                "Action": "sts:AssumeRole"
+            }
+        ]
+    }
+    ```
+
+* Tested kernel:
+    ```
+    kernel: conda_pytorch_p310
+    
+    ```
+
+## Note
+
+1. You may need to adjust IAM roles definition to achieve fine grained access control.
+
+2. Amazon Web Services has no control or authority over the third-party generative AI service referenced in this Workshop, and does not make any representations or warranties that the third-party generative AI service is secure, virus-free, operational, or compatible with your production environment and standards. You are responsible for making your own independent assessment of the content provided in this Workshop, and take measures to ensure that you comply with your own specific quality control practices and standards, and the local rules, laws, regulations, licenses and terms of use that apply to you, your content, and the third-party generative AI service referenced in this Workshop. The content of this Workshop: (a) is for informational purposes only, (b) represents current Amazon Web Services product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from Beijing Sinnet Technology Co., Ltd. (“Sinnet”), Ningxia Western Cloud Data Technology Co., Ltd. (“NWCD”), Amazon Connect Technology Services (Beijing) Co., Ltd. (“Amazon”), or their respective affiliates, suppliers or licensors.  Amazon Web Services’ content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied.  The responsibilities and liabilities of Sinnet, NWCD or Amazon to their respective customers are controlled by the applicable customer agreements. 
diff --git a/inference/generativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/README.pdf b/inference/generativeai/llm-workshop/lab12-hosting-controlnet-models-on-sagemaker/README.pdf
diff --git a/...sting-controlnet-models-on-sagemaker/images/inference/ControlNet/bal-source.png b/...sting-controlnet-models-on-sagemaker/images/inference/ControlNet/bal-source.png