Merge pull request #16 from mohannashahrad/CDKTemplateTimestream

IoT data visualization Timestream CloudFormation template

cloud_templates/.gitignore

@@ -0,0 +1,2 @@
+.DS_Store
+aws_cdk/common/__pycache__/

cloud_templates/README.md

@@ -0,0 +1,32 @@
+# Data ingestion and visualization with AWS CloudFormation templates
+
+These CloudFormation templates are prepared to deploy to the cloud and visualize the data coming from your Expresslink. To learn more about AWS CloudFormation, see [**What is CloudFormation?**](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)
+
+These templates are designed in four different patterns to ingest your IoT data using: 
+
+1.  Amazon Timestream
+<br>
+<center><img src="https://d2908q01vomqb2.cloudfront.net/f6e1126cedebf23e1463aee73f9df08783640400/2022/04/04/Blog-IoT-Reporting-3-Timstream.png" alt="alt text"/></center>
+</br>
+
+2. AWS IoT Analytics 
+<br>
+<center><img src="https://d2908q01vomqb2.cloudfront.net/f6e1126cedebf23e1463aee73f9df08783640400/2022/04/04/Blog-IoT-Reporting-3-IotAnalytics.png" alt="alt text"/></center>
+</br>
+
+3. Amazon Kinesis Data Firehose 
+
+<br>
+<center><img src="https://d2908q01vomqb2.cloudfront.net/f6e1126cedebf23e1463aee73f9df08783640400/2022/04/04/Blog-IoT-Reporting-3-KinesisAthena.png" alt="alt text"/></center>
+</br>
+
+4. Amazon OpenSearch Service
+<br>
+<center><img src="https://d2908q01vomqb2.cloudfront.net/f6e1126cedebf23e1463aee73f9df08783640400/2022/04/04/Blog-IoT-Reporting-3-OpenSearch.png" alt="alt text"/></center>
+</br>
+
+The subdirectories contain the following: 
+
+* Under `demo` directory, there are the demo templates along with a sample program to run on your Expresslink. 
+* Under `aws_cdk` directory, there are the CDK (Cloud Development Kit) stacks used to crate the CloudFormation templates. See [What is the AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html) to learn more about AWS CDK.
+* Under `user_guides` directory you can find the guides to get started with the templates and how to customize them with AWS CDK. 

cloud_templates/aws_cdk/TimestreamPattern/.gitignore

@@ -0,0 +1,9 @@
+*.swp
+__pycache__
+.pytest_cache
+.venv
+*.egg-info
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

cloud_templates/aws_cdk/TimestreamPattern/README.md

@@ -0,0 +1,102 @@
+
+# Welcome to your CDK project! 
+# IoT Data visulaization with Amazon Timestream
+
+The `cdk.json` file tells the CDK Toolkit how to execute your app.
+
+This project is set up like a standard Python project.  The initialization
+process also creates a virtualenv within this project, stored under the `.venv`
+directory.  To create the virtualenv it assumes that there is a `python3`
+(or `python` for Windows) executable in your path with access to the `venv`
+package. If for any reason the automatic creation of the virtualenv fails,
+you can create the virtualenv manually.
+
+To manually create a virtualenv on MacOS and Linux:
+
+```
+$ python3 -m venv .venv
+```
+
+After the init process completes and the virtualenv is created, you can use the following
+step to activate your virtualenv.
+
+```
+$ source .venv/bin/activate
+```
+
+If you are a Windows platform, you would activate the virtualenv like this:
+
+```
+% .venv\Scripts\activate.bat
+```
+
+Once the virtualenv is activated, you can install the required dependencies.
+
+```
+$ pip install -r requirements.txt
+```
+
+At this point you can now synthesize the CloudFormation template for this code.
+
+```
+$ cdk synth
+```
+
+To add additional dependencies, for example other CDK libraries, just add
+them to your `setup.py` file and rerun the `pip install -r requirements.txt`
+command.
+
+## Useful commands
+
+ * `cdk ls`          list all stacks in the app
+ * `cdk synth`       emits the synthesized CloudFormation template
+ * `cdk deploy`      deploy this stack to your default AWS account/region
+ * `cdk diff`        compare deployed stack with current state
+ * `cdk docs`        open CDK documentation
+
+## Context parameters 
+There are multiple context parameters that you need to set before synthesizing or delpoying this CDK stack. You can specify a context variable either as part of an AWS CDK CLI command, or in `cdk.json`.
+To create a command line context variable, use the __--context (-c) option__, as shown in the following example.
+
+```
+$ cdk cdk synth -c bucket_name=mybucket
+```
+
+To specify the same context variable and value in the cdk.json file, use the following sample code.
+
+```
+{
+  "context": {
+    "bucket_name": "mybucket"
+  }
+}
+```
+
+In this project, these are the following parameters to be set: 
+
+* `topic_sql`          
+<br>It is required for IoT Core rule creation to add a simplified SQL syntax to filter messages received on an MQTT topic and push the data elsewhere. 
+<br> __Format__: Enter an SQL statement using the following: ```SELECT <Attribute> FROM <Topic Filter> WHERE <Condition>```. For example: ```SELECT temperature FROM 'iot/topic' WHERE temperature > 50```. To learn more, see AWS IoT SQL Reference.
+
+* `dimensions`
+<br> Each record contains an array of dimensions (minimum 1). Dimensions represent the metadata attributes of a time series data point. Specify the dimension(s) for your data. 
+<br> __Format__: Must be in a format of a list of strings. For example, for the input ```[device_id]``` the following key-value would be attached to the IoT Core rule: 
+<br>```{dimension's name: device_id, dimension_value: ${device_id}}```</br>
+
+* `timestream_db_name`&nbsp;&nbsp;&nbsp;&nbsp;`<Optional>`
+<br> The name of Timestream databse to hold your data. 
+<br> __Format__: Specify a name that is unique for all Timestream databases in your AWS account in the current Region. You can not change this name once you create it. Must be between 3 and 256 characters long. Must contain letters, digits, dashes, periods or underscores.
+
+* `timestream_table_name`&nbsp;&nbsp;&nbsp;&nbsp;`<Optional>`
+<br> The name of Timestream databse to hold your data.  
+<br> __Format__: Specify a table name that is unique within its database. You can not change this name once you create it. Must be between 3 and 256 characters long. Must contain letters, digits, dashes, periods or underscores.
+
+* `timestream_iot_rule_name`&nbsp;&nbsp;&nbsp;&nbsp;`<Optional>`
+<br> The name of the IoT Core rule that is going to be created. 
+<br> __Format__: Should be an alphanumeric string that can also contain underscore (_) characters, but no spaces.
+
+* `timestream_iot_role_name`&nbsp;&nbsp;&nbsp;&nbsp;`<Optional>`
+<br> An IAM role should be created to grant AWS IoT access to your endpoint. This parameter is for setting the name of this role.
+<br> __Format__: Enter a unique role name that contains alphanumeric characters, hyphens, and underscores. A role name can't contain any spaces.
+
+Enjoy!

cloud_templates/aws_cdk/TimestreamPattern/app.py

@@ -0,0 +1,27 @@
+import os
+
+import aws_cdk as cdk
+
+from timestream_pattern.timestream_pattern_stack import TimestreamPatternStack
+
+
+app = cdk.App()
+TimestreamPatternStack(app, "TimestreamPatternStack",
+    # If you don't specify 'env', this stack will be environment-agnostic.
+    # Account/Region-dependent features and context lookups will not work,
+    # but a single synthesized template can be deployed anywhere.
+
+    # Uncomment the next line to specialize this stack for the AWS Account
+    # and Region that are implied by the current CLI configuration.
+
+    #env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')),
+
+    # Uncomment the next line if you know exactly what Account and Region you
+    # want to deploy the stack to. */
+
+    #env=cdk.Environment(account='123456789012', region='us-east-1'),
+
+    # For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html
+    )
+
+app.synth()

cloud_templates/aws_cdk/TimestreamPattern/cdk.json

@@ -0,0 +1,44 @@
+{
+  "app": "python3 app.py",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "requirements*.txt",
+      "source.bat",
+      "**/__init__.py",
+      "python/__pycache__",
+      "tests"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
+    "@aws-cdk/core:stackRelativeExports": true,
+    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
+    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
+    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
+    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
+    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
+    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
+    "@aws-cdk/core:checkSecretUsage": true,
+    "@aws-cdk/aws-iam:minimizePolicies": true,
+    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
+    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
+    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
+    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
+    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
+    "@aws-cdk/core:target-partitions": [
+      "aws",
+      "aws-cn"
+    ],
+    "topic_sql": "SELECT * FROM 'Timestream_demo'",
+    "dimensions": ["Location"],
+    "timestream_db_name": "demo_db",
+    "timestream_table_name": "demo_table",
+    "timestream_iot_role_name": "demo_iot_timestream_role",
+    "timestream_iot_rule_name": "demo_to_timetream_rule"
+  }
+}

cloud_templates/aws_cdk/TimestreamPattern/requirements-dev.txt

@@ -0,0 +1 @@
+pytest==6.2.5

cloud_templates/aws_cdk/TimestreamPattern/requirements.txt

@@ -0,0 +1,2 @@
+aws-cdk-lib==2.37.1
+constructs>=10.0.0,<11.0.0

cloud_templates/aws_cdk/TimestreamPattern/source.bat

@@ -0,0 +1,13 @@
+@echo off
+
+rem The sole purpose of this script is to make the command
+rem
+rem     source .venv/bin/activate
+rem
+rem (which activates a Python virtualenv on Linux or Mac OS X) work on Windows.
+rem On Windows, this command just runs this batch file (the argument is ignored).
+rem
+rem Now we don't need to document a Windows command for activating a virtualenv.
+
+echo Executing .venv\Scripts\activate.bat for you
+.venv\Scripts\activate.bat