Skip to content

Commit

Permalink
v24.2.0
Browse files Browse the repository at this point in the history
  • Loading branch information
@ithomas-veeva
ithomas-veeva committed Aug 12, 2024
1 parent f8b372e commit 2669cfa
Show file tree
Hide file tree
Showing 21 changed files with 4,429 additions and 1,243 deletions.
875 changes: 422 additions & 453 deletions CloudFormationDirectDataTemplateLatest.yaml
View file

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions Public Direct Data Lambda API.postman_collection.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"header": [],
"body": {
"mode": "raw",
"raw": "{\n \"step\": \"retrieve\", //This command is to invoke the listing and downloading of Direct Data files\n \"start_time\": \"2000-01-01T00:00Z\", //This is start of the time window for the specific direct data file\n \"stop_time\": \"2024-03-11T00:00Z\", //This is stop of the time window for the specific direct data file\n \"extract_type\": \"full\", //This is the type of Direct Data file to be retrieved. The options are full, incremental or log\n \"continue_processing\": false //This determines if you want to perform the full extraction process or stop afte this specific step has been completed.\n}",
"raw": "{\n \"step\": \"retrieve\", //This command is to invoke the listing and downloading of Direct Data files\n \"start_time\": \"2000-01-01T00:00Z\", //This is start of the time window for the specific direct data file\n \"stop_time\": \"2024-03-11T00:00Z\", //This is stop of the time window for the specific direct data file\n \"extract_type\": \"full\", //This is the type of Direct Data file to be retrieved. The options are full, incremental or log\n \"continue_processing\": false, //This determines if you want to perform the full extraction process or stop afte this specific step has been completed.\n \"secret\": \"demo\" //This specifies which secret block to use in the Secrets Manager file declared in the Lambda environment variable\n}",
"options": {
"raw": {
"language": "json"
Expand All @@ -42,7 +42,7 @@
"header": [],
"body": {
"mode": "raw",
"raw": "{\n \"step\": \"unzip\", //This denotes that the unzip step will be performed\n \"source_file\": \"direct-data/168629-20240307-0845-N.tar.gz\", //This is the full path in S3 of the file that needs to be unzipped\n \"target_directory\": \"direct-data/168629-20240307-0845-N\", //This is the output directory of where the unzipped contents will be placed\n \"extract_type\": \"incremental\", //This is the type of Direct Data file that is being handled. The options are full, log, or incremental\n \"continue_processing\": false //This determines if you want to perform the full extraction process or stop afte this specific step has been completed.\n}",
"raw": "{\n \"step\": \"unzip\", //This denotes that the unzip step will be performed\n \"source_file\": \"direct-data/168629-20240307-0845-N.tar.gz\", //This is the full path in S3 of the file that needs to be unzipped\n \"target_directory\": \"direct-data/168629-20240307-0845-N\", //This is the output directory of where the unzipped contents will be placed\n \"extract_type\": \"incremental\", //This is the type of Direct Data file that is being handled. The options are full, log, or incremental\n \"continue_processing\": false, //This determines if you want to perform the full extraction process or stop afte this specific step has been completed.\n \"secret\": \"demo\" //This specifies which secret block to use in the Secrets Manager file declared in the Lambda environment variable\n}",
"options": {
"raw": {
"language": "json"
Expand All @@ -68,7 +68,7 @@
"header": [],
"body": {
"mode": "raw",
"raw": "{\n \"step\": \"load_data\",\n \"source_file\": \"direct-data/168629-20240307-0845-N\", //This is the directory of where Direct Data that is going to be loaded exists\n \"extract_type\": \"incremental\" //This is the type of Direct Data file that is being handled. The options are full, log, or incremental\n}",
"raw": "{\n \"step\": \"load_data\",\n \"source_file\": \"direct-data/168629-20240307-0845-N\", //This is the directory of where Direct Data that is going to be loaded exists\n \"extract_type\": \"incremental\", //This is the type of Direct Data file that is being handled. The options are full, log, or incremental\n \"secret\": \"demo\" //This specifies which secret block to use in the Secrets Manager file declared in the Lambda environment variable\n}",
"options": {
"raw": {
"language": "json"
Expand Down
33 changes: 20 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ Note: All resources should be created in the same AWS Region.
* Step 1:
* _Trusted entity type_: `AWS account`
* _An AWS account_: `This account`
* _Use case_: `Cloudformation`
* Step 2: Attach the following AWS managed policies. These are required to access Cloudformation and create the direct data resources.
* AmazonAPIGatewayAdministrator
* AmazonEC2ContainerRegistryFullAccess
Expand Down Expand Up @@ -109,7 +110,8 @@ Note: All resources should be created in the same AWS Region.
### S3 Bucket
* Navigate to the S3 service in the AWS Console
* Search for and select the S3 bucket named `cf-direct-data`
* Search for and select the S3 bucket named `{ACCOUNT_ID}-{REGION}-cf-direct-data`
* Copy the s3 bucket name and note it down separately. This will be used in the Direct Data configuration file
* Create a folder at the root of the bucket named `direct-data`
### Redshift Cluster
Expand All @@ -128,13 +130,15 @@ Note: All resources should be created in the same AWS Region.
### Secrets Manager
* Navigate to the Secrets Manager service in the AWS Console
* Search for and select the secret named `direct-data-config.ini`
* Select `Retrieve secret value` then `Edit`. Update the following values:
* [vault] username
* [vault] password
* [vault] dns
* [redshift] host (Use the previously copied redshift endpoint. Do not include the port number/database name)
* [redshift] iam_redshift_s3_read (Use the previously copied ARN for `cf-direct-data-redshift-role-{REGION}`)
* [redshift] password (If updated in the previous step)
* Select `Retrieve secret value` then `Edit`. Update the following values under the [demo] section:
* vault_username
* vault_password
* vault_dns
* redshift_host (Use the previously copied redshift endpoint. Do not include the port number/database name)
* redshift_iam_redshift_s3_read (Use the previously copied ARN for `cf-direct-data-redshift-role-{REGION}`)
* redshift_password (If updated in the previous step)
* s3_bucket_name
* Additional sections can be added with different vault and/or AWS services specified for multiple Vault and database functionality.
### VPC
* Navigate to the VPC service in the AWS Console
Expand All @@ -159,7 +163,8 @@ Note: All resources should be created in the same AWS Region.
"start_time": "2000-01-01T00:00Z",
"stop_time": "2024-04-19T00:00Z", //Update this value to the current date
"extract_type": "full",
"continue_processing": true
"continue_processing": true,
"secret": "demo"
}
```
* Click `Send`
Expand Down Expand Up @@ -194,7 +199,8 @@ These schedules should be created after the initial `full` extract is invoked.
{
"step": "retrieve",
"extract_type": "incremental",
"continue_processing": true
"continue_processing": true,
"secret": "demo"
}
```
* Select `Next`
Expand All @@ -208,9 +214,10 @@ These schedules should be created after the initial `full` extract is invoked.
* _Payload_:
```json
{
"step": "retrieve",
"extract_type": "log",
"continue_processing": true
"step": "retrieve",
"extract_type": "log",
"continue_processing": true,
"secret": "demo"
}
```

Expand Down
Loading

0 comments on commit 2669cfa

Please sign in to comment.