DPAT: Building out DataHub components #4162

julialawrence · 2023-11-28T11:33:05Z

ministryofjustice/analytical-platform#2454

Implements the following components to support Datahub Deployment:

Ingestion iam role/policy
RDS database (using datasources for resources provisioned in MP repo)]
RDS security group
Encryption key

Note:

Resources deployed in the MP repo are retrieved via data source TF calls.

Caveats:

Helm deployments still done locally.

Still Outstanding:

OpenSearch
AWS Kafka-equivalent

github-actions · 2023-11-28T11:35:15Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:
terraform/environments/hmpps-domain-services terraform/environments/hmpps-domain-services/templates

*****************************

Running TFSEC in terraform/environments/hmpps-domain-services
Excluding the following checks: AWS095

======================================================
tfsec is joining the Trivy family

tfsec will continue to remain available 
for the time being, although our engineering 
attention will be directed at Trivy going forward.

You can read more here: 
https://github.com/aquasecurity/tfsec/discussions/1994
======================================================
  timings
  ──────────────────────────────────────────
  disk i/o             1.506874ms
  parsing              141.807603ms
  adaptation           105.708µs
  checks               3.613901ms
  total                147.034086ms

  counts
  ──────────────────────────────────────────
  modules downloaded   0
  modules processed    5
  blocks processed     268
  files read           71

  results
  ──────────────────────────────────────────
  passed               1
  ignored              0
  critical             0
  high                 0
  medium               0
  low                  0


No problems detected!

tfsec_exitcode=0

*****************************

Running TFSEC in terraform/environments/hmpps-domain-services/templates
Skipping folder as path name contains *templates*

`Checkov Scan` Failed

Show Output

*****************************

Checkov will check the following folders:
terraform/environments/hmpps-domain-services terraform/environments/hmpps-domain-services/templates

*****************************

Running Checkov in terraform/environments/hmpps-domain-services
terraform scan results:

Passed checks: 92, Failed checks: 1, Skipped checks: 19

Check: CKV_AWS_103: "Ensure that load balancer is using at least TLS 1.2"
	FAILED for resource: module.baseline.module.lb_listener.aws_lb_listener.this
	File: /../../modules/lb_listener/main.tf:1-61
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/aws-general-policies/bc-aws-general-43

		Code lines for this resource are too many. Please use IDE of your choice to review the file.

checkov_exitcode=1

*****************************

Running Checkov in terraform/environments/hmpps-domain-services/templates
Skipping folder as path name contains *templates*

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:
terraform/environments/hmpps-domain-services terraform/environments/hmpps-domain-services/templates

*****************************

Running tflint in terraform/environments/hmpps-domain-services
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

*****************************

Running tflint in terraform/environments/hmpps-domain-services/templates
Skipping folder as path name contains *templates*
tflint_exitcode=0

github-actions · 2023-11-28T16:20:39Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:

`Checkov Scan` Success

Show Output

*****************************

Checkov will check the following folders:

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:

github-actions · 2023-11-29T13:58:04Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:

`Checkov Scan` Success

Show Output

*****************************

Checkov will check the following folders:

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:

github-actions · 2023-11-29T15:15:49Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running TFSEC in terraform/environments/hmpps-oem
Excluding the following checks: AWS095

======================================================
tfsec is joining the Trivy family

tfsec will continue to remain available 
for the time being, although our engineering 
attention will be directed at Trivy going forward.

You can read more here: 
https://github.com/aquasecurity/tfsec/discussions/1994
======================================================
  timings
  ──────────────────────────────────────────
  disk i/o             1.41262ms
  parsing              138.437755ms
  adaptation           109.775µs
  checks               10.4848ms
  total                150.44495ms

  counts
  ──────────────────────────────────────────
  modules downloaded   0
  modules processed    5
  blocks processed     270
  files read           71

  results
  ──────────────────────────────────────────
  passed               1
  ignored              0
  critical             0
  high                 0
  medium               0
  low                  0


No problems detected!

tfsec_exitcode=0

`Checkov Scan` Success

Show Output

*****************************

Checkov will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running Checkov in terraform/environments/hmpps-oem
terraform scan results:

Passed checks: 107, Failed checks: 0, Skipped checks: 19


checkov_exitcode=0

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running tflint in terraform/environments/hmpps-oem
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

github-actions · 2023-11-29T15:36:31Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running TFSEC in terraform/environments/hmpps-oem
Excluding the following checks: AWS095

======================================================
tfsec is joining the Trivy family

tfsec will continue to remain available 
for the time being, although our engineering 
attention will be directed at Trivy going forward.

You can read more here: 
https://github.com/aquasecurity/tfsec/discussions/1994
======================================================
  timings
  ──────────────────────────────────────────
  disk i/o             3.874281ms
  parsing              134.916194ms
  adaptation           108.553µs
  checks               3.229211ms
  total                142.128239ms

  counts
  ──────────────────────────────────────────
  modules downloaded   0
  modules processed    5
  blocks processed     270
  files read           71

  results
  ──────────────────────────────────────────
  passed               1
  ignored              0
  critical             0
  high                 0
  medium               0
  low                  0


No problems detected!

tfsec_exitcode=0

`Checkov Scan` Success

Show Output

*****************************

Checkov will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running Checkov in terraform/environments/hmpps-oem
terraform scan results:

Passed checks: 107, Failed checks: 0, Skipped checks: 19


checkov_exitcode=0

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running tflint in terraform/environments/hmpps-oem
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

github-actions · 2023-11-29T15:37:23Z

`TFSEC Scan` Success

Show Output

*****************************

TFSEC will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running TFSEC in terraform/environments/hmpps-oem
Excluding the following checks: AWS095

======================================================
tfsec is joining the Trivy family

tfsec will continue to remain available 
for the time being, although our engineering 
attention will be directed at Trivy going forward.

You can read more here: 
https://github.com/aquasecurity/tfsec/discussions/1994
======================================================
  timings
  ──────────────────────────────────────────
  disk i/o             1.700228ms
  parsing              151.156867ms
  adaptation           114.474µs
  checks               4.172014ms
  total                157.143583ms

  counts
  ──────────────────────────────────────────
  modules downloaded   0
  modules processed    5
  blocks processed     270
  files read           71

  results
  ──────────────────────────────────────────
  passed               1
  ignored              0
  critical             0
  high                 0
  medium               0
  low                  0


No problems detected!

tfsec_exitcode=0

`Checkov Scan` Success

Show Output

*****************************

Checkov will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running Checkov in terraform/environments/hmpps-oem
terraform scan results:

Passed checks: 107, Failed checks: 0, Skipped checks: 19


checkov_exitcode=0

`CTFLint Scan` Success

Show Output

*****************************

Setting default tflint config...
Running tflint --init...
Installing `terraform` plugin...
Installed `terraform` (source: github.com/terraform-linters/tflint-ruleset-terraform, version: 0.2.1)
tflint will check the following folders:
terraform/environments/hmpps-oem

*****************************

Running tflint in terraform/environments/hmpps-oem
Excluding the following checks: terraform_unused_declarations
tflint_exitcode=0

Building out Datahub components.

6745664

github-actions bot added the environments-repository Used to exclude PRs from this repo in our Slack PR update label Nov 28, 2023

julialawrence changed the title ~~#data-platform/2454: Bulding out DataHub components~~ DPAT: Building out DataHub components Nov 28, 2023

julialawrence temporarily deployed to data-platform-apps-and-tools-development November 28, 2023 13:07 — with GitHub Actions Inactive

Defining role policy for ingestion.

7c8d53c

julialawrence had a problem deploying to data-platform-apps-and-tools-development November 28, 2023 16:20 — with GitHub Actions Failure

Adding RDS.

6de844a

julialawrence temporarily deployed to data-platform-apps-and-tools-development November 29, 2023 13:57 — with GitHub Actions Inactive

julialawrence added 2 commits November 29, 2023 15:12

Splitting out into component files.

2507a4b

Splitting out into component files.

c886f3d

julialawrence temporarily deployed to data-platform-apps-and-tools-development November 29, 2023 15:15 — with GitHub Actions Inactive

julialawrence marked this pull request as ready for review November 29, 2023 15:29

julialawrence requested review from a team as code owners November 29, 2023 15:29

julialawrence had a problem deploying to data-platform-apps-and-tools-development November 29, 2023 15:31 — with GitHub Actions Failure

julialawrence added 2 commits November 29, 2023 15:34

Adding newlines.

6175f86

Jacob doesn't like newlines.

d8cf8b0

julialawrence temporarily deployed to data-platform-apps-and-tools-development November 29, 2023 15:36 — with GitHub Actions Inactive

jacobwoffenden approved these changes Nov 29, 2023

View reviewed changes

julialawrence merged commit 86329bd into main Nov 29, 2023
12 checks passed

julialawrence deleted the feature/build-datahub branch November 29, 2023 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DPAT: Building out DataHub components #4162

DPAT: Building out DataHub components #4162

julialawrence commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023

github-actions bot commented Nov 28, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

DPAT: Building out DataHub components #4162

DPAT: Building out DataHub components #4162

Conversation

julialawrence commented Nov 28, 2023 • edited Loading

github-actions bot commented Nov 28, 2023

TFSEC Scan Success

Checkov Scan Failed

CTFLint Scan Success

github-actions bot commented Nov 28, 2023

TFSEC Scan Success

Checkov Scan Success

CTFLint Scan Success

github-actions bot commented Nov 29, 2023

TFSEC Scan Success

Checkov Scan Success

CTFLint Scan Success

github-actions bot commented Nov 29, 2023

TFSEC Scan Success

Checkov Scan Success

CTFLint Scan Success

github-actions bot commented Nov 29, 2023

TFSEC Scan Success

Checkov Scan Success

CTFLint Scan Success

github-actions bot commented Nov 29, 2023

TFSEC Scan Success

Checkov Scan Success

CTFLint Scan Success

julialawrence commented Nov 28, 2023 •

edited

Loading

`TFSEC Scan` Success

`Checkov Scan` Failed

`CTFLint Scan` Success

`TFSEC Scan` Success

`Checkov Scan` Success

`CTFLint Scan` Success

`TFSEC Scan` Success

`Checkov Scan` Success

`CTFLint Scan` Success

`TFSEC Scan` Success

`Checkov Scan` Success

`CTFLint Scan` Success

`TFSEC Scan` Success

`Checkov Scan` Success

`CTFLint Scan` Success

`TFSEC Scan` Success

`Checkov Scan` Success

`CTFLint Scan` Success