Skip to content

Commit

Permalink
Release v1.1.3 (#14)
Browse files Browse the repository at this point in the history
* feat: prepared release v1.1.3

* Update README.md (#13)

Fix typos

* chore: updated README.md

* chore: updated README.md

* fix: github workflow

* chore: updated README.md & CHANGELOG.md

* chore: updated README.md

---------

Co-authored-by: Luca Menichetti <[email protected]>
  • Loading branch information
gmuslia and meniluca authored Dec 14, 2023
1 parent 82ecd45 commit 1a195bc
Show file tree
Hide file tree
Showing 21 changed files with 10,305 additions and 6,674 deletions.
13 changes: 11 additions & 2 deletions .github/workflows/audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,19 @@ jobs:
uses: mikepenz/action-junit-report@0831a82caad2465c31c6dd929978f640cb42556c
if: success() || failure() # always run even if the previous step fails
with:
report_paths: 'junit-reports/*-junit-results.xml'
report_paths: 'junit-reports/bandit-junit-results.xml'
check_name: Bandit Security Findings
- name: Publish Security Scan Report in JUnit
uses: mikepenz/action-junit-report@0831a82caad2465c31c6dd929978f640cb42556c
if: success() || failure() # always run even if the previous step fails
with:
report_paths: 'junit-reports/semgrep-junit-results.xml'
check_name: Semgrep Security Findings
- name: Publish Security Scan Report in Checkstyle
uses: jwgmeligmeyling/checkstyle-github-action@a12be500c097a5cedab881d4785ef9b4a4d3ee6a
if: success() || failure() # always run even if the previous step fails
with:
path: 'junit-reports/*-checkstyle-results.xml'
path: 'junit-reports/shellcheck-checkstyle-results.xml'
name: Shellcheck Security Findings
title: Shellcheck Security Findings

14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ and this project adheres to [Semantic Versioning](http://semver.org/).

## [Released]

## [1.1.3] - 2023-12-14
In this release we have included bugfixes, updates to README as well as refreshed the local environment requirements and all the packages in the package.json and in the lambda layers to the latest available versions. We have also introduced a way how to override the minimum required version of NodeJS for some libraries we have a hard depdendency on and do not have yet the latest NodeJS in use.

### Added
- ```CodeCommitRepositoryAspects``` to override the NodeJS version for the Lambdas and the CustomResources deployed by the ```CodeCommitRepositoryConstruct``` to ```NODEJS_16_X```. Important to know is that NODEJS_16_X which will be deprecated (phase 1) on Jun 12, 2024 as per official documentation [here](https://docs.aws.amazon.com/lambda/latest/dg/lambda-nodejs.html). Check under **Known Issues** for more information regarding the **CodeCommitRepositoryConstruct: NODEJS_16_X support**
### Changed
- Updated local environment version requirements for docker, node, npm, python
- Updated ```scripts/proxy.sh``` to error out in case of any unhandled exceptions
- Wrapped up all the cdk commands to run from the shipped cdk version in the package.json (e.g: `npm run cdk`)
### Fixed
- `scripts/check-deps-python.sh` checks now all the folders that contains valid Python dependency definitions, not only the src/lambda-layer/common folder
- `scripts/check-code-scan-security.sh` local execution of scan execution with out silent mode for improved troubleshooting experience
- typo in wrapper script `scripts/cdk-synth-no-lookup.sh`

## [1.1.2] - 2023-11-29
In this release we have included bugfixes.

Expand Down
15,834 changes: 9,704 additions & 6,130 deletions NOTICE

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions OSS_License_Summary.csv
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@
"(MIT OR CC0-1.0)","5"
"(MIT OR GPL-3.0-or-later)","1"
"0BSD","1"
"Apache-2.0","25"
"Apache-2.0","23"
"BSD-2-Clause","11"
"BSD-3-Clause","18"
"CC-BY-3.0","1"
"CC-BY-4.0","1"
"CC0-1.0","1"
"ISC","48"
"MIT","525"
"ISC","49"
"MIT","520"
"MIT*","2"
"Python-2.0","1"
#########################
Expand All @@ -22,5 +22,5 @@
"License","Count"
"Apache Software License","3"
"Apache Software License; BSD License","2"
"BSD License","3"
"BSD License","2"
"MIT License","3"
45 changes: 25 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ You need to have the following dependencies in place:
- AWS Account (RES/DEV/INT/PROD)
- Mac OS / Cloud9 with Ubuntu Server 22.04 LTS Platform in RES Account
- Bash/ZSH terminal
- Docker version 20.10.x
- Docker version 24.0.x
- aws-cli v2 [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
- AWS credentials and profiles for each environment under ~/.aws/config [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
- Node v18.10.* && NPM v8.19.*
- Python 3.*
- Node v18.17.* && NPM v10.2.*
- Python >= 3.11
- Pipenv 2023.* [here](https://pipenv.pypa.io/en/latest/)
- jq command line JSON processor jq-1.5

Expand Down Expand Up @@ -73,9 +73,9 @@ If you do not have the AWS CodeStar connection ready please check on how to conf
### Optional: Determine VPC and Proxy settings for your pipeline
By default, the Pipeline is configured to run **without** a VPC. To run inside a VPC, there are two options: `VPC` and `VPC_FROM_LOOK_UP`. These options are configured using `scripts/configure.sh` described in the next section.

Use `VPC` if you want a single, self-contained pipeline running in a VPC. Not reccomended for use with multiple code pipelines in the same account. The VPC is created using defaulted settings.
Use `VPC` if you want a single, self-contained pipeline running in a VPC. Not recommended for use with multiple code pipelines in the same account. The VPC is created using defaulted settings.

Use `VPC_FROM_LOOK_UP` to look up an existing VPC based on its vpc ID. It is reccomended to create this VPC prior to deploying the pipeline. Multiple deployments of the pipeline can share the same VPC.
Use `VPC_FROM_LOOK_UP` to look up an existing VPC based on its vpc ID. It is recommended to create this VPC prior to deploying the pipeline. Multiple deployments of the pipeline can share the same VPC.

Note: Switching between VPC options may require a complete tear down and redeploy of the pipeline

Expand Down Expand Up @@ -133,21 +133,21 @@ create_compliance_log_bucket $INT_ACCOUNT_AWS_PROFILE $ACCOUNT_INT $AWS_REGION
#### Prepare the RES stage

```bash
cdk bootstrap --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} aws://${ACCOUNT_RES}/${AWS_REGION}
npm run cdk bootstrap -- --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} aws://${ACCOUNT_RES}/${AWS_REGION}
```

#### Prepare the DEV stage

```bash
cdk bootstrap --profile $DEV_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
npm run cdk bootstrap -- --profile $DEV_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust ${ACCOUNT_RES} aws://${ACCOUNT_DEV}/${AWS_REGION}
```

#### Prepare the INT stage

```bash
cdk bootstrap --profile $INT_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
npm run cdk bootstrap -- --profile $INT_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust ${ACCOUNT_RES} aws://${ACCOUNT_INT}/${AWS_REGION}
```
Expand All @@ -158,15 +158,15 @@ arn:aws:iam::aws:policy/AdministratorAccess \
*Make sure to check also the AppConfig.ts on how to enable PROD stage as there you also need to export more environment variables which are then used to propagate the ACCOUNT_PROD to the CDK Pipeline.

```bash
cdk bootstrap --profile prod --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
npm run cdk bootstrap -- --profile prod --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust resources_account_id aws://prod_account_id/your_aws_region
```

#### Deploy all the stacks

```bash
cdk deploy --all --region ${AWS_REGION} --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER}
npm run cdk deploy -- --all --region ${AWS_REGION} --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER}
```

Once the command finishes there will be the following CDK Stacks deployed into your RES Account:
Expand Down Expand Up @@ -321,20 +321,20 @@ source exports_vars.sh ### source the env vars with the right account ids and pr
npm run cdk synth ### this command generates the cdk.context.json
### 3. Add the cdk.context.json to git remote
git add cdk.context.json ### re-add cdk.context.json
git commit -am "Re-added cdk.context.json"
git commit -am "feat: re-added cdk.context.json"
git push -u origin ### Push changes to remote

```

### Working with Python dependencies
The project utilize the [Pipenv](https://pipenv.pypa.io/en/latest/). Pipenv automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your `Pipfile` as you install/uninstall packages. It also generates a project `Pipfile.lock`, which is used to produce deterministic builds.
The project utilizes the [Pipenv](https://pipenv.pypa.io/en/latest/). Pipenv automatically creates and manages a virtualenv for your project, as well as adds/removes packages from your `Pipfile` as you install/uninstall packages. It also generates a project `Pipfile.lock`, which is used to produce deterministic builds.

The Python dependencies are maintained in `Pipfile` instead of the `requirements.txt` file and requirements.txt files should not be commited into Git.
The Python dependencies are maintained in `Pipfile` instead of the `requirements.txt` file and requirements.txt files should not be commited to git.

#### How to install Pipenv
The recommended approach is to use `pip install pipenv -U` command. More information can be found [here](https://pipenv.pypa.io/en/latest/installation/#installing-pipenv).

The `pipenv` command is not added to the $PATH by default that need to be done manually. The `pipenv` command location can be determined by executing:
The `pipenv` command is not added to the $PATH by default, that needs to be done manually. The `pipenv` command location can be determined by executing the following:
```bash
python3 -m site --user-base
```
Expand All @@ -344,10 +344,10 @@ This will return a value like `/Users/user/Library/Python/3.11`. Then the $PATH
export PATH="${PATH}:/Users/user/Library/Python/3.11/bin";
```

You can add this your `$HOME/.zshrc` or `$HOME/.bashrc` to have this folder permanently.
You can add this to your `$HOME/.zshrc` or `$HOME/.bashrc` to have this folder permanently available.

#### Migrating existing `requirements.txt`
Existing `requirements.txt` can transformed into a `Pipfile` with the `pipenv install` command:
Existing `requirements.txt` can be transformed into a `Pipfile` with the `pipenv install` command, e.g:

```bash
cd path-to-the-module
Expand Down Expand Up @@ -418,8 +418,13 @@ The project applies the Day 0 mentality in Security. This means there are clearl

For more information, see [SECURITY](SECURITY.md)

## Common Issues
## Known Issues
Check below the list of all the known issues for which we do not yet have a final fix and/or take longer than 1 release cycle to completely fix them:
- **⚠ CodeCommitRepositoryConstruct: NODEJS_16_X support** If you are using AWS CodeCommit as repository type in your package.json (by setting "repositoryType": "CODECOMMIT") you need to know that we have overriden the Lambda runtime from NODEJS_14_X to NODEJS_16_X and also suppressed the false positive warnings for all the resources coming from the following construct: `CodeCommitRepositoryConstruct` in a separate construct: `CodeCommitRepositoryAspects` utilizing the [Aspects](https://docs.aws.amazon.com/cdk/v2/guide/aspects.html). The NODEJS_16_X runtime will be deprecated on Jun 12, 2024. This means that if you want to deploy new pipelines using CodeCommit the `CodeCommitRepositoryConstruct` will fail to deploy, the existing deployments won't be affected. Until we have a fix the only easy solution is the minimum version overriding from NODEJS_14_X to NODEJS_16_X (the overriding to NODEJS_18_X is not compatible to be done in this way, hence not done).


## Common Issues
Check below the list of potential issues you might encounter due to misconfigurations of your local environment when developing on top of the CICD Boot:
- When using Cloud9 in RES account and want to deploy the code cross-account then you need to define the profiles for the DEV and INT Account as usual (adding them in the ~/.aws/config). The RES profile can be omitted in this case while doing the initial bootstrap, except for the DEV and INT or PROD stages where the profile is mandatory to establish the trust between the RES account and the other environments (DEV/INT/PROD).
- `when calling the PutParameter operation: The security token included in the request is invalid`: This usually happens if you use Cloud9. Make sure to disable AWS managed temporary credentials and give the full admin access to your Cloud9 Managed role in order to be able to execute everything necessary. See the screenshot here: ![Disable AWS managed temporary credentials](docs/disable-managed-aws-creds.png "AWS managed temporary credentials")
- `Resource handler returned message: "Policy contains a statement with one or more invalid principals. (Service: Kms, Status Code: 400, Request ID: a9f9e73b-cf2c-4862-9536-af92aa0ed656)" (RequestToken: 949e9034-f910-7eb3-a4a2-427bc9e676b9, HandlerErrorCode: InvalidRequest)`
Expand Down Expand Up @@ -461,9 +466,9 @@ be consistent across those files.
* `npm run license:macos` validate the NOTICE file on MacOS systems
* `npm run lint` check for linting issues in the project
* `npm run lint:fix` fix linting issues in the project (do not forget to add & commit the fixed files)
* `cdk deploy` deploy this stack to your default AWS account/region
* `cdk diff` compare deployed stack with current state
* `cdk synth --all` emits the synthesized CloudFormation template for all stacks
* `npm run cdk deploy` deploy this stack to your default AWS account/region
* `npm run cdk diff` compare deployed stack with current state
* `npm run cdk synth -- --all` emits the synthesized CloudFormation template for all stacks

## Pointers to external documentation

Expand Down
1 change: 0 additions & 1 deletion bin/aspects.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ export class SecurityControls implements IAspect {
private readonly logRetentionInDays: string;
private readonly complianceLogBucketName: string;


constructor(kmsKey: aws_kms.Key, stage: string, logRetentionInDays: string, complianceLogBucketName: string) {
this.encryptionKey = kmsKey;
this.stage = stage;
Expand Down
2 changes: 1 addition & 1 deletion lib/stacks/app/LambdaStack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ export class LambdaStack extends cdk.Stack {

new lambda.Function(this, 'Function', {
functionName: `${props.applicationName}-${props.stageName}-test-lambda`,
runtime: lambda.Runtime.PYTHON_3_11,
runtime: lambda.Runtime.PYTHON_3_12,
code: lambda.Code.fromAsset('src/lambda-functions/test'),
handler: 'test-lambda.lambda_handler',
layers: [...layers],
Expand Down
4 changes: 2 additions & 2 deletions lib/stacks/app/constructs/PythonLambdaLayer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ export class PythonLambdaLayer extends lambda.LayerVersion {
super(scope, id, {
code: lambda.Code.fromAsset(props.folderPath, {
bundling: {
image: lambda.Runtime.PYTHON_3_11.bundlingImage,
image: lambda.Runtime.PYTHON_3_12.bundlingImage,
command: [
'bash', '-c',
// create a new virtualenv for python to use
Expand All @@ -29,7 +29,7 @@ export class PythonLambdaLayer extends lambda.LayerVersion {
],
},
}),
compatibleRuntimes: [lambda.Runtime.PYTHON_3_11],
compatibleRuntimes: [lambda.Runtime.PYTHON_3_11, lambda.Runtime.PYTHON_3_12],
description: props.description,
});
}
Expand Down
79 changes: 79 additions & 0 deletions lib/stacks/core/constructs/CodeCommitRepositoryAspects.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: MIT-0

import { aws_lambda, IAspect, Annotations, CustomResourceProvider, CfnResource } from 'aws-cdk-lib';
import { CfnFunction, Function, RuntimeFamily } from 'aws-cdk-lib/aws-lambda';
import { IConstruct } from 'constructs';

export class CodeCommitRepositoryAspects implements IAspect {

constructor(readonly minimumNodeRuntimeVersion : aws_lambda.Runtime = aws_lambda.Runtime.NODEJS_16_X) {
}

public visit(node: IConstruct): void {
this.overrideNodeJsVersion(node);
}

private parseNodeRuntimeVersion(runtimeName: string): number {
const runtimeVersion = runtimeName.replace('nodejs', '').split('.')[0];
return +runtimeVersion;
}

private overrideNodeJsVersionCFNFunction(node: IConstruct) {
if (node instanceof CfnFunction) {
if (!node.runtime) {
throw new Error(`Runtime not specified for ${node.node.path}`);
}

if (!node.runtime.includes('nodejs')) return;

const actualNodeJsRuntimeVersion = this.parseNodeRuntimeVersion(node.runtime);
const minimumNodeJsRuntimeVersion = this.parseNodeRuntimeVersion(this.minimumNodeRuntimeVersion.name);

if (actualNodeJsRuntimeVersion < minimumNodeJsRuntimeVersion) {
node.runtime = this.minimumNodeRuntimeVersion.name;
Annotations
.of(node)
.addInfo(`Node.js runtime version was changed to the minimum required: ${this.minimumNodeRuntimeVersion.name}.`);
}
}
}

private overrideNodeJsVersionFunction(node: IConstruct) {
if (node instanceof Function) {
if (!node.runtime) {
throw new Error(`Runtime not specified for ${node.node.path}`);
}

if (node.runtime.family != RuntimeFamily.NODEJS) return;

const actualNodeJsRuntimeVersion = this.parseNodeRuntimeVersion(node.runtime.name);
const minimumNodeJsRuntimeVersion = this.parseNodeRuntimeVersion(this.minimumNodeRuntimeVersion.name);

if (actualNodeJsRuntimeVersion < minimumNodeJsRuntimeVersion) {
(node as any).runtime = this.minimumNodeRuntimeVersion;
Annotations
.of(node)
.addInfo(`Node.js runtime version was changed to the minimum required: ${this.minimumNodeRuntimeVersion.name}.`);
}
}
}

private overrideNodeJsVersionCustomResource(node: IConstruct) {
if (node instanceof CustomResourceProvider) {
const cfnFunction = node.node.findChild('Handler') as CfnResource;
try {
cfnFunction.addOverride('Properties.Runtime', this.minimumNodeRuntimeVersion.name);
} catch (warning) {
Annotations.of(node).addInfo(`Node.js runtime version was changed to the minimum ${this.minimumNodeRuntimeVersion.name}.`);
}
}
}

private overrideNodeJsVersion(node: IConstruct) {
this.overrideNodeJsVersionCFNFunction(node);
this.overrideNodeJsVersionFunction(node);
this.overrideNodeJsVersionCustomResource(node);
}
}

Loading

0 comments on commit 1a195bc

Please sign in to comment.