Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parameters): ability to set maxAge and decrypt via environment variables #1384

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,20 +293,22 @@ Core utilities such as Tracing, Logging, and Metrics will be available across al
???+ info
Explicit parameters take precedence over environment variables

| Environment variable | Description | Utility | Default |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------- | ------------------- |
| **POWERTOOLS_SERVICE_NAME** | Sets service name used for tracing namespace, metrics dimension and structured logging | All | `service_undefined` |
| **POWERTOOLS_METRICS_NAMESPACE** | Sets namespace used for metrics | [Metrics](./core/metrics) | `default_namespace` |
| **POWERTOOLS_TRACE_ENABLED** | Explicitly disables tracing | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_RESPONSE** | Captures Lambda or method return as metadata. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_ERROR** | Captures Lambda or method exception as metadata. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS** | Captures HTTP(s) requests as segments. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_LOGGER_LOG_EVENT** | Logs incoming event | [Logger](./core/logger) | `false` |
| **POWERTOOLS_LOGGER_SAMPLE_RATE** | Debug log sampling | [Logger](./core/logger) | `0` |
| **POWERTOOLS_DEV** | Increase JSON indentation to ease debugging when running functions locally or in a non-production environment | [Logger](./core/logger) | `false` |
| **LOG_LEVEL** | Sets logging level | [Logger](./core/logger) | `INFO` |

Each Utility page provides information on example values and allowed values
| Environment variable | Description | Utility | Default |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------------------- |
| **POWERTOOLS_SERVICE_NAME** | Sets service name used for tracing namespace, metrics dimension and structured logging | All | `service_undefined` |
| **POWERTOOLS_METRICS_NAMESPACE** | Sets namespace used for metrics | [Metrics](./core/metrics) | `default_namespace` |
| **POWERTOOLS_TRACE_ENABLED** | Explicitly disables tracing | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_RESPONSE** | Captures Lambda or method return as metadata. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_ERROR** | Captures Lambda or method exception as metadata. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS** | Captures HTTP(s) requests as segments. | [Tracer](./core/tracer) | `true` |
| **POWERTOOLS_LOGGER_LOG_EVENT** | Logs incoming event | [Logger](./core/logger) | `false` |
| **POWERTOOLS_LOGGER_SAMPLE_RATE** | Debug log sampling | [Logger](./core/logger) | `0` |
| **POWERTOOLS_DEV** | Increase JSON indentation to ease debugging when running functions locally or in a non-production environment | [Logger](./core/logger) | `false` |
| **LOG_LEVEL** | Sets logging level | [Logger](./core/logger) | `INFO` |
| **POWERTOOLS_PARAMETERS_MAX_AGE** | Adjust how long values are kept in cache (in seconds) | [Parameters](./utilities/parameters) | `5` |
| **POWERTOOLS_PARAMETERS_SSM_DECRYPT** | Sets whether to decrypt or not values retrieved from AWS Systems Manager Parameters Store | [Parameters](./utilities/parameters) | `false` |

Each Utility page provides information on example values and allowed values.

## Tenets

Expand Down
8 changes: 4 additions & 4 deletions docs/snippets/parameters/adjustingCacheTTL.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ import { SSMProvider } from '@aws-lambda-powertools/parameters/ssm';
const parametersProvider = new SSMProvider();

export const handler = async (): Promise<void> => {
// Retrieve a single parameter
const parameter = await parametersProvider.get('/my/parameter', { maxAge: 60 }); // 1 minute
// Retrieve a single parameter and cache it for 1 minute
const parameter = await parametersProvider.get('/my/parameter', { maxAge: 60 }); // (1)
console.log(parameter);

// Retrieve multiple parameters from a path prefix
const parameters = await parametersProvider.getMultiple('/my/path/prefix', { maxAge: 120 }); // 2 minutes
// Retrieve multiple parameters from a path prefix and cache them for 2 minutes
const parameters = await parametersProvider.getMultiple('/my/path/prefix', { maxAge: 120 });
for (const [ key, value ] of Object.entries(parameters || {})) {
console.log(`${key}: ${value}`);
}
Expand Down
2 changes: 1 addition & 1 deletion docs/snippets/parameters/ssmProviderDecryptAndRecursive.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { SSMProvider } from '@aws-lambda-powertools/parameters/ssm';
const parametersProvider = new SSMProvider();

export const handler = async (): Promise<void> => {
const decryptedValue = await parametersProvider.get('/my/encrypted/parameter', { decrypt: true });
const decryptedValue = await parametersProvider.get('/my/encrypted/parameter', { decrypt: true }); // (1)
console.log(decryptedValue);

const noRecursiveValues = await parametersProvider.getMultiple('/my/path/prefix', { recursive: false });
Expand Down
16 changes: 13 additions & 3 deletions docs/utilities/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,17 +127,22 @@ The following will retrieve the latest version and store it in the cache.

### Adjusting cache TTL

???+ tip
`maxAge` parameter is also available in high level functions like `getParameter`, `getSecret`, etc.

By default, the provider will cache parameters retrieved in-memory for 5 seconds.

You can adjust how long values should be kept in cache by using the param `maxAge`, when using `get()` or `getMultiple()` methods across all providers.

???+ tip
If you want to set the same TTL for all parameters, you can set the `POWERTOOLS_PARAMETERS_MAX_AGE` environment variable. **This will override the default TTL of 5 seconds but can be overridden by the `maxAge` parameter**.

```typescript hl_lines="7 11" title="Caching parameters values in memory for longer than 5 seconds"
--8<-- "docs/snippets/parameters/adjustingCacheTTL.ts"
```

1. Options passed to `get()`, `getMultiple()`, and `getParametersByName()` will override the values set in `POWERTOOLS_PARAMETERS_MAX_AGE` environment variable.

???+ info
The `maxAge` parameter is also available in high level functions like `getParameter`, `getSecret`, etc.

### Always fetching the latest

If you'd like to always ensure you fetch the latest parameter from the store regardless if already available in cache, use the `forceFetch` parameter.
Expand Down Expand Up @@ -166,10 +171,15 @@ The AWS Systems Manager Parameter Store provider supports two additional argumen
| **decrypt** | `false` | Will automatically decrypt the parameter (see required [IAM Permissions](#iam-permissions)). |
| **recursive** | `true` | For `getMultiple()` only, will fetch all parameter values recursively based on a path prefix. |

???+ tip
If you want to always decrypt parameters, you can set the `POWERTOOLS_PARAMETERS_SSM_DECRYPT=true` environment variable. **This will override the default value of `false` but can be overridden by the `decrypt` parameter**.

```typescript hl_lines="6 9" title="Example with get() and getMultiple()"
--8<-- "docs/snippets/parameters/ssmProviderDecryptAndRecursive.ts"
```

1. Options passed to `get()`, `getMultiple()`, and `getParametersByName()` will override the values set in `POWERTOOLS_PARAMETERS_SSM_DECRYPT` environment variable.

#### SecretsProvider

```typescript hl_lines="4-5" title="Example with SecretsProvider for further extensibility"
Expand Down
2 changes: 2 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions packages/commons/src/config/ConfigService.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Housekeeping

Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,25 @@ abstract class ConfigService {
*/
public abstract getServiceName(): string;

/**
* It returns the value of the _X_AMZN_TRACE_ID environment variable.
*
* The AWS X-Ray Trace data available in the environment variable has this format:
* `Root=1-5759e988-bd862e3fe1be46a994272793;Parent=557abcec3ee5a047;Sampled=1`,
*
* The actual Trace ID is: `1-5759e988-bd862e3fe1be46a994272793`.
*
* @returns {string|undefined}
*/
public abstract getXrayTraceId(): string | undefined;

/**
* It returns true if the string value represents a boolean true value.
*
* @param {string} value
* @returns boolean
*/
public abstract isValueTrue(value: string): boolean;
}

export {
Expand Down
12 changes: 12 additions & 0 deletions packages/commons/src/config/EnvironmentVariablesService.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this PR this method was used only in Logger. Now it's used also by Parameters so I am extracting it to the commons package to avoid duplication.

Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,18 @@ class EnvironmentVariablesService extends ConfigService {
return xRayTraceId.split(';')[0].replace('Root=', '');
}

/**
* It returns true if the string value represents a boolean true value.
*
* @param {string} value
* @returns boolean
*/
public isValueTrue(value: string): boolean {
const truthyValues: string[] = [ '1', 'y', 'yes', 't', 'true', 'on' ];

return truthyValues.includes(value.toLowerCase());
}

}

export {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,31 @@ describe('Class: EnvironmentVariablesService', () => {

});

describe('Method: isValueTrue', () => {

const valuesToTest: Array<Array<string | boolean>> = [
[ '1', true ],
[ 'y', true ],
[ 'yes', true ],
[ 't', true ],
[ 'TRUE', true ],
[ 'on', true ],
[ '', false ],
[ 'false', false ],
[ 'fasle', false ],
[ 'somethingsilly', false ],
[ '0', false ]
];

test.each(valuesToTest)('it takes string "%s" and returns %s', (input, output) => {
// Prepare
const service = new EnvironmentVariablesService();
// Act
const value = service.isValueTrue(input as string);
// Assess
expect(value).toBe(output);
});

});

});
12 changes: 0 additions & 12 deletions packages/logger/src/config/EnvironmentVariablesService.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Original file line number Diff line number Diff line change
Expand Up @@ -117,18 +117,6 @@ class EnvironmentVariablesService extends CommonEnvironmentVariablesService impl
return this.isValueTrue(value);
}

/**
* It returns true if the string value represents a boolean true value.
*
* @param {string} value
* @returns boolean
*/
public isValueTrue(value: string): boolean {
const truthyValues: string[] = [ '1', 'y', 'yes', 't', 'true', 'on' ];

return truthyValues.includes(value.toLowerCase());
}

}

export {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -249,31 +249,4 @@ describe('Class: EnvironmentVariablesService', () => {

});

describe('Method: isValueTrue', () => {

const valuesToTest: Array<Array<string | boolean>> = [
[ '1', true ],
[ 'y', true ],
[ 'yes', true ],
[ 't', true ],
[ 'TRUE', true ],
[ 'on', true ],
[ '', false ],
[ 'false', false ],
[ 'fasle', false ],
[ 'somethingsilly', false ],
[ '0', false ]
];

test.each(valuesToTest)('it takes string "%s" and returns %s', (input, output) => {
// Prepare
const service = new EnvironmentVariablesService();
// Act
const value = service.isValueTrue(input as string);
// Assess
expect(value).toBe(output);
});

});

});
1 change: 1 addition & 0 deletions packages/parameters/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
"aws-sdk-client-mock-jest": "^2.0.1"
},
"dependencies": {
"@aws-lambda-powertools/commons": "^1.7.0",
"@aws-sdk/util-base64-node": "^3.209.0"
}
}
14 changes: 11 additions & 3 deletions packages/parameters/src/BaseProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@ import { GetMultipleOptions } from './GetMultipleOptions';
import { ExpirableValue } from './ExpirableValue';
import { TRANSFORM_METHOD_BINARY, TRANSFORM_METHOD_JSON } from './constants';
import { GetParameterError, TransformParameterError } from './Exceptions';
import type { BaseProviderInterface, GetMultipleOptionsInterface, GetOptionsInterface, TransformOptions } from './types';
import { EnvironmentVariablesService } from './config/EnvironmentVariablesService';
import type {
BaseProviderInterface,
GetMultipleOptionsInterface,
GetOptionsInterface,
TransformOptions
} from './types';

// These providers are dinamycally intialized on first use of the helper functions
const DEFAULT_PROVIDERS: Record<string, BaseProvider> = {};
Expand All @@ -29,10 +35,12 @@ const DEFAULT_PROVIDERS: Record<string, BaseProvider> = {};
* this should be an acceptable tradeoff.
*/
abstract class BaseProvider implements BaseProviderInterface {
public envVarsService: EnvironmentVariablesService;
protected store: Map<string, ExpirableValue>;

public constructor() {
this.store = new Map();
this.envVarsService = new EnvironmentVariablesService();
}

/**
Expand Down Expand Up @@ -62,7 +70,7 @@ abstract class BaseProvider implements BaseProviderInterface {
* @param {GetOptionsInterface} options - Options to configure maximum age, trasformation, AWS SDK options, or force fetch
*/
public async get(name: string, options?: GetOptionsInterface): Promise<undefined | string | Uint8Array | Record<string, unknown>> {
const configs = new GetOptions(options);
const configs = new GetOptions(options, this.envVarsService);
const key = [ name, configs.transform ].toString();

if (!configs.forceFetch && !this.hasKeyExpiredInCache(key)) {
Expand Down Expand Up @@ -97,7 +105,7 @@ abstract class BaseProvider implements BaseProviderInterface {
* @returns
*/
public async getMultiple(path: string, options?: GetMultipleOptionsInterface): Promise<undefined | Record<string, unknown>> {
const configs = new GetMultipleOptions(options || {});
const configs = new GetMultipleOptions(options, this.envVarsService);
const key = [ path, configs.transform ].toString();

if (!configs.forceFetch && !this.hasKeyExpiredInCache(key)) {
Expand Down
21 changes: 11 additions & 10 deletions packages/parameters/src/GetMultipleOptions.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a lot of duplication with the GetOptions class here, so opted for inheritance.

Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
import { DEFAULT_MAX_AGE_SECS } from './constants';
import type { GetMultipleOptionsInterface, TransformOptions } from './types';
import { GetOptions } from './GetOptions';
import { EnvironmentVariablesService } from './config/EnvironmentVariablesService';
import type { GetMultipleOptionsInterface } from './types';

/**
* Options for the `getMultiple` method.
*
* It merges the default options with the provided options.
* Extends the `GetOptions` class and adds the `throwOnTransformError` option.
*/
class GetMultipleOptions implements GetMultipleOptionsInterface {
public forceFetch: boolean = false;
public maxAge: number = DEFAULT_MAX_AGE_SECS;
public sdkOptions?: unknown;
class GetMultipleOptions extends GetOptions implements GetMultipleOptionsInterface {
public throwOnTransformError: boolean = false;
public transform?: TransformOptions;

public constructor(options: GetMultipleOptionsInterface) {
Object.assign(this, options);
public constructor(options: GetMultipleOptionsInterface = {}, envVarsService: EnvironmentVariablesService) {
super(options, envVarsService);

if (options.throwOnTransformError !== undefined) {
this.throwOnTransformError = options.throwOnTransformError;
}
}
}

Expand Down
9 changes: 7 additions & 2 deletions packages/parameters/src/GetOptions.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { DEFAULT_MAX_AGE_SECS } from './constants';
import { EnvironmentVariablesService } from './config/EnvironmentVariablesService';
import type { GetOptionsInterface, TransformOptions } from './types';

/**
Expand All @@ -8,12 +9,16 @@ import type { GetOptionsInterface, TransformOptions } from './types';
*/
class GetOptions implements GetOptionsInterface {
public forceFetch: boolean = false;
public maxAge: number = DEFAULT_MAX_AGE_SECS;
public maxAge!: number;
public sdkOptions?: unknown;
public transform?: TransformOptions;

public constructor(options: GetOptionsInterface = {}) {
public constructor(options: GetOptionsInterface = {}, envVarsService: EnvironmentVariablesService) {
Object.assign(this, options);

if (options.maxAge === undefined) {
this.maxAge = envVarsService.getParametersMaxAge() ?? DEFAULT_MAX_AGE_SECS;
}
}
}

Expand Down
9 changes: 4 additions & 5 deletions packages/parameters/src/appconfig/AppConfigProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,8 @@ import type {
*/
class AppConfigProvider extends BaseProvider {
public client: AppConfigDataClient;
protected configurationTokenStore: Map<string, string> = new Map();
protected valueStore: Map<string, Uint8Array> = new Map();
protected configurationTokenStore = new Map<string, string>();
protected valueStore = new Map<string, Uint8Array>();
private application?: string;
private environment: string;

Expand All @@ -187,13 +187,12 @@ class AppConfigProvider extends BaseProvider {
this.client = new AppConfigDataClient(options.clientConfig || {});
}

if (!options?.application && !process.env['POWERTOOLS_SERVICE_NAME']) {
this.application = options?.application || this.envVarsService.getServiceName();
if (!this.application || this.application.trim().length === 0) {
throw new Error(
'Application name is not defined or POWERTOOLS_SERVICE_NAME is not set'
);
}
this.application =
options.application || process.env['POWERTOOLS_SERVICE_NAME'];
this.environment = options.environment;
}

Expand Down
Loading