diff --git a/design/external-references.md b/design/external-references.md new file mode 100644 index 0000000000000..c4a00d0cba3a8 --- /dev/null +++ b/design/external-references.md @@ -0,0 +1,400 @@ +# RFC: External Resource References + +## Background + +PR [#1436](https://github.com/awslabs/aws-cdk/pull/1436) had recently introduced implicit +references of resource attributes across stacks within the same CDK app: + + +```ts +const user = new iam.User(stackWithUser, 'User'); +const group = new iam.Group(stackWithGroup, 'Group'); + +group.addUser(user); +``` + +Will synthesize the following two templates: + +```yaml +Outputs: + ExportsOutputRefGroupC77FDACD8CF7DD5B: + Value: + Ref: GroupC77FDACD + Export: + Name: ExportsOutputRefGroupC77FDACD8CF7DD5B +Resources: + GroupC77FDACD: + Type: AWS::IAM::Group +``` + +and: + +```yaml +Resources: + User00B015A1: + Type: AWS::IAM::User + Properties: + Groups: + - Fn::ImportValue: ExportsOutputRefGroupC77FDACD8CF7DD5B +``` + +You can see that the reference automatically resulted in an `Output+Export` and +an `Fn::ImportValue` to be used in order to share this information across the +stacks within the same application. + +This mechanism practically replaces our "import/export" pattern which looks like this (same use case and same synthesized output): + +```ts +const user = new iam.User(stackWithUser, 'User'); +const group = new iam.Group(stackWithGroup, 'Group'); + +// creates an `Output` in "stackWithUser" with a unique export name +// returns an `Fn::ImportValue` that can be used to import this user +// in any stack in the same account/region +const importValues = user.export(); + +// instantiates an `IUser` whose attributes are `Fn::ImportValue` tokens +// in "stackWithGroup", and it can now be used "naturally" in that stack +const importedUser = iam.User.import(stackWithGroup, 'ImportedUser', importValues); +group.addUser(importedUser); +``` + +Obvsiouly the world is a better place now. Thanks [@rix0rrr](https://github.com/users/rix0rrr). + +## Problem Statement + +So, "what seems to be the problem?", you ask. Well, we want more! The current model only supports references that occur between stacks within the same CDK app. What about the use case where I want to reference a resource that was created in a totally different app? + +A common example is for a common infrastructure team to maintain a CDK app which defines networking resources such as VPCs, security groups, LBs, etc and application teams need to reference these resources in their own CDK apps. + +__So the problem we are trying to solve is how to enable users to reference resources (and potentially higher level constructs) across CDK apps.__ + +## Scope + +As an initial step, we'll address the use case of referencing resources across within __the same AWS environment__ (account + region). This limitation is based on the limitation AWS CloudFormation has for exports/imports. + +We differentiate two types of references: + +1. Reference a resource created through another CDK app. +2. Reference a resource created by some other means. + +In the former case (app to app), the producing app can explicitly "export" the resource and the consuming app can explicitly "import" the resource. + +In the latter case (environment to app), we usually just have a resource's physical identity such as ARN or name and we want to be able to use it in a CDK app. + +## Requirements + +- **REQ1**: Allow constructs to be published from one CDK app and consumed by other CDK apps within the same environment +- **REQ2**: Support "simple" constructs, which are resources that can be represented by a set of attributes such as an ARN and also support "composite" constructs, which are resources that encapsulate other resources (such as security groups, roles, etc). +- **REQ3**: Allow CDK apps to reference resources that were created elsewhere (by other CDK apps, manually or any other mean) by referencing the physical identity. For example, an S3 Bucket ARN should be sufficient in order to reference an existing S3 bucket. +- **REQ4**: Implement the mechanism such that it can be later used to publish and consume constructs through other key/value mechanism such as environment variables, SSM parameters, etc. +- **REQ5**: If a resource is exported twice under the same export name, only a single set of outputs will be created ([#1496](https://github.com/awslabs/aws-cdk/issues/1496)). +- **REQ6**: If the same resource is imported twice into the same stack, the "import" method **may** return the same object (as in, the same instance). +- **REQ7**: For some resources (like VPC for example), it should be possible look up the resource in the current environment by querying through an environmental context provider. +- **REQ8**: It should be possible to resolve import values either during synthesis time (i.e. via an environmental context provider) or during deploy time (i.e. via `Fn::ImportValue`). There are certain resources that will _require_ the use of synthesis-time resolution due to their complex representation. + +## Approach + +As usual, we'll take a layered approach. + +**Serializable constructs**: At the low-level, we'll define what it means for a construct to be "__serializable__" through string key/value context. The mechanism will be recursive such that it will be possible to write/read singular values and write/read serializable objects (which, themselves will write/read their own values and so forth). + +**Serialization context**: the notion of a "string key/value context" represents the lowest common denominator for serialization, and specifically is the only type supported by CloudFormation's cross-stack export/import mechanism (CloudFormation outputs can only be of "string" type). If a construct can serialize itself through a set of key/values, we can pass it via CloudFormation's import/export, environment variables, SSM parameters, etc. + +**CloudFormation imports/exports as a serialization context**: on top of that, we will implement a serialization/deserialization context based on AWS CloudFormation's import/exports mechanism. "writing" a value means defining an Output with an Export and "reading" a value means `Fn::ImportValue` with this export name. The "export name" will be used as a prefix that represent an object, and sub-objects will be serialized by adding another component to the export name. + +**Synthesis-time Imports**: we will also define an environmental context provider that will import values during synthesis and store them in `cdk.json`. This approach is more robust since it will allow CDK code to reason about concrete values instead of opaque tokens. It's especially needed for situations where a list of values needs to be passed and the arity is required. + +**Convenience methods**: now that constructs can be serializable through imports/exports, we can implement a set of convenience methods for each AWS resource to provide nice ergonomics for the specific use case of exporting and importing resources across apps. + +## Design + +### Serialization + +If a construct is said to be serializable, it must implement the `ISerializable` interface: + +```ts +interface ISerializable { + serialize(ctx: ISerializationContext): void; +} +``` + +When an object is serialized, the `serialize` method is called with an object which implements the following interface: + +```ts +interface ISerializationContext { + writeString(key: string, value: string, options?: SerializationOptions): void + writeStringList(key: string, value: string[], options?: SerializationOptions): void; + writeObject(key: string, obj: ISerializable, options?: SerializationOptions): void; +} + +interface SerializationOptions { + description?: string; +} +``` + +The serialization context allows the object serialize itself through key/value strings via calls to `writeString(key, value)` and `writeStringList(key, array)`. If the object encapsulates another serializable object, it can use `writeObject(key, obj)`, which will result in a subsequent call to the sub-object's `serialize` method with an appropriate context. + +### Deserialization + +To support deserialization, classes must also include a public static `deserializeXxx` method which reads the object from a deserialization context and returns an object that implements the resource type interface: + +> The reason we indicate the type in the method name is because static methods in JavaScript are inherited, so we can differentiate between `ExtraBucket.deserializeExtraBucket` and `ExtraBucket.deserizlizeBucket`. + +```ts +class MyResource extends Construct implements ISerializable { + static deserializeMyResource(ctx: IDeserializationContext): IMyResource; +} +``` + +The deserialization context is an object that implements the following interface: + +```ts +interface IDeserializationContext { + scope: Construct; + id: string; + readString(key: string, options?: DeserializationOptions): string; + readStringList(key: string, options?: DeserializationOptions): string[]; + readObject(key: string): IDeserializationContext; +} + +interface DeserializationOptions { + allowUnresolved?: boolean; +} +``` + +The method `readString` can be used to read values stored by `writeString`. + +The method `readStringList` can be used to read string list values stored by `writeStringList`. + +The method `readObject` returns a deserialization context for composite deserialization written via `writeObject`. + +The `allowUnresolved` option can be used by constructs to indicate that returned value __must be a resolved value__ (i.e. not a token). This implies, for example, that when importing this value, users cannot use the `resolveType: Deployment` option (REQ8). + +The CloudFormation import/export serializer is unable to support unresolved imports for string lists, so `allowUnresolved` must be either undefined or set to `false`. If it is set to `true` and `readStringList` is used, an error will be thrown. + +Since `deserializeXxx` will need to create new construct objects, the deserialization context will supply a consistent `scope` and `id` which can be used to instantiate a construct object that represents this object. For example, `scope` can be mapped to the current `Stack` and `id` can be mapped to `exportName` which is ensured to be unique within the environment (and therefore, the current stack). + +Implementers of `deserializeXxx` should check if a construct with `id` already exists within `scope` and return it instead of instantiating and new object (REQ6). + +--- + +Here's an example that demonstrates how serialization and deserialization can be implemented for an application load balancer (ALB), which is a composite resource. As you can see, ALBs encapsulate a security group, which is serialized and deserialized together with the ALB itself. + +```ts +class ApplicationLoadBalancer { + serialize(ctx: ISerializationContext): void { + ctx.writeString('LoadBalancerArn', this.loadBalancerArn); + ctx.writeObject('SecurityGroup', this.securityGroup); + } + + static deserializeApplicationLoadBalancer(ctx: IDeserializationContext): IApplicationLoadBalancer { + const exists = ctx.scope.findChild(ctx.id); + if (exists) { + return exists; + } + + return new ImportedApplicationLoadBalancer(ctx.scope, ctx.id, { + loadBalancerArn: ctx.readString('LoadBalancerArn'), + securityGroup: ec2.SecurityGroup.deserializeSecurityGroup(ctx.readObject('SecurityGroup')) + }); + } +} +``` + +### Imports/Exports + +Now that constructs can be serialized and deserialized into a key-value context, we can implement a serialization mechanism for exporting resources from stacks and importing them in another stack. Importing can be done either at synthesis time using an environmental context provider or at runtime by returning `Fn::ImportValue` tokens for `readString`. + +The following methods will be added to the `Stack` class: + +```ts +class Stack { + exportString(exportName: string, value: string, options?: ExportOptions): void; + importString(exportName: string, options?: ImportOptions): string; +} + +interface ExportOptions { + description?: string +} + +interface ImportOptions { + type?: ResolveType // default is Synthesis + weak?: boolean; // default is "false" +} + +enum ResolveType { + Synthesis, + Deployment +} +``` + +The `exportString` method creates an AWS CloudFormation Output for this value assigned to this export name. + +The `importString` method returns the value for this specific export name. When importing a value, users can specify the following `ImportOptions`: + +* If `type` is set to `Deployment`, the method will return an `Fn::ImportValue(exportName)` token. This means that CDK code cannot reason about the concrete value, which will only be resolved when the stack is deployed. +* If `type` is set to `Synthesis` (default) the method will exercise an environmental context provider to look up the export value __during synthesis__. The concrete value will be propagated to the CDK app and can be reasoned about like any normal value. +* The `weak` option is only relevant for synth-time resolution. If it is `false` (which is the default), the CDK will automatically embed a `Metadata` entry on the consuming resource with an `Fn::ImportValue`. This will force CloudFormation to take a strong reference on the export, even through the actual value is concretely resolved during synthesis. This ensures, for example, that the producing stack can't be deleted as long as there stacks consuming the exported values. This behavior (which is the default), can be disabled by settings `weak: true`, in which case the `Fn::ImportValue` will simply not be included. + +On top of these two methods, we can now define import and export methods for serializable objects: + +```ts +class Stack { + exportObject(exportName: string, obj: ISerializable): void; + importObject(exportName: string, options?: ImportOptions): IDeserializationContext; +} +``` + +The `exportObject` method will invoke `obj.serialize` with a serialization context "bound" to this export name. This means the export name will be used as a prefix to all written keys. `writeObject` will be implemented with a nested serialization context that adds another component to the export name prefix. + +The `importObject` method will be used like this: + +```ts +const importedBucket = Bucket.deserializeBucket(stack.importObject('MyBucketExportName')); +``` + +The method will return a deserialization context that's bound to the export name. Similarly to the serialization context, it will prefix all values read through `readString` with the export name, and so forth with `readObject`. + +As mentioned about, the default resolve type for imports is `Synthesis` (with strong-references). This means that the values returned by `readString` will be actual concrete values. If users opt-in to deploy-time resolution (by setting using `Deployment` resolve type), the values returned will be tokens. In some cases this would be fine, but there could be constructs that cannot deal with opaque values (i.e. if the value is an list of strings and needs to be deconstructed). In those cases (REQ8), constructs should invoke `readString` with `allowUnresolved: false` to indicate that this specific value cannot be a token. + +#### Export names + +AWS CloudFormation export names must be unique within an environment (account/region), and they will be formed by concatenating root `exportName` and all the keys that lead to a value in the serialization tree. + +We will use `-` as a component separator devising fully qualified export names. To avoid collisions, if the main export name or any subsequent serialization key includes a `-` it will be removed. + +Since AWS CloudFormation has a limit on export name length, and we wouldn't want to restrict the serialization depth, the import/export serializer should trim the name and add a hash of the full name, but only if the total length exceeds the limit. + +### Synthesis-time Imports (REQ8) + +As mentioned in the previous section, `importObject` will support both synthesis and deploy-time imports by export name. + +In order to implement synthesis-time imports, we will add a new environmental context provider to the toolkit which will be able to retrieve a value for a certain CloudFormation named export. + +The implementation of this provider will use the CloudFormation ListExports operation to find the exports needed and pass their values in through the CDK context mechanism. + +Bear in mind that once a value has been retrieved, it will automatically be saved in the local `cdk.json` and won't be retrieved again until `cdk context --reset` is called. + +Since synthesis-time resolution doesn't create strong coupling between the stacks at the CloudFormation +level, production and operational issues can arise if the producing stack deletes an exported resource. On the other hand, we hear from customers that strong-referencing behavior of `Fn::ImportValue` is sometimes a curse. Customers tell us that they found themselves stuck with unremovable or stacks that cannot be updates due to imports. + +Luckily, we can enable both capabilities. By default, when synthesis-time resolution is used, the CDK will automatically add a resource metadata entry to the template with an `Fn::ImportValue`. This will create the strong coupling between the stacks. Users can opt-out of this behavior by setting `weak: true` when they import the resource. + +### Convenience Methods + +At the top layer, we will implement a bunch of convenience methods for each AWS resource will provide nice ergonomics for cross-app import/export: + +Here's the usage: + +```ts +// this will export "myAwesomeBucket" under the export name "JohnnyBucket" +myAwesomeBucket.exportBucket('JohnnyBucket'); + +// now, any CDK app that wishes to refer to this bucket can do this: +const importedBucket: IBucket = Bucket.importBucket(this, 'JohnnyBucket', { + weak: true // weak-reference +}); +``` + +> The reason we call this `importBucket` (and `exportBucket`) is because `import` is a reserved word in Java ([#89](https://github.com/awslabs/aws-cdk/issues/89)). Also, in JavaScript both static and instance methods are inherited, so if someone extends `Bucket` (say `ExtraBucket`), we should have a way to distinguish between `ExtraBucket.importBucket` and `ExtraBucket.importExtraBucket`. + +The implementation of these two methods is ~trivial: + +```ts +class Bucket { + public static importBucket(scope: Construct, exportName: string, options?: ImportOptions): IBucket { + return Bucket.deserializeBucket(Stack.find(scope).importObject(exportName, options)); + } + + public exportBucket(exportName: string): void { + Stack.find(this).exportObject(exportName, this); + } +} +``` + +### Reference by Physical Name (REQ3) + +AWS resources should allow users to reference them by specifying a physical name attribute such as ARN or name: + +```ts +const myBucket = Bucket.fromBucketArn(this, 'arn:aws:s3:::my_bucket'); +const yourBucket = Bucket.fromBucketName(this, 'your_bucket'); +``` + +Since these methods need to create a new construct, they should utilize the resource's physical name as the construct ID, and also ensure idempotency. Here's an example: + +```ts +public static fromBucketArn(scope: Construct, bucketArn: string): IBucket { + const stack = Stack.find(scope); + const id = `fromBucketArn:${bucketArn}`; + const existing = stack.findChild(id); + if (existing) { + return existing; + } + + return new ImportedBucket(stack, id, { bucketArn }); +} +``` + +### Lookup from Environment (REQ7) + +In composite cases, the resource's physical identity is not sufficient. For example, a `VpcNetwork` resource encapsulates many resources behind it such as subnets, NAT Gateways, etc. In those cases we still want to provide a great experience for developers who wish to reference a VPC: + +```ts +const vpc = VpcNetwork.lookupVpc(this, { + tags: { + department: 'sales', + stage: 'prod' + } +}); +``` + +The underlying implementation here is different, it uses an environmental context provider to lookup the VPC and extract all the relevant information from it, such as subnets, NAT Gateways and route tables. + +Here too, we expect idempotent behavior, which can be implemented in a similar manner. + +## Other Applications + +The construct serialization mechanism opens an opportunity for other applications that may benefit from being able to reference CDK constructs from outside the app. This section describes a few examples. + +### Serialization to JSON + +It should be trivial to implement a JSON serialization context: + +```ts +const ctx = new JsonSerializtionContext(); +alb.serialize(ctx); + +assertEquals(resolve(ctx.json), { + "loadBalancerArn": { "Fn::GetAtt": [ "MyALB1288xxx", "Arn" ] }, + "securityGroup": { + "securityGroupId": { "Ref": "MyALBSecurityGroup4444" } + } +}); +``` + +Representing a construct's runtime attributes as JSON (or a stringified JSON if needed via `CloudFormationJSON`) opens up a few interesting applications as described below. + +### Cross-Account/Region References + +The serialization mechanism, together with a bunch of custom resources can be used to reference constructs across accounts and region via, e.g. an S3 bucket. + +The producing stack can write a file to an S3 bucket with e.g. the JSON serialized representation of the construct and the consuming stack can read this file during deployment and deserialize the construct. + +### Environment Variables Serialization + +When runtime code needs to interact with resources defined in a CDK app, it needs to be able to reference these resources. + +The current practice is to manually wire specific resource attributes via environment variables so they will be available for runtime code. This may be sufficient for simple use cases such as simple AWS resources where a single attribute might be sufficient to represent the construct, but more complex scenarios (such as composite constructs) may benefit from the ability to serialize the entire construct through environment variables either through individual keys or as a single key + JSON value. + +## Implementation Notes + +The underlying pattern we use today for supporting imports/exports (`IBucket`, `BucketBase`, `Bucket` and `ImportedBucket`) continues to be **recommended** for implementing serialization and the `fromXxx` methods. + +The various static import methods (`deserializeXxx` `importXxx`, `fromXxx`) can all return an object that implements `IXxx`. The concrete type of this object can be implemented as an module-internal class `ImportedXxx` that includes the heuristics of how to represent an external resource of this type. For example, is may include the logic that determines how to convert an ARN to a name and vice versa, construct URLs, etc. + +## Open issues/questions + +- [ ] Can we provide a nicer API for implementing idempotency? Seems like this is a repeating pattern. We can definitely implement something very nice that's not jsii-compatible, but that might be fine as long as non-jsii users can still use the same mechanism. +- [ ] Consider renaming the `ImportXxx` classes to something that's not coupled with export/import. Maybe `ExternalXxx` or `ExistingXxx`. Those are internal classes, so it doesn't really matter, but still. diff --git a/packages/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.ts b/packages/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.ts index 8a4ab68ed574f..ef0afa42a2249 100644 --- a/packages/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.ts +++ b/packages/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.ts @@ -85,24 +85,10 @@ export class CfnReference extends Token { throw new Error('Can only reference cross stacks in the same region and account.'); } - // Ensure a singleton "Exports" scoping Construct - // This mostly exists to trigger LogicalID munging, which would be - // disabled if we parented constructs directly under Stack. - // Also it nicely prevents likely construct name clashes - - const exportsName = 'Exports'; - let stackExports = producingStack.node.tryFindChild(exportsName) as Construct; - if (stackExports === undefined) { - stackExports = new Construct(producingStack, exportsName); - } - // Ensure a singleton Output for this value const resolved = producingStack.node.resolve(tokenValue); - const id = 'Output' + JSON.stringify(resolved); - let output = stackExports.node.tryFindChild(id) as Output; - if (!output) { - output = new Output(stackExports, id, { value: tokenValue }); - } + const id = JSON.stringify(resolved); + const output = producingStack.exportString(`AutoExport-${id}`, tokenValue.toString()); // We want to return an actual FnImportValue Token here, but Fn.importValue() returns a 'string', // so construct one in-place. @@ -112,5 +98,4 @@ export class CfnReference extends Token { } import { Construct } from "../core/construct"; -import { Output } from "./output"; import { Stack } from "./stack"; diff --git a/packages/@aws-cdk/cdk/lib/cloudformation/stack.ts b/packages/@aws-cdk/cdk/lib/cloudformation/stack.ts index fd8b4b272a59c..3554a968f2cdc 100644 --- a/packages/@aws-cdk/cdk/lib/cloudformation/stack.ts +++ b/packages/@aws-cdk/cdk/lib/cloudformation/stack.ts @@ -2,7 +2,12 @@ import cxapi = require('@aws-cdk/cx-api'); import { App } from '../app'; import { Construct, IConstruct } from '../core/construct'; import { Environment } from '../environment'; +import { CloudFormationImportContextProvider } from '../serialization/import-context-provider'; +import { ExportSerializationContext, ImportDeserializationContext } from '../serialization/import-export'; +import { IDeserializationContext, ISerializable, SerializationOptions } from '../serialization/serialization'; +import { ArnComponents, arnFromComponents, parseArn } from './arn'; import { CfnReference } from './cfn-tokens'; +import { Fn } from './fn'; import { HashedAddressingScheme, IAddressingScheme, LogicalIDs } from './logical-id'; export interface StackProps { @@ -237,6 +242,109 @@ export class Stack extends Construct { return Array.from(this.stackDependencies.values()); } + /** + * Returns a deserialization context for importing an object which was exported under the + * specified export name. + * + * @param exportName The export name specified when the object was exported. + * @param options Import options + */ + public importObject(exportName: string, options?: ImportOptions): IDeserializationContext { + return new ImportDeserializationContext(this, exportName, options); + } + + /** + * Produces CloudFormation outputs for a serializable object under the specified + * export name. + * + * @param exportName The export name prefix for all the outputs. + * @param obj The object to serialize + */ + public exportObject(exportName: string, obj: ISerializable): void { + const ctx = new ExportSerializationContext(this, exportName); + obj.serialize(ctx); + } + + /** + * Imports a string from another stack in the same account/region which was + * exported under the specified export name. + * @param exportName The export name under which the string was exported + * @param options Import options + */ + public importString(exportName: string, options: ImportOptions = { }): string { + const stack = this; + const resolve = options.resolve === undefined ? ResolveType.Synthesis : options.resolve; + const weak = options.weak === undefined ? false : options.weak; + + if (resolve === ResolveType.Deployment && weak) { + throw new Error(`Deployment-time import resolution cannot be "weak"`); + } + + switch (resolve) { + case ResolveType.Deployment: + return Fn.importValue(exportName); + case ResolveType.Synthesis: + const value = new CloudFormationImportContextProvider(stack, { exportName }).parameterValue(); + if (!weak) { + stack.addStrongReference(exportName); + } + + return value; + } + } + + /** + * Exports a string value under an export name. + * @param exportName The export name under which to export the string. Export + * names must be unique within the account/region. + * @param value The value to export. + * @param options Export options, such as description. + */ + public exportString(exportName: string, value: string, options: ExportOptions = { }): Output { + let output = this.node.tryFindChild(exportName) as Output; + if (!output) { + output = new Output(this.exportsScope, exportName, { + description: options.description, + export: exportName, + value + }); + } else { + if (output.value !== value) { + // tslint:disable-next-line:max-line-length + throw new Error(`Trying to export ${exportName}=${value} but there is already an export with a similar name and a different value (${output.value})`); + } + } + return output; + } + + /** + * Adds a strong reference from this stack to a specific export name. + * + * Technically, if the stack has strong references, a WaitCondition resource + * will be synthesized, and a metadata entry with Fn::ImportValue will be + * added for each export name. + * + * @param exportName The name of the CloudFormation export to reference + */ + public addStrongReference(exportName: string) { + const id = 'StrongReferences8A180F'; + let strongRef = this.node.tryFindChild(id) as Resource; + if (!strongRef) { + strongRef = new Resource(this, id, { type: 'AWS::CloudFormation::WaitCondition' }); + } + + strongRef.options.metadata = strongRef.options.metadata || { }; + strongRef.options.metadata[exportName] = Fn.importValue(exportName); + } + + private get exportsScope() { + const exists = this.node.tryFindChild('Exports') as Construct; + if (exists) { + return exists; + } + return new Construct(this, 'Exports'); + } + /** * The account in which this stack is defined * @@ -388,7 +496,7 @@ export class Stack extends Construct { protected prepare() { // References for (const ref of this.node.findReferences()) { - if (CfnReference.isCfnReference(ref)) { + if (CfnReference.isInstance(ref)) { ref.consumeFromStack(this); } } @@ -505,11 +613,18 @@ function stackElements(node: IConstruct, into: StackElement[] = []): StackElemen return into; } -// These imports have to be at the end to prevent circular imports -import { ArnComponents, arnFromComponents, parseArn } from './arn'; -import { Aws } from './pseudo'; -import { Resource } from './resource'; -import { StackElement } from './stack-element'; +export interface ExportOptions { + description?: string; +} + +export interface ImportOptions { + resolve?: ResolveType; + weak?: boolean; +} + +export enum ResolveType { + Synthesis, + Deployment /** * Find all resources in a set of constructs @@ -521,3 +636,9 @@ function findResources(roots: Iterable): Resource[] { } return ret; } + +// These imports have to be at the end to prevent circular imports +import { ArnComponents, arnFromComponents, parseArn } from './arn'; +import { Aws } from './pseudo'; +import { Resource } from './resource'; +import { StackElement } from './stack-element'; diff --git a/packages/@aws-cdk/cdk/lib/index.ts b/packages/@aws-cdk/cdk/lib/index.ts index 49ffef7cdb7bf..cbefac4277e17 100644 --- a/packages/@aws-cdk/cdk/lib/index.ts +++ b/packages/@aws-cdk/cdk/lib/index.ts @@ -4,6 +4,7 @@ export * from './aspects/tag-aspect'; export * from './core/construct'; export * from './core/tokens'; export * from './core/tag-manager'; +export * from './serialization/serialization'; export * from './core/dependency'; export * from './cloudformation/cloudformation-json'; diff --git a/packages/@aws-cdk/cdk/lib/serialization/import-context-provider.ts b/packages/@aws-cdk/cdk/lib/serialization/import-context-provider.ts new file mode 100644 index 0000000000000..f281cd8fa2ad9 --- /dev/null +++ b/packages/@aws-cdk/cdk/lib/serialization/import-context-provider.ts @@ -0,0 +1,29 @@ +import cxapi = require('@aws-cdk/cx-api'); +import { ContextProvider } from '../context'; +import { Construct } from '../core/construct'; + +export interface CloudFormationImportContextProviderProps { + /** + * The name of the export to resolve + */ + exportName: string; +} +/** + * Context provider that will read values from the SSM parameter store in the indicated account and region + */ +export class CloudFormationImportContextProvider { + private readonly provider: ContextProvider; + private readonly exportName: string; + + constructor(scope: Construct, props: CloudFormationImportContextProviderProps) { + this.provider = new ContextProvider(scope, cxapi.CLOUDFORMATION_IMPORT_PROVIDER, props); + this.exportName = props.exportName; + } + + /** + * Return the SSM parameter string with the indicated key + */ + public parameterValue(defaultValue = `dummy-imported-value-for-${this.exportName}`): any { + return this.provider.getStringValue(defaultValue); + } +} diff --git a/packages/@aws-cdk/cdk/lib/serialization/import-export.ts b/packages/@aws-cdk/cdk/lib/serialization/import-export.ts new file mode 100644 index 0000000000000..27564ce831fab --- /dev/null +++ b/packages/@aws-cdk/cdk/lib/serialization/import-export.ts @@ -0,0 +1,102 @@ +import { Fn } from '../cloudformation/fn'; +import { ImportOptions, ResolveType, Stack } from '../cloudformation/stack'; +import { Construct } from '../core/construct'; +import { unresolved } from '../core/tokens'; +import { DeserializationOptions, IDeserializationContext, ISerializable, ISerializationContext, SerializationOptions } from './serialization'; + +const LIST_SEP = '||'; + +export class ExportSerializationContext implements ISerializationContext { + constructor( + private readonly stack: Stack, + private readonly exportName: string, + private readonly options: SerializationOptions = { }) { + } + + public writeString(key: string, value: string, options: SerializationOptions = { }): void { + let description = this.options.description; + if (options.description) { + if (!description) { + description = options.description; + } else { + description += ' - ' + options.description; + } + } + this.stack.exportString(this.exportNameForKey(key), value, { description }); + } + + public writeStringList(key: string, list: string[], options?: SerializationOptions): void { + // we use Fn.join instead of Array.join in case "list" is a token. + const value = Fn.join(LIST_SEP, list); + this.writeString(key, value, options); + } + + public writeObject(key: string, obj: ISerializable, options?: SerializationOptions): void { + const ctx = new ExportSerializationContext(this.stack, this.exportNameForKey(key), options); + obj.serialize(ctx); + } + + private exportNameForKey(key: string) { + return `${this.exportName}-${key}`; + } +} + +export class ImportDeserializationContext implements IDeserializationContext { + private resolve: ResolveType; + private weak: boolean; + + constructor( + private readonly stack: Stack, + private readonly exportName: string, + private readonly importOptions: ImportOptions = { }) { + + this.resolve = importOptions.resolve === undefined ? ResolveType.Synthesis : importOptions.resolve; + this.weak = importOptions.weak === undefined ? false : importOptions.weak; + + if (this.resolve === ResolveType.Deployment && this.weak) { + throw new Error(`Deployment-time import resolution cannot be "weak"`); + } + } + + public get scope() { + const exists = this.stack.node.tryFindChild('Imports') as Construct; + if (exists) { + return exists; + } + + return new Construct(this.stack, 'Imports'); + } + + public get id() { + return this.exportName; + } + + public readString(key: string, options: DeserializationOptions = { }): string { + const allowUnresolved = options.allowUnresolved === undefined ? true : false; + const exportName = this.exportNameForKey(key); + const value = this.stack.importString(exportName, this.importOptions); + + if (!allowUnresolved && unresolved(value)) { + throw new Error(`Imported value for export "${exportName}" is an unresolved token and "allowUnresolved" is false`); + } + + return value; + } + + public readStringList(key: string, options: DeserializationOptions = { }): string[] { + if (this.resolve === ResolveType.Deployment) { + throw new Error(`Cannot deserialize a string list for export "${this.exportName}-${key}" using deploy-time resolution`); + } + + // we are overriding "allowUnresolved" to "false" because we can't split an unresolved list. + return this.readString(key, { ...options, allowUnresolved: false }).split(LIST_SEP); + } + + public readObject(key: string): IDeserializationContext { + return new ImportDeserializationContext(this.stack, this.exportNameForKey(key), this.importOptions); + } + + private exportNameForKey(key: string) { + return `${this.exportName}-${key}`; + } +} \ No newline at end of file diff --git a/packages/@aws-cdk/cdk/lib/serialization/serialization.ts b/packages/@aws-cdk/cdk/lib/serialization/serialization.ts new file mode 100644 index 0000000000000..d295adb05546a --- /dev/null +++ b/packages/@aws-cdk/cdk/lib/serialization/serialization.ts @@ -0,0 +1,100 @@ +import { Construct } from '../core/construct'; + +/** + * Objects that implement this interface can serialize themselves through a + * key/value store. + */ +export interface ISerializable { + /** + * Called by as specific serializer (e.g. `stack.exportObject`) in order to + * serialize an object. + * @param ctx The serialization context for this object. + */ + serialize(ctx: ISerializationContext): void; +} + +/** + * The context for serialization. Serializable objects will use this to serialize themselves. + */ +export interface ISerializationContext { + /** + * Serialize a string value under the specified key. + * @param key The key under which to store the value + * @param value The string value to store + * @param options Serialization options. + */ + writeString(key: string, value: string, options?: SerializationOptions): void; + + /** + * Serialize a string array under the specified key. + * @param key + * @param value The array to serialize. + * @param options Serialization options. + */ + writeStringList(key: string, value: string[], options?: SerializationOptions): void; + + /** + * Serializes a serializable sub-object under the specified key. + * @param key The key under which to store the object. + * @param obj The object to store. + * @param options Serialization options. + */ + writeObject(key: string, obj: ISerializable, options?: SerializationOptions): void; +} + +/** + * Passed to `deserializeXxx` static methods in order to deserizlize an object. + * This context is bound to a specific object.s + */ +export interface IDeserializationContext { + scope: Construct; + id: string; + + /** + * Deserializes a string value which was serizlized under the specified key. + * @param key The key + * @param options Deserialization options + */ + readString(key: string, options?: DeserializationOptions): string; + + /** + * Deserializes a string array which was serizlied under the specified key. + * @param key The key + * @param options Deserialization options + */ + readStringList(key: string, options?: DeserializationOptions): string[]; + + /** + * Returns a deserialization context for a sub-object which was serizlized + * under the specified key. + * + * You will normally pass this to a `SubObject.deserializeSubObject(dctx)` + * + * @param key The key + */ + readObject(key: string): IDeserializationContext; +} + +/** + * Options for serialization. + */ +export interface SerializationOptions { + /** + * Description of this serialization node. + */ + description?: string; +} + +/** + * Options for deserialization. + */ +export interface DeserializationOptions { + /** + * Allows this specific value to include unresolved tokens, such as + * deploy-time imports. For `readString` this is `true` by default, but can be + * set to `false` by users to force synth-time resolution. For + * `readStringList` this cannot be `true`. String lists must always be + * resolved during synthesis. + */ + allowUnresolved?: boolean; +} \ No newline at end of file diff --git a/packages/@aws-cdk/cdk/test/test.serialization.ts b/packages/@aws-cdk/cdk/test/test.serialization.ts new file mode 100644 index 0000000000000..d7e1f90bde5c6 --- /dev/null +++ b/packages/@aws-cdk/cdk/test/test.serialization.ts @@ -0,0 +1,266 @@ +import { Test } from 'nodeunit'; +import cdk = require('../lib'); +import { ISerializable, ISerializationContext, ResolveType } from '../lib'; + +export = { + 'exportString'(test: Test) { + // GIVEN + const stack = new cdk.Stack(); + + // WHEN + stack.exportString('MyExportName', 'MyExportValue'); + + // THEN + test.deepEqual(stack.toCloudFormation(), { + Outputs: { + ExportsMyExportName4397ED14: { Value: 'MyExportValue', Export: { Name: 'MyExportName' } } + } + }); + test.done(); + }, + + 'exportString: description'(test: Test) { + // GIVEN + const stack = new cdk.Stack(); + + // WHEN + stack.exportString('MyExportName', 'MyExportValue', { + description: 'hello hello' + }); + + // THEN + test.deepEqual(stack.toCloudFormation(), { + Outputs: { + ExportsMyExportName4397ED14: { + Description: 'hello hello', + Value: 'MyExportValue', + Export: { Name: 'MyExportName' } + } + } + }); + test.done(); + }, + + 'importString: resolve=synth, weak=false (default)'(test: Test) { + // GIVEN + const stack = new TestStack(); + + // WHEN + const x = stack.importString('ExportMyExport'); + new cdk.Resource(stack, 'MyResource', { + type: 'AWS::Resource::Type', + properties: { + Input: x + } + }); + + // THEN + test.deepEqual(x, 'dummy-imported-value-for-ExportMyExport'); + test.deepEqual(stack.toCloudFormation(), { + Resources: { + StrongReferences8A180F: { + Type: 'AWS::CloudFormation::WaitCondition', + Metadata: { ExportMyExport: { 'Fn::ImportValue': 'ExportMyExport' } } + }, + MyResource: { + Type: 'AWS::Resource::Type', + Properties: { + Input: 'dummy-imported-value-for-ExportMyExport' + } + } + } + }); + test.deepEqual(stack.missingContext, { + 'cloudformation-import:account=11111:exportName=ExportMyExport:region=us-east-1': { + provider: 'cloudformation-import', + props: { + account: stack.env.account, + region: stack.env.region, + exportName: 'ExportMyExport' + } + } + }); + test.done(); + }, + + 'importString: resolve=synth, weak=true'(test: Test) { + // GIVEN + const stack = new TestStack(); + + // WHEN + const x = stack.importString('ExportMyExport', { weak: true }); + new cdk.Resource(stack, 'MyResource', { + type: 'AWS::Resource::Type', + properties: { + Input: x + } + }); + + // THEN + test.deepEqual(stack.toCloudFormation(), { + Resources: { + MyResource: { + Type: 'AWS::Resource::Type', + Properties: { + Input: 'dummy-imported-value-for-ExportMyExport' + } + } + } + }); + test.deepEqual(stack.missingContext, { + 'cloudformation-import:account=11111:exportName=ExportMyExport:region=us-east-1': { + provider: 'cloudformation-import', + props: { + account: stack.env.account, + region: stack.env.region, + exportName: 'ExportMyExport' + } + } + }); + test.done(); + }, + + 'importString: resolve=deploy, weak=false'(test: Test) { + // GIVEN + const stack = new TestStack(); + + // WHEN + const x = stack.importString('ExportMyExport', { resolve: cdk.ResolveType.Deployment }); + new cdk.Resource(stack, 'MyResource', { + type: 'AWS::Resource::Type', + properties: { + Input: x + } + }); + + // THEN + test.deepEqual(stack.toCloudFormation(), { + Resources: { + MyResource: { + Type: 'AWS::Resource::Type', + Properties: { + Input: { 'Fn::ImportValue': 'ExportMyExport' } + } + } + } + }); + test.deepEqual(stack.missingContext, {}); + test.done(); + }, + + 'importString: resolve=deploy, weak=true (error)'(test: Test) { + // GIVEN + const stack = new TestStack(); + + // WHEN/THEN + test.throws(() => + stack.importString('ExportMyExport', { resolve: cdk.ResolveType.Deployment, weak: true }), + /Deployment-time import resolution cannot be "weak"/); + + test.done(); + }, + + 'importObject (synth + strong): readString, readStringArray, readObject'(test: Test) { + // GIVEN + const stack = new TestStack(); + + // WHEN + const ctx = stack.importObject('ObjectExportName'); + const value1 = ctx.readString('key1'); + const value2 = ctx.readString('key2'); + const subobj = ctx.readObject('key3'); + const value4 = subobj.readString('key4'); + const value5 = subobj.readStringList('key5'); + + // THEN + test.deepEqual(value1, 'dummy-imported-value-for-ObjectExportName-key1'); + test.deepEqual(value2, 'dummy-imported-value-for-ObjectExportName-key2'); + test.deepEqual(value4, 'dummy-imported-value-for-ObjectExportName-key3-key4'); + test.deepEqual(value5, [ 'dummy-imported-value-for-ObjectExportName-key3-key5' ]); + + // since the default is a strong reference, we expect a wait condition with + // import values for all the keys. + test.deepEqual(stack.toCloudFormation(), { + Resources: { + StrongReferences8A180F: { + Type: 'AWS::CloudFormation::WaitCondition', + Metadata: { + 'ObjectExportName-key1': { 'Fn::ImportValue': 'ObjectExportName-key1' }, + 'ObjectExportName-key2': { 'Fn::ImportValue': 'ObjectExportName-key2' }, + 'ObjectExportName-key3-key4': { 'Fn::ImportValue': 'ObjectExportName-key3-key4' }, + 'ObjectExportName-key3-key5': { 'Fn::ImportValue': 'ObjectExportName-key3-key5' } + } + } + } + }); + + test.done(); + }, + + 'exportObject: writeString, writeStringList, writeObject'(test: Test) { + // GIVEN + const stack = new TestStack(); + + const myObj: ISerializable = { + serialize: (ctx: ISerializationContext) => { + ctx.writeString('k1', 'v1'); + ctx.writeStringList('k2', [ 'v2', 'v3' ]); + ctx.writeObject('k3', { + serialize: ctx2 => { + ctx2.writeString('k4', 'v4', { description: 'desc of k4' }); + } + }); + } + }; + + // WHEN + stack.exportObject('BoomBoom', myObj); + + // THEN + test.deepEqual(stack.toCloudFormation(), { + Outputs: { + ExportsBoomBoomk1B31E94EE: { Value: 'v1', Export: { Name: 'BoomBoom-k1' } }, + ExportsBoomBoomk277969FC7: { Value: 'v2||v3', Export: { Name: 'BoomBoom-k2' } }, + ExportsBoomBoomk3k48482CDA1: { + Description: 'desc of k4', + Value: 'v4', + Export: { Name: 'BoomBoom-k3-k4' } + } + } + }); + test.done(); + }, + + 'importObject: deploy-time resolution with allowUnresolved true/false'(test: Test) { + // GIVEN + const stack = new TestStack(); + const obj = stack.importObject('MyExportName', { resolve: ResolveType.Deployment }); + + // WHEN/THEN + + // readString succeeds because "allowUnresolved" is defaulted to "true" here + obj.readString('alright'); + + // readString with "allowUnresolved=false" will fail + test.throws(() => obj.readString('boom', { allowUnresolved: false }), + /Imported value for export "MyExportName-boom" is an unresolved token and "allowUnresolved" is false/); + + // readStringList fails because "allowUnresolved" is always false + test.throws(() => obj.readStringList('error'), + /Cannot deserialize a string list for export "MyExportName-error" using deploy-time resolution/); + + // can't force it to true, sorry! + test.throws(() => obj.readStringList('error', { allowUnresolved: true }), + /Cannot deserialize a string list for export "MyExportName-error" using deploy-time resolution/); + + test.done(); + } +}; + +class TestStack extends cdk.Stack { + constructor() { + super(undefined, 'test-stack', { env: { + account: '11111', region: 'us-east-1' + }}); + } +} \ No newline at end of file diff --git a/packages/@aws-cdk/cx-api/lib/context/cloudformation-import.ts b/packages/@aws-cdk/cx-api/lib/context/cloudformation-import.ts new file mode 100644 index 0000000000000..fc856ccc99c5b --- /dev/null +++ b/packages/@aws-cdk/cx-api/lib/context/cloudformation-import.ts @@ -0,0 +1,23 @@ +export const CLOUDFORMATION_IMPORT_PROVIDER = 'cloudformation-import'; + +/** + * Query to hosted zone context provider + */ +export interface CloudFormationImportContextQuery { + /** + * Query account + */ + account?: string; + + /** + * Query region + */ + region?: string; + + /** + * Name of export to look up + */ + exportName?: string; +} + +// Response is a string diff --git a/packages/@aws-cdk/cx-api/lib/index.ts b/packages/@aws-cdk/cx-api/lib/index.ts index 8be9dc02a8bd7..389e6303f254a 100644 --- a/packages/@aws-cdk/cx-api/lib/index.ts +++ b/packages/@aws-cdk/cx-api/lib/index.ts @@ -4,4 +4,5 @@ export * from './context/hosted-zone'; export * from './context/vpc'; export * from './context/ssm-parameter'; export * from './context/availability-zones'; +export * from './context/cloudformation-import'; export * from './metadata/assets';