RFC 249: V2 Experiments

[ericzbeard]

• Tracking issue: 249
• Rendered version

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache-2.0 license

[skinny85]

Some initial comments.

In general, I feel like I'm missing some details. What are the names of the experimental modules? What are their versions? Are they released together, or does each and every one have a separate version? How does this affect our usage statistics gathering?

In my opinion, releasing them as separate modules is a mistake. This will bring back the same dependency problems that mono-CDK was designed to solve, except just for the experimental modules. What this will effectively mean is that 1) experimental modules cannot depend on other experimental modules, and 2) third-party modules cannot depend on experimental modules. This will make experimental modules much less useful than today, which will affect their usage (I suspect it will be considerably lower than today), which will make graduating them more difficult (less customer feedback and engagement). I consider that a serious risk.

I also don't see why do we need a separate repository for this. Why can't we release from the one we have today, just separately for the experimental modules? This hasn't been addressed in the doc at all.

[NetaNir]

Some initial comments.

In general, I feel like I'm missing some details. What are the names of the experimental modules? What are their versions? Are they released together, or does each and every one have a separate version? How does this affect our usage statistics gathering?

In my opinion, releasing them as separate modules is a mistake. This will bring back the same dependency problems that mono-CDK was designed to solve, except just for the experimental modules. What this will effectively mean is that 1) experimental modules cannot depend on other experimental modules, and 2) third-party modules cannot depend on experimental modules. This will make experimental modules much less useful than today, which will affect their usage (I suspect it will be considerably lower than today), which will make graduating them more difficult (less customer feedback and engagement). I consider that a serious risk.

I also don't see why do we need a separate repository for this. Why can't we release from the one we have today, just separately for the experimental modules? This hasn't been addressed in the doc at all.

@skinny85 experimental modules will be just like any other third party construct, any limitation they will have will also effect the eco system, which is what worries me in your statement. Maybe we need to think about a solution to that (huge) problem.
(yeah I know this it's a bold, rabbit hole statement)

[skinny85]

Some initial comments.
In general, I feel like I'm missing some details. What are the names of the experimental modules? What are their versions? Are they released together, or does each and every one have a separate version? How does this affect our usage statistics gathering?
In my opinion, releasing them as separate modules is a mistake. This will bring back the same dependency problems that mono-CDK was designed to solve, except just for the experimental modules. What this will effectively mean is that 1) experimental modules cannot depend on other experimental modules, and 2) third-party modules cannot depend on experimental modules. This will make experimental modules much less useful than today, which will affect their usage (I suspect it will be considerably lower than today), which will make graduating them more difficult (less customer feedback and engagement). I consider that a serious risk.
I also don't see why do we need a separate repository for this. Why can't we release from the one we have today, just separately for the experimental modules? This hasn't been addressed in the doc at all.

@skinny85 experimental modules will be just like any other third party construct, any limitation they will have will also effect the eco system, which is what worries me in your statement. Maybe we need to think about a solution to that (huge) problem.
(yeah I know this it's a bold, rabbit hole statement)

I don't think that statement is true though. The Construct Library is special in the ecosystem - if it was the same as other third-party modules, we wouldn't be doing mono-CDK in the first place.

[skinny85]

I pulled some data from our dependencies. Here's all instances of stable modules depending on unstable ones:

⚠️  Stable package '@aws-cdk/assert' depends on unstable package '@aws-cdk/cloudformation-diff'
⚠️  Stable package '@aws-cdk/assert' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-apigateway' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-apigateway' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-applicationautoscaling' depends on unstable package '@aws-cdk/aws-autoscaling-common'
⚠️  Stable package '@aws-cdk/aws-autoscaling' depends on unstable package '@aws-cdk/aws-autoscaling-common'
⚠️  Stable package '@aws-cdk/aws-autoscaling' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-codebuild' depends on unstable package '@aws-cdk/aws-ecr-assets'
⚠️  Stable package '@aws-cdk/aws-codebuild' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-codebuild' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-codepipeline' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-ec2' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-ec2' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-ec2' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-ecs' depends on unstable package '@aws-cdk/aws-ecr-assets'
⚠️  Stable package '@aws-cdk/aws-ecs' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-ecs' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-elasticloadbalancingv2' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-elasticloadbalancingv2-actions' depends on unstable package '@aws-cdk/aws-cognito'
⚠️  Stable package '@aws-cdk/aws-events-targets' depends on unstable package '@aws-cdk/aws-batch'
⚠️  Stable package '@aws-cdk/aws-iam' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-lambda' depends on unstable package '@aws-cdk/aws-efs'
⚠️  Stable package '@aws-cdk/aws-lambda' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-lambda' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package '@aws-cdk/aws-logs' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/aws-route53-patterns' depends on unstable package '@aws-cdk/aws-cloudfront'
⚠️  Stable package '@aws-cdk/aws-route53-patterns' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-route53-targets' depends on unstable package '@aws-cdk/aws-cloudfront'
⚠️  Stable package '@aws-cdk/aws-route53-targets' depends on unstable package '@aws-cdk/aws-cognito'
⚠️  Stable package '@aws-cdk/aws-route53-targets' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package '@aws-cdk/aws-stepfunctions-tasks' depends on unstable package '@aws-cdk/aws-batch'
⚠️  Stable package '@aws-cdk/aws-stepfunctions-tasks' depends on unstable package '@aws-cdk/aws-ecr-assets'
⚠️  Stable package '@aws-cdk/aws-stepfunctions-tasks' depends on unstable package '@aws-cdk/aws-glue'
⚠️  Stable package '@aws-cdk/aws-stepfunctions-tasks' depends on unstable package '@aws-cdk/aws-s3-assets'
⚠️  Stable package '@aws-cdk/core' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package 'aws-cdk' depends on unstable package '@aws-cdk/cloudformation-diff'
⚠️  Stable package 'aws-cdk' depends on unstable package '@aws-cdk/cx-api'
⚠️  Stable package 'aws-cdk' depends on unstable package '@aws-cdk/region-info'
⚠️  Stable package 'aws-cdk' depends on unstable package 'cdk-assets'

(this is after I've marked @aws-cdk/assert as "stable", as we have hundreds more of those)

None of those dependencies would be possible after this change.

[skinny85]

Another serious dependency issue is that we have a lot of private modules used in the repository for common things like build, test lint, etc., and if you want to split experimental packages into a separate repository, all those private modules will now have to be published to NPM (or copy-pasted, I guess, but that doesn't sound like a great idea).

[skinny85]

And here is the list of unstable modules depending on each other - relevant to the discussion whether there should be a single experimental package released, or multiple ones:

ℹ️️  Unstable package '@aws-cdk/aws-appsync' depends on unstable package '@aws-cdk/aws-cognito'
ℹ️️  Unstable package '@aws-cdk/aws-appsync' depends on unstable package '@aws-cdk/aws-s3-assets'
ℹ️️  Unstable package '@aws-cdk/aws-backup' depends on unstable package '@aws-cdk/aws-efs'
ℹ️️  Unstable package '@aws-cdk/aws-backup' depends on unstable package '@aws-cdk/aws-rds'
ℹ️️  Unstable package '@aws-cdk/aws-cloudfront-origins' depends on unstable package '@aws-cdk/aws-cloudfront'
ℹ️️  Unstable package '@aws-cdk/aws-docdb' depends on unstable package '@aws-cdk/aws-efs'
ℹ️️  Unstable package '@aws-cdk/aws-ecr-assets' depends on unstable package '@aws-cdk/cx-api'
ℹ️️  Unstable package '@aws-cdk/aws-efs' depends on unstable package '@aws-cdk/cx-api'
ℹ️️  Unstable package '@aws-cdk/aws-s3-assets' depends on unstable package '@aws-cdk/cx-api'
ℹ️️  Unstable package '@aws-cdk/aws-s3-deployment' depends on unstable package '@aws-cdk/aws-cloudfront'
ℹ️️  Unstable package '@aws-cdk/aws-s3-deployment' depends on unstable package '@aws-cdk/aws-s3-assets'
ℹ️️  Unstable package '@aws-cdk/aws-ses-actions' depends on unstable package '@aws-cdk/aws-ses'
ℹ️️  Unstable package '@aws-cdk/aws-synthetics' depends on unstable package '@aws-cdk/aws-s3-assets'
ℹ️️  Unstable package '@aws-cdk/pipelines' depends on unstable package '@aws-cdk/aws-s3-assets'
ℹ️️  Unstable package '@aws-cdk/pipelines' depends on unstable package '@aws-cdk/cx-api'
️ℹ️  Unstable package '@aws-cdk/pipelines' depends on unstable package '@aws-cdk/aws-ecr-assets'
ℹ️️  Unstable package 'cdk-assets' depends on unstable package '@aws-cdk/cx-api'

[NetaNir]

Some initial comments.
In general, I feel like I'm missing some details. What are the names of the experimental modules? What are their versions? Are they released together, or does each and every one have a separate version? How does this affect our usage statistics gathering?
In my opinion, releasing them as separate modules is a mistake. This will bring back the same dependency problems that mono-CDK was designed to solve, except just for the experimental modules. What this will effectively mean is that 1) experimental modules cannot depend on other experimental modules, and 2) third-party modules cannot depend on experimental modules. This will make experimental modules much less useful than today, which will affect their usage (I suspect it will be considerably lower than today), which will make graduating them more difficult (less customer feedback and engagement). I consider that a serious risk.
I also don't see why do we need a separate repository for this. Why can't we release from the one we have today, just separately for the experimental modules? This hasn't been addressed in the doc at all.

@skinny85 experimental modules will be just like any other third party construct, any limitation they will have will also effect the eco system, which is what worries me in your statement. Maybe we need to think about a solution to that (huge) problem.
(yeah I know this it's a bold, rabbit hole statement)

I don't think that statement is true though. The Construct Library is special in the ecosystem - if it was the same as other third-party modules, we wouldn't be doing mono-CDK in the first place.

Why is that true? What's special about the L2s? We are doing monocdk because of the inevitable dependencies between L2s which stems from the dependencies between AWS services (i.e you need S3 bucket for triggering a lambda etc.). Every L2 will depend on aws-cdk/aws-s3, aws-cdk/aws-iam, aws-cdk/aws-cloudwatch... for the most basic construct you will need to depend on at least 30 aws-cdk/* modules, this is why for these modules monocdk make sense. On the other hand you will have modules like aws-cdk/aws-synthetics that are more like third party constructs in that aspect - much less construct will depend on them. Are we saying that third party constructs can't depend on each others as well? What is the technical difference between aws-cdk/aws-synthetics and any other third party construct?

[skinny85]

So is your argument that @aws-cdk/aws-synthetics should not be in mono-CDK @NetaNir ?

[NetaNir]

So is your argument that @aws-cdk/aws-synthetics should not be in mono-CDK @NetaNir ?

I don't know if it should be outside of monocdk forever, overall it is nice that everything is in the same package. I was giving it as an example to an L2 which is similar to a third party construct in terms of dependencies.

And it's not my argument, I was pointing out the fact that experimental modules, if released separately, will be the same as third party packages, if we think it's impossible to maintain/use then we should solve it, as it will hurt the eco system.

[skinny85]

Is it even feasible to go with the multi-repository approach here at all? Let's say a customer wants to submit a PR adding a new L2 to a module that was previously CFN-only. They can't do that, because there is no experimental module for this AWS service, and thus no GitHub repository for it, so there is nowhere to submit the Pull Request...

[eladb]

Is it even feasible to go with the multi-repository approach here at all? Let's say a customer wants to submit a PR adding a new L2 to a module that was previously CFN-only. They can't do that, because there is no experimental module for this AWS service, and thus no GitHub repository for it, so there is nowhere to submit the Pull Request...

Yeah you are right. But how does this connect with the Uluru import solution? Eventually we are working towards a point where L1s are going to be generated on-demand by users same as in cdk8s import (cdk import AWS::S3::Bucket).

So we won't need these empty L1 modules anyway... we will just have a projen project type for AWS CDK modules that will generate a module repo and people can submit that as their PR.

[skinny85]

Re: the discussion on separate vs one module, and what should go into mono-CDK. I want to summarize the existing points.

We can do one of two things.

1. Decide that all our L2s are "special" in some way, and they should all go to one of two NPM packages: either the mono-CDK (in case of stable modules), or a single, separate module for all experimental modules (to avoid the same dependency issue that mono-CDK was designed to solve). The name and version of this umbrella packages for all experimental modules is TBD. This pretty much kills the "separate Git repository per module" idea (in fact, I think we should release both from one Git repository, and just divide them at build time). We can go through a process for modules that are still not stable right before we release 2.0, to assign them to one of the 2 buckets. When a module graduates from the "unstable" bucket to "stable", it gets added to mono-CDK.

2. Decide that only some L2s are "special", and only those should be in mono-CDK. I assume the set of all "special" L2s is pretty much the same as the set of current stable modules (we can go through all modules right before releasing 2.0, and I guess, if there is a module that we want to include but which is still "unstable", we can graduate it to stable immediately). In this scheme, a module graduating from "unstable" to "stable" just means a new version; no new modules will be added to mono-CDK (unless a module can go from being "not special" to "special" somehow...?), and mono-CDK will not contain the L1s for non-"special" modules. In this scheme, we can go with either one, or separate Git repositories for the unstable modules (although, in my opinion, multiple repositories don't make sense because of RFC 249: V2 Experiments #250 (comment)). I don't know what criteria we would use to call a module "special" - @NetaNir , seems like you had some ideas here, care to comment?

I'm personally leaning towards 1., but I don't care that much. What I really don't want is the current proposal in this RFC, which essentially buckets the modules as "special" on the criteria that "if it has an L2 contributed to it before the 2.0 release, then it's automatically "special" (even if it's a brand new L2 that was pushed a week before), and every module that doesn't have an L2 before that date is not special".

[ericzbeard]

Let's say a customer wants to submit a PR adding a new L2 to a module that was previously CFN-only.

I'm a little confused by the discussion around this point, so let's use a concrete example. If a member of the community wants to submit an L2 for, let's say Neptune, after we go to v2, what do they do? If we go with one repo per experimental module, they would need to ask us to first create the repo. If we go with a single repo for all experiments, then they could submit a PR with the new code. It's logistically more work for us to create a repo per module, but maybe it's a good thing to force a discussion about the new construct library before accepting a new PR out of nowhere. Do we want contributors to do the initial work in their own personal repositories first? What would the process be to convert an L2 that someone wrote into an "official" AWS experimental L2?

Adam makes good points for not creating any new repositories at all, and using the build and package process to make sure we never introduce any breaking changes to a 2.x minor release, but I worry that it will be difficult to enforce.

[skinny85]

If we go with one repo per experimental module, they would need to ask us to first create the repo. If we go with a single repo for all experiments, then they could submit a PR with the new code. It's logistically more work for us to create a repo per module

I don't think it's a matter of which is more work.... I don't think the "Git repository per package" is viable at all - where would the customer "ask us to first create the repo"...? I don't think we have a communication channel with customers that can be reliably used for that...

but maybe it's a good thing to force a discussion about the new construct library before accepting a new PR out of nowhere. Do we want contributors to do the initial work in their own personal repositories first? What would the process be to convert an L2 that someone wrote into an "official" AWS experimental L2?

I feel like that will actually be more work for us, not less. If they submit PRs with us starting from the beginning, we can make sure they're small, and they get feedback from us early in the process. If we have to move an entire, already existing L2 into our repository, that will be a huge PR (that might involve dramatic changes to the library, that we didn't have the opportunity to specify earlier). In fact, we already have few of those (SageMaker?) that we refused to do, exactly for that reason.

using the build and package process to make sure we never introduce any breaking changes to a 2.x minor release, but I worry that it will be difficult to enforce.

We do this already for stable modules in 1.x, so this is a non-issue.

Pull request has been modified.

[RomainMuller]

Another serious dependency issue is that we have a lot of private modules used in the repository for common things like build, test lint, etc., and if you want to split experimental packages into a separate repository, all those private modules will now have to be published to NPM (or copy-pasted, I guess, but that doesn't sound like a great idea).

Or git submodule'd

[skinny85]

Or git submodule'd

God, please no. Git submodules are terrible. I would avoid them like the plague.

[nija-at]

This paragraph can be made crisper by dropping the statements around "clever techniques to simply hide the experiments".

We're in the 'Motivation' section, so this feels off place. Perhaps this can be cleaned up and moved as the second paragraph under 'Summary'.

Pull request has been modified.

[skinny85]

Some comments from me. I see some naming inconsistencies that I think we need to sort out, and some details look left over from the previous versions.

General note: I would suggest using Markdown code backtics to enclose names used in code (so, aws-cdk-lib, not aws-cdk-lib). Makes it much easier to read that way, as the names stand out much more.

I think Elad's suggestion ("What if experimental packages are just in a sub-namespace of the main package?") is a great one, and worth exploring. It would allow us to keep the capability of stable modules to depend on experimental ones, which the current proposal bans.

Pull request has been modified.

[eladb]

• Missing final recommendation
• How do we enable the goal "The CDK team can still perform API experiments" for APIs in stable modules?

[eladb]

Maybe developer preview means it moves to the core module?

[eladb]

We also need to add another goal which is to enable a healthy 3rd-party library ecosystem that "peer depends" on aws-cdk-lib (this is one of the main reasons we eventually decided that we can't introduce breaking changes to aws-cdk-lib).

[eladb]

All modules are marked stable in aws-cdk-lib, no?

[eladb]

We already have these compatibility checks in our builds. We just need to disallow @experimental.

[eladb]

Move to appendix

[eladb]

This option conflicts with "No unstable code in the main mono-CDK modules". I don't see a reason to propose it.

[eladb]

I think this makes this a no starter

[eladb]

Suggested change

	- Graduating a module to stable will be a breaking change for customers. We can
	- Graduating a module to stable will be a breaking change for customers who consume the unstable module from the experimental library. We can

[eladb]

I think using "3.x" is a bit misleading because this is not necessarily staging the next major version but rather just the experiments

[ericzbeard]

What about [email protected]? I think that's the semver convention for pre-releases.

[eladb]

It's also a bit "wrong" because this is not the alpha for 3.x, it's a staging module for experimental APIs for 2.x, no?

[eladb]

Didn't we say there are no unstable APIs in aws-cdk-lib? what am I missing?

[nija-at]

I like this option. The constraints (disadvantages) feel reasonable.

We can mitigate this downside by keeping the old unstable package around, but that leads to duplicated classes.

Re: duplicate classes - we can completely delete the code off the code base and mark the released packages as deprecated, so no code duplication. Existing customers can continue to use them but won't get newer features.

[nija-at]

Not a fan of this option. Maintaining 2 long-lived branches is not easy and is going to cause us a lot of maintainance pain. Separate repos or folders are much easier to manage.

[nija-at]

Plus, the disadvantages section points the other problems quite well.

Pull request has been modified.