(jsii)(v2): Cannot apply Aspects to EKS cluster stack #16625

ivanmarquescepsa · 2021-09-23T11:16:51Z

I'm trying to apply Aspects to a EKS cluster stack, but it fails with a jsii.errors.JavaScriptError. The reason to apply aspects is to apply permission boundaries to every role generated.

Is there any way to work-around this situation? i'm interested in modifying the roles generated by EKS custom resources E.G: Kubectlhandler.

Reproduction Steps

#!/usr/bin/env python3
import os
import jsii
from aws_cdk import App, IAspect, Aspects, Tags, Stack, Environment, aws_eks as eks, aws_iam as iam, aws_ec2 as ec2
from constructs import Construct, IConstruct

@jsii.implements(IAspect)
class PermissionsBoundary:

    def __init__(self, permission_boundary_arn: str):
        self.permissions_boundary_arn = permission_boundary_arn
  

    def visit(self, node: IConstruct):
        print("Visited", node.node.path)

class EksCluster(Stack):

    def __init__(self, scope: Construct, id: str, vpc_id:str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc.from_lookup(self, "VPC", vpc_id=vpc_id)
        eks_cluster_role = iam.Role(self, id + "-role",
                                    assumed_by=iam.ServicePrincipal("eks.amazonaws.com"),
                                    managed_policies=[
                                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEKSServicePolicy"),
                                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEKSClusterPolicy")])

        eks_master_role = iam.Role(self, id + "-admin",
                                   assumed_by=iam.AccountRootPrincipal())

        cluster = eks.Cluster(self, id,
                              masters_role=eks_master_role,
                              version=eks.KubernetesVersion.V1_21,
                              role=eks_cluster_role,
                              vpc=vpc,
                              endpoint_access=eks.EndpointAccess.PRIVATE,
                              default_capacity=0,
                              output_cluster_name=True,
                              output_masters_role_arn=True
                              )

# CDK app
app = App()
account = os.getenv('ACCOUNT_ID')
env_EU = Environment(region="eu-west-1", account=account)

stack = EksCluster(app, "test-cdkv2-eks-cluster", vpc_id=os.getenv('VPC_ID'), env=env_EU )

Aspects.of(stack).add(PermissionsBoundary(f"arn:aws:iam::{os.getenv('ACCOUNT_ID')}:policy/platform-main-boundary"))

app.synth()

What did you expect to happen?

Print the node paths of every node.

What actually happened?

jsii.errors.JavaScriptError:
Error: Unknown type: aws-cdk-lib.custom_resources.Provider
at KernelHost.completeCallback (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/tmprxijvdhs/lib/program.js:9675:35)
at KernelHost.callbackHandler (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/tmprxijvdhs/lib/program.js:9666:41)
at Object.value (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/tmprxijvdhs/lib/program.js:8536:49)
at recurse (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:83:20)
at recurse (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:95:17)
at recurse (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:95:17)
at recurse (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:95:17)
at invokeAspects (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:69:5)
at Object.synthesize (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/private/synthesis.js:15:5)
at App.synth (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-wOF0bw/node_modules/aws-cdk-lib/lib/core/lib/stage.js:97:41)

Environment

CDK CLI Version : 1.123.0
Framework Version: v2.0.0-rc23
Node.js Version: v14.17.6
OS : macOS 11.2
Language (Version): Python (3.8.9)

Other

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

otaviomacedo · 2021-10-01T15:13:35Z

Hi, @keepler-ivanmarques. Thanks for catching this.

The fully qualified name of the class in Python is aws_cdk.custom_resources.Provider, whereas the type being referenced is aws-cdk-lib.custom_resources.Provider, according to the error message. There is probably something wrong in the rewrite that happens when the v2 build is generated.

Marking this as a V2 GA milestone bug.

rix0rrr · 2021-10-08T10:33:19Z

Error seems to be coming from here:

https://github.com/aws/jsii/blob/2ba2ec47af1b3007844c00bd44aa81ccc7729040/packages/%40jsii/python-runtime/src/jsii/_reference_map.py#L126

rix0rrr · 2021-10-08T10:44:01Z

Aha, found it. The issue here is lazy loading.

jsii doesn't know about the custom_resources.Provider class because you never wrote import aws_cdk.custom_resources

Adding this to the project makes it synth correctly:

from aws_cdk import custom_resources, lambda_layer_awscli, lambda_layer_kubectl

It's not specific to Aspects... any form of Node.of(app).findChild('Complex/Path/To/Construct') would lead to the same error.

I'm not sure what the best solution is. Obviously we need to load the type information whenever necessary from either side (from Python side or from JavaScript side), but I don't know how to best arrange that.

I'm also guessing this is not a v2-specific error. Pretty sure the exact same code would be broken for v1 as well.

rix0rrr · 2021-10-08T10:52:25Z

Huh.

For reasons that I don't quite fathom, it does not happen for v1:

setup.py

    install_requires=[
        "aws-cdk.core==1.126.0",
        "aws-cdk.aws-eks==1.126.0",
        "aws-cdk.aws-ec2==1.126.0",
        "aws-cdk.aws-iam==1.126.0",
    ],

app.py

#!/usr/bin/env python3
import os
import jsii
from aws_cdk.core import App, IAspect, Aspects, Tags, Stack, Environment
from aws_cdk import aws_eks as eks, aws_iam as iam, aws_ec2 as ec2
from constructs import Construct, IConstruct

# from aws_cdk import custom_resources, lambda_layer_awscli, lambda_layer_kubectl

@jsii.implements(IAspect)
class PermissionsBoundary:

    def __init__(self, permission_boundary_arn: str):
        self.permissions_boundary_arn = permission_boundary_arn


    def visit(self, node: IConstruct):
        print("Visited", node.node.path)

class EksCluster(Stack):

    def __init__(self, scope: Construct, id: str, vpc_id:str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc.from_lookup(self, "VPC", vpc_id=vpc_id)
        eks_cluster_role = iam.Role(self, id + "-role",
                                    assumed_by=iam.ServicePrincipal("eks.amazonaws.com"),
                                    managed_policies=[
                                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEKSServicePolicy"),
                                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEKSClusterPolicy")])

        eks_master_role = iam.Role(self, id + "-admin",
                                   assumed_by=iam.AccountRootPrincipal())

        cluster = eks.Cluster(self, id,
                              masters_role=eks_master_role,
                              version=eks.KubernetesVersion.V1_21,
                              role=eks_cluster_role,
                              vpc=vpc,
                              endpoint_access=eks.EndpointAccess.PRIVATE,
                              default_capacity=0,
                              output_cluster_name=True,
                              output_masters_role_arn=True
                              )

# CDK app
app = App()
env_EU = Environment(region="eu-west-1", account=os.getenv('CDK_DEFAULT_ACCOUNT'))

stack = EksCluster(app, "test-cdkv2-eks-cluster", vpc_id=os.getenv('VPC_ID'), env=env_EU )

Aspects.of(stack).add(PermissionsBoundary(f"arn:aws:iam::{os.getenv('ACCOUNT_ID')}:policy/platform-main-boundary"))

app.synth()

Synths fine...

rix0rrr · 2021-10-08T10:56:55Z

Pretty sure it works fine in v1 because .venv/lib/python3.9/site-packages/aws_cdk/aws_eks/_jsii/__init__.py contains:

import aws_cdk.aws_autoscaling._jsii
import aws_cdk.aws_ec2._jsii
import aws_cdk.aws_iam._jsii
import aws_cdk.aws_kms._jsii
import aws_cdk.aws_lambda._jsii
import aws_cdk.aws_lambda_nodejs._jsii
import aws_cdk.aws_ssm._jsii
import aws_cdk.core._jsii
import aws_cdk.custom_resources._jsii
import aws_cdk.lambda_layer_awscli._jsii
import aws_cdk.lambda_layer_kubectl._jsii
import aws_cdk.lambda_layer_node_proxy_agent._jsii
import constructs._jsii

And for v2 there is no such file!

rix0rrr · 2021-10-08T11:01:05Z

Generating the appropriate import statements between submodules would seem to be the solution... either that or just loading all modules (and dependencies!) at once in the top-level import (aws_cdk/__init__.py).

Callbacks may make references to types from submodules of loaded packages, however due to how the type registration works in the Python runtime, if the submodule has not been loaded, then the types it contains are not registered and will fail resolving. The simplest possible fix is to preemtively load all submodules upfront at the end of the root module's `__init__.py` file, which guarantees that all types have correctly been registered in the runtime (at the expense of some time and memory). Fixes aws/aws-cdk#16625

github-actions · 2021-10-11T13:19:07Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

njlynch · 2021-10-11T13:26:30Z

This was fixed in the underlying software (jsii), and will go out with the next jsii release (this week). We will then include this fix in the next CDK v2 release.

ivanmarquescepsa · 2021-10-19T07:55:26Z

Hi! is this fix be available in CDK V2rc24?

Thanks

njlynch · 2021-10-19T10:06:39Z

@keepler-ivanmarques - Yes, rc24 depends on [email protected], which contains the fix.

ivanmarquescepsa · 2021-10-19T11:23:02Z

Hi again!

Using the same example with rc24 i'm getting this error:


jsii.errors.JavaScriptError: 
  Error: ENOENT: no such file or directory, open '/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/lambda-layer-node-proxy-agent/layer/package.json'
      at Object.openSync (fs.js:497:3)
      at Object.readFileSync (fs.js:393:35)
      at hashFile (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/lambda-layer-node-proxy-agent/lib/node-proxy-agent-layer.js:35:20)
      at new NodeProxyAgentLayer (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/lambda-layer-node-proxy-agent/lib/node-proxy-agent-layer.js:23:28)
      at new ClusterResourceProvider (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/aws-eks/lib/cluster-resource-provider.js:34:31)
      at Function.getOrCreate (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/aws-eks/lib/cluster-resource-provider.js:63:85)
      at new ClusterResource (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/aws-eks/lib/cluster-resource.js:26:78)
      at new Cluster (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/aws-eks/lib/cluster.js:414:50)
      at /private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/tmpyohbwe46/lib/program.js:8432:58
      at Kernel._wrapSandboxCode (/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/tmpyohbwe46/lib/program.js:8860:24)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 48, in <module>
    stack = EksCluster(app, "test-cdkv2-eks-cluster", vpc_id=os.getenv('VPC_ID'), env=env_EU )
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/jsii/_runtime.py", line 86, in __call__
    inst = super().__call__(*args, **kwargs)
  File "app.py", line 32, in __init__
    cluster = eks.Cluster(self, id,
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/jsii/_runtime.py", line 86, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/aws_cdk/aws_eks/__init__.py", line 9232, in __init__
    jsii.create(self.__class__, self, [scope, id, props])
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 290, in create
    response = self.provider.create(
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 344, in create
    return self._process.send(request, CreateResponse)
  File "/workspace/cdk-eks-cluster-demo-aspects/.venvcdk2/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 326, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: ENOENT: no such file or directory, open '/private/var/folders/dk/g35p0fxd719dz_nbyct9y1b40000gp/T/jsii-kernel-za59XW/node_modules/aws-cdk-lib/lib/lambda-layer-node-proxy-agent/layer/package.json'

`

jzhn · 2021-10-19T18:45:39Z

I had the same issue as @keepler-ivanmarques posted above that package.json not found when declaring EKS Cluster in my stack. (using TypeScript/JavaScript).

I believe this is a separate issue, created #17059 to track that.

ivanmarquescepsa added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 23, 2021

github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Sep 23, 2021

github-actions bot assigned otaviomacedo Sep 23, 2021

ivanmarquescepsa changed the title ~~(aws-eks): Cannot~~ (aws-eks): Cannot apply Aspects to EKS cluster stack Sep 23, 2021

otaviomacedo added effort/small Small work item – less than a day of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Oct 1, 2021

otaviomacedo added this to the [GA] CDK V2 milestone Oct 1, 2021

otaviomacedo removed their assignment Oct 1, 2021

rix0rrr changed the title ~~(aws-eks): Cannot apply Aspects to EKS cluster stack~~ (aws-eks)(v2): Cannot apply Aspects to EKS cluster stack Oct 8, 2021

github-actions bot assigned otaviomacedo Oct 8, 2021

rix0rrr changed the title ~~(aws-eks)(v2): Cannot apply Aspects to EKS cluster stack~~ (jsii)(v2): Cannot apply Aspects to EKS cluster stack Oct 8, 2021

rix0rrr added the jsii This issue originates in jsii, or this feature must be implemented in jsii. label Oct 8, 2021

github-actions bot added the aws-cdk-lib Related to the aws-cdk-lib package label Oct 8, 2021

github-actions bot assigned nija-at Oct 8, 2021

njlynch unassigned nija-at Oct 8, 2021

otaviomacedo removed their assignment Oct 8, 2021

RomainMuller mentioned this issue Oct 8, 2021

fix(python): unknown type when submodule is not loaded aws/jsii#3049

Merged

RomainMuller closed this as completed in aws/jsii#3049 Oct 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(jsii)(v2): Cannot apply Aspects to EKS cluster stack #16625

(jsii)(v2): Cannot apply Aspects to EKS cluster stack #16625

ivanmarquescepsa commented Sep 23, 2021 •

edited

Loading

otaviomacedo commented Oct 1, 2021

rix0rrr commented Oct 8, 2021

rix0rrr commented Oct 8, 2021 •

edited

Loading

rix0rrr commented Oct 8, 2021

rix0rrr commented Oct 8, 2021

rix0rrr commented Oct 8, 2021 •

edited

Loading

github-actions bot commented Oct 11, 2021

njlynch commented Oct 11, 2021

ivanmarquescepsa commented Oct 19, 2021

njlynch commented Oct 19, 2021

ivanmarquescepsa commented Oct 19, 2021 •

edited

Loading

jzhn commented Oct 19, 2021

(jsii)(v2): Cannot apply Aspects to EKS cluster stack #16625

(jsii)(v2): Cannot apply Aspects to EKS cluster stack #16625

Comments

ivanmarquescepsa commented Sep 23, 2021 • edited Loading

Reproduction Steps

What did you expect to happen?

What actually happened?

Environment

Other

otaviomacedo commented Oct 1, 2021

rix0rrr commented Oct 8, 2021

rix0rrr commented Oct 8, 2021 • edited Loading

rix0rrr commented Oct 8, 2021

Huh.

rix0rrr commented Oct 8, 2021

rix0rrr commented Oct 8, 2021 • edited Loading

github-actions bot commented Oct 11, 2021

⚠️COMMENT VISIBILITY WARNING⚠️

njlynch commented Oct 11, 2021

ivanmarquescepsa commented Oct 19, 2021

njlynch commented Oct 19, 2021

ivanmarquescepsa commented Oct 19, 2021 • edited Loading

jzhn commented Oct 19, 2021

ivanmarquescepsa commented Sep 23, 2021 •

edited

Loading

rix0rrr commented Oct 8, 2021 •

edited

Loading

rix0rrr commented Oct 8, 2021 •

edited

Loading

ivanmarquescepsa commented Oct 19, 2021 •

edited

Loading