CodePipeline: Referencing a CodeBuild project that creates also an S3 bucket from another stack is marked as circular dependency

[devsheva]

Describe the bug

When referencing a CodeBuild Project to use it in a separate stack for a pipeline, synth fails cause this action is marked as possible circular dependency.

The CodeBuild Project stack creates also an associated S3 bucket for test reports, which seems the cause of the issue.

Expected Behavior

It should not be a circular dependency, since doing all the stuff in a single stack works fine, so I'm just doing some refactor

Current Behavior

jsii.errors.JavaScriptError: 
  @jsii/kernel.RuntimeError: Error: 'CodeBuildTestProject' depends on 'PIPELINE' (CodeBuildTestProject -> PIPELINE/pipeline-bucket/Resource.Arn). Adding this dependency (PIPELINE -> CodeBuildTestProject/test-project/Resource.Ref) would create a cyclic reference.
      at Kernel._Kernel_ensureSync (/private/var/folders/ct/5z32007s04z29v36fhnqq21c0000gn/T/tmp176amtgg/lib/program.js:10502:23)
      at Kernel.invoke (/private/var/folders/ct/5z32007s04z29v36fhnqq21c0000gn/T/tmp176amtgg/lib/program.js:9866:102)
      at KernelHost.processRequest (/private/var/folders/ct/5z32007s04z29v36fhnqq21c0000gn/T/tmp176amtgg/lib/program.js:11707:36)
      at KernelHost.run (/private/var/folders/ct/5z32007s04z29v36fhnqq21c0000gn/T/tmp176amtgg/lib/program.js:11667:22)
      at Immediate._onImmediate (/private/var/folders/ct/5z32007s04z29v36fhnqq21c0000gn/T/tmp176amtgg/lib/program.js:11668:46)
      at process.processImmediate (node:internal/timers:478:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mateo/cdk-bug-src/app.py", line 18, in <module>
    app.synth()
  File "/Users/mateo/cdk-bug-src/.venv/lib/python3.10/site-packages/aws_cdk/__init__.py", line 19855, in synth
    return typing.cast(_CloudAssembly_c693643e, jsii.invoke(self, "synth", [options]))
  File "/Users/mateo/cdk-bug-src/.venv/lib/python3.10/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/Users/mateo/cdk-bug-src/.venv/lib/python3.10/site-packages/jsii/_kernel/__init__.py", line 399, in invoke
    response = self.provider.invoke(
  File "/Users/mateo/cdk-bug-src/.venv/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 380, in invoke
    return self._process.send(request, InvokeResponse)
  File "/Users/mateo/cdk-bug-src/.venv/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 342, in send
    raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Error: 'CodeBuildTestProject' depends on 'PIPELINE' (CodeBuildTestProject -> PIPELINE/pipeline-bucket/Resource.Arn). Adding this dependency (PIPELINE -> CodeBuildTestProject/test-project/Resource.Ref) would create a cyclic reference.

Subprocess exited with error 1

Reproduction Steps

app.py

#!/usr/bin/env python3
import os

import aws_cdk as cdk

from cdk_bug_src.codebuild_test_project import CodeBuildTestProject
from cdk_bug_src.pipeline_stack import PipelineV2Stack

app = cdk.App()

test_build_stack = CodeBuildTestProject(app, "CodeBuildTestProject",
                                        )

PipelineV2Stack(app, "PIPELINE",
                cb_test_build=test_build_stack.project
                )

app.synth()

codebuild_test_project.py

from aws_cdk import (
    aws_s3 as s3,
    aws_codebuild as codebuild,
    Duration,
    aws_logs as logs, Stack, RemovalPolicy
)
from constructs import Construct


class CodeBuildTestProject(Stack):

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # create test build project
        cb_test_build = codebuild.PipelineProject(self, "test-project",
                                                  project_name="test-project",
                                                  )

        # Create S3 Bucket for test reports
        reports_bucket = s3.Bucket(
            self,
            "reports-bucket",
            bucket_name="reports-bucket",
            auto_delete_objects=True,
            removal_policy=RemovalPolicy.DESTROY
        )

        reports_bucket.grant_read_write(cb_test_build.role)

        # Create Report Group for test reports
        codebuild.ReportGroup(
            self,
            "test-project-report-group",
            report_group_name="test-project-report-group",
            export_bucket=reports_bucket,
            removal_policy=RemovalPolicy.DESTROY
        )

        self._cb_test_build = cb_test_build

    @property
    def project(self):
        return self._cb_test_build

pipeline_stack.py

import pprint

from aws_cdk import (
    Stack,
    aws_codecommit as codecommit,
    aws_codebuild as codebuild,
    aws_codedeploy as codedeploy,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as codepipeline_actions,
    aws_logs as logs,
    aws_ecr as ecr,
    aws_s3 as s3,
    Tags, Duration, RemovalPolicy
)
from constructs import Construct


class PipelineV2Stack(Stack):

    def __init__(self, scope: Construct, construct_id: str, cb_test_build: codebuild.IProject,
                 **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # S3 BUCKER FOR PIPELINE
        ## ---------------------------------------------------------------- ##
        # Pipeline requires versioned bucket
        bucket = s3.Bucket(self, f"pipeline-bucket",
                           bucket_name=f"pipeline-bucket",
                           versioned=True,
                           auto_delete_objects=True,
                           removal_policy=RemovalPolicy.DESTROY
                           )

        # CODEPIPELINE DEFINITION
        ## ---------------------------------------------------------------- ##
        # Trace outputs throw pipeline actions
        source_output = codepipeline.Artifact()

        # Define pipeline construct
        pipeline = codepipeline.Pipeline(self, f"pipeline",
                                         artifact_bucket=bucket,
                                         )

        # CODEPIPELINE ACTION: SOURCE
        ## ---------------------------------------------------------------- ##
        # Create codecommit source action for pipeline
        source_action = codepipeline_actions.CodeStarConnectionsSourceAction(
            action_name="BitBucket",
            owner="test",
            repo='bitbucket-test-repo',
            branch='dev',
            output=source_output,
            connection_arn='codestar:arn'
        )

        # Add source action to the pipeline
        pipeline.add_stage(
            stage_name="Source",
            actions=[source_action]
        )

        # CODEPIPELINE ACTION: BUILD
        ## ---------------------------------------------------------------- ##

        # Add Test stage before real build
        # define test action with codebuild project
        test_action = codepipeline_actions.CodeBuildAction(
            action_name='RSpecTest',
            input=source_output,
            project=cb_test_build
        )

        # Add test action to pipeline
        pipeline.add_stage(
            stage_name="Test",
            actions=[test_action]
        )

        # Create build project for the pipeline
        cb_docker_build = codebuild.PipelineProject(self, f"build-project",
                                                    project_name=f"build-project",
                                                    environment=codebuild.BuildEnvironment(
                                                        build_image=codebuild.LinuxBuildImage.AMAZON_LINUX_2_ARM,
                                                        privileged=True,
                                                        compute_type=codebuild.ComputeType.SMALL
                                                    ),
                                                    description=f"CodeBuild Project",
                                                    )

        # define build action with codebuild project
        build_action = codepipeline_actions.CodeBuildAction(
            action_name="CodeBuild",
            project=cb_docker_build,
            input=source_output,
            outputs=[codepipeline.Artifact(artifact_name='deploy')]
        )

        # Add build action to the pipeline
        pipeline.add_stage(
            stage_name="Build",
            actions=[build_action]
        )

        # CODEPIPELINE ACTION: DEPLOY
        ## ---------------------------------------------------------------- ##
        # Define deployment group with existing EC2
        deployment_group = codedeploy.ServerDeploymentGroup(self, f"deployment-group",
                                                            )

        deploy_action = codepipeline_actions.CodeDeployServerDeployAction(
            action_name="CodeDeploy",
            input=codepipeline.Artifact(artifact_name='deploy'),
            deployment_group=deployment_group
        )

        # Add deploy action to the pipeline
        pipeline.add_stage(
            stage_name="Deploy",
            actions=[deploy_action]
        )

        # GRANT PERMISSIONS TO CODEBUILD PROJECT
        ## ---------------------------------------------------------------- ##
        # codebuild iam permissions to read write s3
        bucket.grant_read_write(cb_docker_build)

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.135.0 (build d46c474)

Framework Version

No response

Node.js Version

18.17.1

OS

macOS Sonoma 14

Language

Python

Language Version

3.10.14

Other information

No response

[pahud]

Error: 'CodeBuildTestProject' depends on 'PIPELINE' (CodeBuildTestProject -> PIPELINE/pipeline-bucket/Resource.Arn). Adding this dependency (PIPELINE -> CodeBuildTestProject/test-project/Resource.Ref) would create a cyclic reference.

You will need to refactor your code to break the cyclic dependency/reference.

CodeBuildTestProject -> PIPELINE/pipeline-bucket/Resource.Arn

Condition 1: This is because your codebuild project defined in CodeBuildTestProject stack has to access the pipeline-bucket defined in PipelineV2Stack

PIPELINE -> CodeBuildTestProject/test-project/Resource.Ref

Condition 2: This is because the pipeline in PipelineV2Stack requires the codebuild project as in its build stage which is defined in CodeBuildTestProject stack.

You will need to eliminate either condition 1 or 2.

I guess break the dep from condition 1 might be easier. You can try create the bucket in CodeBuildTestProject and allow PipelineV2Stack to reference that bucket.

I haven't tested it in my account but generally you will need to list conditions as above and try to refactor your code to break the cyclic dependency somewhere.

Hope this helps.

[devsheva]

I'll try your suggestion, which probably will be breaking the dependency from condition 1. I'll let you know if it fixes the issue.
In the meantime may I ask if it's a good way to create the pipeline bucket inside the CodeBuildTestProject stack?

Edit:

Important: (Related to Condition 1)

I forgot the fact that the CodeBuildTestProject must be only one, so it will be shared to all pipelines. The problem is that the pipeline stack gets invoked for each environment, so creating the pipeline bucket in the CodeBuildTestProject stack would lead to a single bucket without a specific environment to attach to.

[devsheva]

bump?