'cdk deploy' fails when using 'cdk bootstrap' while being behind a corporate proxy

[Obirah]

Hi,

I am using the CDK (currently 0.24.1) behind a corporate proxy. It is a squid proxy using NTLMv2; on the machine on which I am running the CDK, the proxy connection is established via CNTLM. So the application has to connect to localhost:3128

Until now, most of it worked nicely and all cdk diff and cdk deploy commands worked through the proxy. Today I hit the 50kb mark with the CF template of one of the CDK apps I'm deploying. Therefore, I started to use cdk bootstrap, which I didn't before. The CDKToolkit stack got setup without a problem and the staging S3 bucket exists.

Unfortunately, now I'm getting the following error on my machine when trying to use the same cdk deploy as before:

my-stack: deploying...

 ❌  my-stack failed: NetworkingError: Parse Error
Parse Error

When specifying the CNTLM proxy on my machine explicitly in cdk deploy ... --proxy http://localhost:3128 I get:

my-stack: deploying...

 ❌  my-stack failed: 407: null
null

I strongly assume, that this problem is related to the following issue in aws-sdk-js: aws/aws-sdk-js#1619 (comment)

Regarding the 407: I verified that the proxy is generally working on the machine by successfully curling s3.amazonaws.com - the 407 seems to be a common error when trying to use the JavaScript AWS SDK with CONNECT proxies.

The fact, that in my case the CDKToolkit bucket exists but is empty looks like the error happens during the upload of the CF template to the bucket. This is consistent with another experience I had with the CDK when the upload of a Lambda function to S3 failed.

The solution mentioned in the linked answer, cannot be used here as the user cannot change the proxy agent library in the CDK as the used agent implementation cannot be configured, if I understand this part of the code correctly: https://github.com/awslabs/aws-cdk/blob/54789139611bdae1e9d59e88c6355c45d7ceffe2/packages/aws-cdk/lib/api/util/sdk.ts#L71

Does someone have any suggestion for me on how to resolve this? Should the user be able to pass a custom proxy agent to the CDK or are there any other possible solutions to be considered here?

It would be great to find a solution for this, as those parts of the enterprise world that are running on Amazon get more and more interested in the CDK. Therefore, it would be great for it to run as well behind a corporate proxy as the AWS CLI does.

[Obirah]

Update:

I managed to adapt the solution for the "plain" JavaScript SDK (=> aws/aws-sdk-js#1619 (comment)) in my enterprise environment.

I can successfully use the following proxy agent with the SDK + CNTLM:

const TunnelAgent = require('tunnel-agent').httpsOverHttp;
const agent = new TunnelAgent({
    proxy: {
        hostname: 'localhost',
        port: 3128,
        protocol: 'http:',
        tunnel: true,
        headers: {
            'Connection': 'Keep-Alive'
        }
    }
});

AWS.config.update({httpOptions: { agent: agent}});
const s3 = new AWS.S3();

const params = {};
s3.listBuckets(params, function(err, data) {
    if (err) console.log(err, err.stack);
    else     console.log(data);
});

In order to confirm that the CDK deployment works with this agent, I put that piece of code into my local file /usr/local/lib/node_modules/aws-cdk/lib/api/util/sdk.js and the deployment worked.

So the big question is: what would be a nice way to get that into the CDK in a flexible and configurable way?
I'm also happy to contribute code once we have determined a path to the solution.

[eladb]

I suspect this might have been resolved by #693 - @Obirah if you are able to confirm, that would be great.

[Obirah]

I suspect this might have been resolved by #693 - @Obirah if you are able to confirm, that would be great.

Hi, this does not solve the issue. Specifying a proxy via args definitely works with the current version and last versions of the CDK.
The problem is the concrete implementation of the proxy-agent that is being used. The issue is in sdk.ts, we enterprise users need to be able to specify a concrete proxy agent implementation we want to use.

Please re-open this issue.

[Obirah]

@eladb @sam-goodwin just confirmed again that this issue still exists with 0.28.0. Please re-open it.