Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS IAM auth 'Content-Length' header causes validation error when sent as number instead of string or []strings #3763

Closed
DaniGuardiola opened this issue Jan 8, 2018 · 23 comments
Milestone

Comments

@DaniGuardiola
Copy link

DaniGuardiola commented Jan 8, 2018

Environment:

  • Vault Version: v0.9.1
  • Operating System/Architecture: Ubuntu 16.04 LTS on EC2 instance

Vault Config File:

(dev server)

Expected Behavior:

Vault returns the IAM auth login response, as described in the docs

Actual Behavior:

Vault returned this error:
Error parsing iam_request_headers: header "Content-Length" value "43" has type json.Number, not string or []interface

Steps to Reproduce:

Create request and sign it with aws4 npm module:

Original signed request (aws4)

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { 'X-Vault-AWS-IAM-Server-ID': 'i-99999999999993',
     Host: 'sts.amazonaws.com',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 43,
     'X-Amz-Security-Token': 'FQoaaaaEML//////////wEaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaL7O0gU=',
     'X-Amz-Date': 20210401T171213Z',
     Authorization: 'AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAA/20210401/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-vault-aws-iam-server-id, Signature=8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

Formatted for vault

{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'some-role',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHRaaaaaaaaaaaaaaaaaaaaaaavbS8=',
     iam_request_body: 'QWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaQ==',
     iam_request_headers: 'eyJYLAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAafQ==' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }

Important Factoids:

  • I faked the data up there for obvious reasons, but I can guarantee the base64 text contains the correct (stringified) version of the json (with 'Content-Length': 43 - as a number not a string)
  • Running the dev server
  • Vault aws backend initialized
  • Vault AWS role and user are setup and access key and secret are configured in aws auth: the role policy is copied from the aws auth documentation without the `assumeRole' permission (not needed)
  • No header enforcement setup (the header in the request serves no purpose for now, but I don't think it influences this issue)
  • Using the request-promise-native npm module for requests
  • Using the aws4 module for aws request signing. This module adds the Content-Length header before signing the request, but it does so correctly (as a number).
  • Programming this as part of the node-vault module (I'm adding some functionalities), here's the failing code

Please help!
This is blocking and urgent for my company :/ thanks in advance!

@DaniGuardiola
Copy link
Author

DaniGuardiola commented Jan 8, 2018

I removed the additional headers and now vault returns this error:

error making upstream request: error parsing STS response

This is the request object that aws4 returns now (includes aws4 settings that are later discarded when formatting the request to vault):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { 'X-Vault-AWS-IAM-Server-ID': 'i-0222222222223',
     Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=AAAAAAAAAAFA/20222222/eu-west-1/sts/aws4_request, SignedHeaders=host;x-vault-aws-iam-server-id, Signature=eAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAa8e' },
  region: 'eu-west-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

This is the actual code now

@DaniGuardiola
Copy link
Author

Tested without the X-Vault-AWS-IAM-Server-ID header with the same result error making upstream request: error parsing STS response

@DaniGuardiola DaniGuardiola changed the title AWS IAM auth: Error parsing iam_request_headers: header "Content-Length" value "43" has type json.Number, not string or []interface AWS IAM auth does not work for some reason Jan 8, 2018
@joelthompson
Copy link
Contributor

Hey @DaniGuardiola -- I think there are two issues here.

On the first issue you mentioned, the AWS NodeJS SDK is returning a content-length header as an int, when HTTP headers should either a string or array of strings. This is the same issue mentioned in #2810 (comment) but it's a different error message now due to changes to support a string or array of strings (as opposed to just the latter).

On the second, try setting the region to us-east-1. Some background here, but in short, your request's Authorization header has a credential scope with a region of eu-west-1 but the Host header is set to sts.amazonaws.com which, in my experiments, expects a region of us-east-1. If the Vault admin hasn't overridden the config/client sts_endpoint value, this is what you need to do. And if the Vault admin has, then you need to ensure that the AWS SDK generates the right host header for that region-specific endpoint.

@DaniGuardiola
Copy link
Author

Hi @joelthompson! Thank you very much for your response!

I tried changing the region to us-east-1 and I get the same error message :(

This is the new request (aws4 input):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20051111/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=dasddddasdasdasdawdawdasdasdasdasdasdasdasdasdasdasd' },
  region: 'us-east-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

@jefferai
Copy link
Member

jefferai commented Jan 8, 2018

@DaniGuardiola Any chance you're able to build and test from our repo?

@jefferai
Copy link
Member

jefferai commented Jan 8, 2018

Please try the https://github.com/hashicorp/vault/tree/aws-jsonnumber branch and let me know if it fixes the json.Number error with your original data set.

@DaniGuardiola
Copy link
Author

@jefferai Done

This is the signed request options (ignore non-request stuff, it's aws4's)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: false,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 43,
     'X-Amz-Date': '20180108T220028Z',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=b09aa0e47e9980357ab95970b301d64eadb3f116a2f5ce58c3902133c4284f96' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

And this is the request passed to the request(-promise-native) module

{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQ7IGNoYXJzZXQ9dXRmLTgiLCJDb250ZW50LUxlbmd0aCI6NDMsIlgtQW16LURhdGUiOiIyMDE4MDEwOFQyMjAyMjFaIiwiQXV0aG9yaXphdGlvbiI6IkFXUzQtSE1BQy1TSEEyNTYgQ3JlZGVudGlhbD11bmRlZmluZWQvMjAxODAxMDgvdXMtZWFzdC0xL3N0cy9hd3M0X3JlcXVlc3QsIFNpZ25lZEhlYWRlcnM9Y29udGVudC1sZW5ndGg7Y29udGVudC10eXBlO2hvc3Q7eC1hbXotZGF0ZSwgU2lnbmF0dXJlPTY1OWE5NTk5NmQxNjc0YzIyMmZkZTk3Mjg3OWUxMmRlZDYzNDVjZTAzM2RkYTllZGE4NjM1MTJiY2RmY2NiYWIifQ==' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }

(The data is not really sensitive so I'm not gonna censor it anymore for now)

Now I get this error:
Error parsing iam_request_headers: header "Content-Length" value %!q(float64=43) has type float64, not string or []interface

@DaniGuardiola
Copy link
Author

And when I remove the extra headers I still get the error parsing STS response message

@DaniGuardiola
Copy link
Author

Full example without the Content-Length header

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=c558f10dbf80e34b1e167d99d84e2b2f51144dcf510e73918096d19c52ce9b75' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

@DaniGuardiola
Copy link
Author

DaniGuardiola commented Jan 8, 2018

The vault cli also fails:

$ vault write auth/aws/login role=ssh-admin \
	iam_http_request_method=POST \
	iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
	iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
	iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9'
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://vault.vpc/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response

@DaniGuardiola
Copy link
Author

DaniGuardiola commented Jan 8, 2018

The credentials were not set because of a small bug I just fixed, now it's not undefined but I get the same error. Halp! :P @joelthompson @jefferai

@joelthompson
Copy link
Contributor

Was just going to comment on that (wasn't sure if undefined was your redaction or not earlier), can you repost the Vault login data?

The error you get should only happen when STS returns a 200 response but the XML is in a format that's not recognized by Vault, which shouldn't ever happen (though the error gets eaten). Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

Also, since you're able to test dev builds, I'll try to push a branch to my fork later tonight that should add more verbose logging messages.

@DaniGuardiola
Copy link
Author

DaniGuardiola commented Jan 8, 2018

@joelthompson here :)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=ASIAIBLI3BFCJY7GUO4A/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=e7d3a23018c8ba39cc1f415a0308597670ec183bb9b42dbb424027cb393259f9' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFJQkxJM0JGQ0pZN0dVTzRBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1lN2QzYTIzMDE4YzhiYTM5Y2MxZjQxNWEwMzA4NTk3NjcwZWMxODNiYjliNDJkYmI0MjQwMjdjYjM5MzI1OWY5In0=' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

@DaniGuardiola
Copy link
Author

Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

I don't think so :/

Would it help if I insert some logs somewhere, rebuild and re-run the dev server? Can I extend that error message in the code to add more information? Don't wanna pressure you but I gotta get this done soon so if there's any way I can speed up the process just tell me :)

@DaniGuardiola
Copy link
Author

Wait, I'm using nginx to publish the vault dev server, I proxy localhost:8200 through the 80 port to expose it to other instances in the VPC. Is that a problem? Every other request works just fine (like help or status).

@DaniGuardiola
Copy link
Author

:(

ubuntu@ip-redacted:~$ export VAULT_ADDR='http://127.0.0.1:8200'
ubuntu@ip-redacted:~$ vault status
Seal Type: shamir
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.1
Cluster Name: vault-cluster-5376bd99
Cluster ID: dcf4b559-0607-5b7a-2993-28caa1bc5011

High-Availability Enabled: false
ubuntu@ip-redacted:~$ vault write auth/aws/login role=ssh-admin \
> iam_http_request_method=POST \
> iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
> iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
> iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFKQ1NRNlg1RUhKTFZRTkJBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1mYjY0YjM2MWQyMDY1YTFkMzhjZjUyYjRhN2ZhYmFiMWZlZDM2ZDEyOTRkOWU4N2Q5OWE5OTU3NGZjMDVkNzc3In0='
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response

@joelthompson
Copy link
Contributor

I'm starting to think the issue is your setting doNotModifyHeaders to true which then causes the Content-Length and Content-Type headers to be missing from iam_request_headers.

First, can you take your original code without doNotModifyHeaders but modify the Content-Length header to be a string before JSON-serializing it?

If that doesn't work, try changing this line:

return nil, fmt.Errorf("error parsing STS response")

To something like:

		return nil, fmt.Errorf("error parsing STS response; original response: %q; error: %q", string(responseBody), err)

and then rebuild and rerun the test; I think that should give more information.

@DaniGuardiola
Copy link
Author

@joelthompson I can modify the aws4 version locally to do so, gimme a sec

@DaniGuardiola
Copy link
Author

DaniGuardiola commented Jan 8, 2018

@joelthompson bingo!!

{ request_id: 'asdfa3af4-as43-dsf3-sdf234-asdf7667sd7f6',
  lease_id: '',
  renewable: false,
  lease_duration: 0,
  data: null,
  wrap_info: null,
  warnings: null,
  auth: 
   { client_token: '1a9ec23d-bcf0-f810-a193-ea6dfb8ec420',
     accessor: '96d9ba25-7aad-dfb1-1ef8-ad2ef4135a7e',
     policies: [ 'default', 'dev', 'prod' ],
     metadata: 
      { account_id: '81972398723',
        auth_type: 'iam',
        canonical_arn: 'arn:aws:iam::98787996897898:role/ssh-admin',
        client_arn: 'arn:aws:sts::192837981273:assumed-role/ssh-admin/i-78327237832782376',
        client_user_id: 'ASGDJKWAGAWDWAD',
        inferred_aws_region: '',
        inferred_entity_id: '',
        inferred_entity_type: '' },
     lease_duration: 2764800,
     renewable: true,
     entity_id: '21hj3ghdsfg-asda-kj2hy3-asdf-8a7s5df5sadf' } }

(data modified for privacy)

@DaniGuardiola
Copy link
Author

The Content-Length header should always contain a string or an array of strings for all AWS request to any service, right? Because then I'll send a pull request to the aws4 project with the fix. For now I'll use my own version as a workaround.

Thank you both for your fast response! If you need me for further testing you can hmu :)

@jefferai
Copy link
Member

jefferai commented Jan 9, 2018

@DaniGuardiola I've updated (via force-push) the aws-jsonnumber branch. It should now accept the Content-Length properly. Can you check?

@jefferai jefferai added this to the 0.9.2 milestone Jan 9, 2018
@DaniGuardiola
Copy link
Author

Checked (9b13355e78dcb48a07049fc71f3ce64d8835403e) and it works just fine, thanks! Will this be ready in 0.9.2? I'm gonna add the workaround in node-vault to support older versions anyway, but good to know it's solved.

@DaniGuardiola DaniGuardiola changed the title AWS IAM auth does not work for some reason AWS IAM auth 'Content-Length' header causes validation error when sent as number instead of string or []strings Jan 10, 2018
@jefferai
Copy link
Member

It'll be in 0.9.2, yep!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants