S3 Output Plugin not working with IMDSv1 and kube2iam

[joshbranham]

Bug Report

Describe the bug
When using the AWS S3 output plugin, coupled with IAM assume roles in Kubernetes, Fluent-bit fails to get credentials, seemingly because it does not support IMDSv1? To work around this, we can explicitly set the role_arn value to the role that kube2iam would have otherwise automatically assumed on the Pods behalf if it had hit the IMDSv1 metadata endpoint for credentials.

To Reproduce
Run Fluent-bit on an EC2 instance with IMDSv1 only, or in a Kubernetes cluster using kube2iam or similar (which only supports IMDSv1 currently) and attempt to write logs to an S3 bucket.

Expected behavior
Fluent-bit will make a request to the metadata service, when provided no credentials, to receive the instance profile.

Your Environment

• Version used: 1.9.4
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes, using kube2iam for IAM roles
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context
Is this expected to just only work with IMDSv2? I noticed the filter_s3 plugin supports setting imds_version v1 but output_s3 does not.

Here is the error output:

[2022/11/29 18:55:08] [error] [aws_client] auth error, refreshing creds
[2022/11/29 18:55:08] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2022/11/29 18:55:08] [error] [output:s3:s3.0] PutObject API responded with error='AccessDenied', message='Access Denied'
[2022/11/29 18:55:08] [error] [output:s3:s3.0] Raw PutObject response: HTTP/1.1 403 Forbidden
x-amz-request-id: 5S6TFXEWGJEF2T70
x-amz-id-2: <redacted>
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Tue, 29 Nov 2022 18:55:07 GMT
Server: AmazonS3
Connection: close

I would have expected the above to log trying to connect to IMDS?

This might be related, although we are running 1.9.4 #4388

[sebsthiel]

I am having the same problem. We have a fluentbit sidecar that needs to write to an S3 bucket.
Even though we have specified the role_arn we are also getting:
[2022/12/02 13:26:00] [error] [aws_client] auth error, refreshing creds
[2022/12/02 13:26:00] [error] [aws_credentials] Shared credentials file /.aws/credentials does not exist

[joshbranham]

I am having the same problem. We have a fluentbit sidecar that needs to write to an S3 bucket.

Even though we have specified the role_arn we are also getting:

[2022/12/02 13:26:00] [error] [aws_client] auth error, refreshing creds

[2022/12/02 13:26:00] [error] [aws_credentials] Shared credentials file /.aws/credentials does not exist

To clarify, it only works for us if we specify the role_arn, where because of kube2iam, if a process makes a request to the metadata service, kube2iam hijacks that, assumes the role we define in the Pod annotations, and then hands the sts credentials back to the process.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[joshbranham]

Still an issue..

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[manojkal]

@joshbranham @sebsthiel
Even I am facing same issue, did you guys figure out the solution ?

[joshbranham]

@joshbranham @sebsthiel Even I am facing same issue, did you guys figure out the solution ?

We "fixed" it by expliciting specifying the role to assume in the [OUTPUT] config as shown below. In theory this shouldn't be required, since the sdk should be reaching out to the metadata service, where kube2iam will hijack the request and assume the correct role based on pod annotations.

[OUTPUT]
    Name            s3
    Match           ......
    bucket          ${SOME_BUCKET}
    region          us-east-1
    compression     gzip
    use_put_object  on
    total_file_size 5M

    # For some reason, builtin kube2iam isn't working out of the box so we need to explicitly
    # assume this role in the config.
    role_arn        arn:aws:iam::ACCOUNT:role/role-name
   ........