aws: Include IMDSv2 support for EC2 instances #4086

matthewfala · 2021-09-13T20:59:39Z

Add IMDSv2 Support to AWS Fluent Bit plugins

Instances with only IMDSv2 enabled should now have access to Fluent Bit.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

[N/A] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Config File

[SERVICE]
     Grace 30
     Log_Level debug
[INPUT]
     Name dummy
     Tag dummy
[OUTPUT]
     Name cloudwatch_logs
     Match *
     log_stream_prefix test
     log_group_name fluent_group
     auto_create_group true
     region us-west-2

Debug Log

@"\e[1mFluent Bit v1.9.0\e[0m\r\n"
@"* \e[1m\e[93mCopyright (C) 2019-2021 The Fluent Bit Authors\e[0m\r\n"
@"* \e[1m\e[93mCopyright (C) 2015-2018 Treasure Data\e[0m\r\n"
@"* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd\r\n"
@"* https://fluentbit.io\r\n"
@"\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] Configuration:\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  flush time     | 5.000000 seconds\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  grace          | 30 seconds\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  daemon         | 0\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] ___________\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  inputs:\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]      dummy\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] ___________\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  filters:\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] ___________\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  outputs:\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]      cloudwatch_logs.0\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] ___________\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m]  collectors:\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] [engine] started (pid=52932)\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [engine] coroutine stack size: 12288 bytes (12.0K)\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [storage] [cio stream] new stream registered: dummy.0\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] [storage] version=1.1.1, initializing...\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] [storage] in-memory\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[92m info\e[0m] [cmetrics] version=0.2.1\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [cloudwatch_logs:cloudwatch_logs.0] created event channels: read=19 write=20\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Initialized Env Provider in standard chain\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Initialized AWS Profile Provider in standard chain\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:51\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set\r\n"
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Initialized EC2 Provider in standard chain\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Sync called on the EC2 provider\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Init called on the env provider\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Init called on the profile provider\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Reading shared config file.\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Reading shared credentials file.\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Shared credentials file /Users/falamatt/.aws/credentials does not exist\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Init called on the EC2 IMDS provider\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] requesting credentials from EC2 IMDS\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server 169.254.169.254:80 will close connection #21\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_client] (null): http_do=0, HTTP Status: 401\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:53\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:54\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server 169.254.169.254:80 will close connection #21\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:57\e[1m]\e[0m [\e[93mdebug\e[0m] **_[imds] using IMDSv2_**\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:57\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:58\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server 169.254.169.254:80 will close connection #21\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:58\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Requesting credentials for instance role Fluent_Bit_Test_Instance\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:59\e[1m]\e[0m [\e[93mdebug\e[0m] [imds] using IMDSv2\r\n"
@"\e[1m[\e[0m2021/09/13 13:04:59\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:00\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server 169.254.169.254:80 will close connection #21\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:01\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] upstream_set called on the EC2 provider\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:01\e[1m]\e[0m [\e[93mdebug\e[0m] [router] match rule dummy.0:cloudwatch_logs.0\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:01\e[1m]\e[0m [\e[92m info\e[0m] [sp] stream processor started\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:06\e[1m]\e[0m [\e[93mdebug\e[0m] [task] created task=0x1027040d0 id=0 OK\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:06\e[1m]\e[0m [\e[92m info\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group fluent_group\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Requesting credentials from the EC2 provider..\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server logs.us-west-2.amazonaws.com:443 will close connection #28\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_client] logs.us-west-2.amazonaws.com: http_do=0, HTTP Status: 400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogGroup http status=400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[92m info\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group fluent_group already exists\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[92m info\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream testdummy in log group fluent_group\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Requesting credentials from the EC2 provider..\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server logs.us-west-2.amazonaws.com:443 will close connection #28\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_client] logs.us-west-2.amazonaws.com: http_do=0, HTTP Status: 400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream http status=400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[92m info\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Log Stream testdummy already exists\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sending 4 events\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream testdummy\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Requesting credentials from the EC2 provider..\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] server logs.us-west-2.amazonaws.com:443 will close connection #28\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_client] logs.us-west-2.amazonaws.com: http_do=0, HTTP Status: 400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=400\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sequence token was invalid, will retry\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sending 4 events\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:07\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream testdummy\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [http_client] not using http_proxy for header\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [aws_credentials] Requesting credentials from the EC2 provider..\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [upstream] KA connection #28 to logs.us-west-2.amazonaws.com:443 is now available\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sent events to testdummy\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [output:cloudwatch_logs:cloudwatch_logs.0] Sent 4 events to CloudWatch\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [out coro] cb_destroy coro_id=0\r\n"
@"\e[1m[\e[0m2021/09/13 13:05:08\e[1m]\e[0m [\e[93mdebug\e[0m] [task] destroy task=0x1027040d0 (task_id=0)\r\n"

Disabled IMDS -> Enabled IMDS

aws ec2 modify-instance-metadata-options --instance-id i-001234567 --http-endpoint disabled
Run fluent bit on instance i-001234567
aws ec2 modify-instance-metadata-options --instance-id i-004f8a124a6f1a9eb --http-endpoint enabled
Result:

Valgrind

System Operation

Unit Tests

test_ec2_provider_v2
test_ec2_provider_v1
test_ec2_provider_version_detection_error
test_ec2_provider_acquire_token_error
test_ec2_provider_metadata_request_error
test_ec2_imds_create_and_destroy
Unit test result

Design Motivation

Motivation

Here is my motivation behind creating a new structure for IMDS. The plugins/filters_aws/aws.c file has essentially the same routines listed here, but it takes the specialized filter context for each routine. The previous version of IMDSv1 that was here, had some of the same code, copy pasted, but this code took a flb_aws_client as context. I could have continued the copy paste pattern, but though that code duplication would result in greater overhead in maintenance in the future.

aws.c Compatability

The IMDS struct is designed to be fully compatible with plugins/aws_filters/aws.c with only about 5 lines of code changes. Moreover, much of aws.c's code can be deleted. Wiring the IMDS struct to the aws filter is out of the scope of this feature (since I am not going to retest the aws plugin), but also list the 5 code changes in aws.c that need to take place.

I expect that in the future, as changes are needed in aws.c and aws.c is being actively tested again, we can wire up the IMDS struct to aws.c and delete all the duplicated code.

src/aws/flb_aws_credentials_ec2.c

PettitWesley · 2021-09-14T02:08:53Z

include/fluent-bit/aws/flb_aws_imds.h

+/*
+ * Create IMDS context
+ * Returns NULL on error
+ * Note: Setting the FLB_IO_ASYNC flag is the job of the client.
+ * Flag Set Example: flags &= ~(FLB_IO_ASYNC)
+ */


I love the code comments on each function and the thought put into overall organization here 👏

src/aws/flb_aws_imds.c

PettitWesley

First pass review, will do a second look tomorrow

PettitWesley · 2021-09-14T02:15:42Z

include/fluent-bit/aws/flb_aws_imds.h

+struct flb_aws_imds_config {
+    int use_imds_version;  // FLB_AWS_IMDS_VERSION_EVALUATE for automatic detection
+};
+


So I like the constants, esp FLB_AWS_IMDS_VERSION_EVALUATE, and I am fine with this separate config structure, but I want to say tho that it feels a bit like over-kill to have a structure for a single option.

Here is my motivation behind the config structure. A default config structure is declared in imds.c. This default config structure is used as a config template when our IMDS struct is initialized. That way in the future, if we want to add some more options to customize the IMDS' behavior (e.g. fallback behavior or number of retries) we can add a default configuration in one place and not have to worry about changing IMDS initialization code in other places.

Use

You can see the use of the default config structure used as a configuration template in create_ec2_provider() (and anywhere else IMDS is initialized future).

If you think that this ease of maintenance motivation is not strong enough to warrant the additional code, I can take the parameter in the create function. That's not a problem. Lmk.

src/aws/flb_aws_imds.c

zhonghui12

Most looks code. Good work! As it does refactor many work, be sure to test everything is fine - not only the feature support.

zhonghui12 · 2021-09-14T23:09:13Z

include/fluent-bit/aws/flb_aws_imds.h

+/*
+ * Create IMDS context
+ * Returns NULL on error
+ * Note: Setting the FLB_IO_ASYNC flag is the job of the client.
+ * Flag Set Example: flags &= ~(FLB_IO_ASYNC)
+ */


src/aws/flb_aws_imds.c

zhonghui12 · 2021-09-14T23:23:25Z

src/aws/flb_aws_imds.c

+    }
+
+    if (c->resp.status != 200) {
+        if (c->resp.payload_size > 0) {


More debug information is good but should we add more checks just for debug log? Just want to discuss this a little bit,

src/aws/flb_aws_imds.c

src/aws/flb_aws_credentials_ec2.c

zhonghui12

Good work! Mostly LGTM. Left some small comments. Also, you might want to rebase your commits. Like one for supporting IMDSv2 and the other one is for adding unit tests.

include/fluent-bit/aws/flb_aws_imds.h

tests/internal/aws_client_mock.c

src/aws/flb_aws_imds.c

tests/internal/aws_client_mock.c

tests/internal/aws_client_mock.h

PettitWesley · 2021-09-21T23:22:48Z

@matthewfala may be we should call out IMDSv2 support here: https://github.com/fluent/fluent-bit-docs/blob/master/administration/aws-credentials.md

matthewfala · 2021-09-22T17:12:46Z

Here's the diff from the last approved commit to after the forced push:

falamatt$ git diff d9b623cd3
diff --git a/src/aws/flb_aws_imds.c b/src/aws/flb_aws_imds.c
index ef0b699f6..d0193292b 100644
--- a/src/aws/flb_aws_imds.c
+++ b/src/aws/flb_aws_imds.c
@@ -148,17 +148,13 @@ int flb_aws_imds_request_by_key(struct flb_aws_imds *ctx, const char *metadata_p
         ec2_imds_client, FLB_HTTP_GET, metadata_path, NULL, 0, &token_header,
         (imds_version == FLB_AWS_IMDS_VERSION_1) ? 0 : 1);
     if (!c) {
+        // Exit gracefully allowing for retrys
+        flb_warn("[imds] failed to retrieve metadata");
         return -1;
     }
 
     /* Detect invalid token */
     if (imds_version == FLB_AWS_IMDS_VERSION_2 && c->resp.status == 401) {
-        /* TODO: Detect IMDS version requirement change */
-        /* Currently unable to test the following */
-        /* if (imds_version == FLB_AWS_IMDS_VERSION_1) {
-            ctx->imds_version = FLB_AWS_IMDS_VERSION_2;
-        } */
-
         /* Refresh token and retry request */
         flb_http_client_destroy(c);
         ret = refresh_imds_v2_token(ctx);
diff --git a/tests/internal/aws_client_mock.c b/tests/internal/aws_client_mock.c
index b6b3d0255..7f14ece0e 100644
--- a/tests/internal/aws_client_mock.c
+++ b/tests/internal/aws_client_mock.c
@@ -141,11 +141,14 @@ static struct flb_http_client *flb_aws_client_mock_vtable_request(
     struct flb_aws_client *aws_client, int method, const char *uri, const char *body,
     size_t body_len, struct flb_aws_header *dynamic_headers, size_t dynamic_headers_len)
 {
+    int ret;
+
     /* Get access to mock */
     struct flb_aws_client_mock *mock = (struct flb_aws_client_mock *)aws_client;
 
     /* Check that a response is left in the chain */
-    if (!TEST_CHECK(mock->next_request_index < mock->request_chain->length)) {
+    ret = TEST_CHECK(mock->next_request_index < mock->request_chain->length);
+    if (!ret) {
         TEST_MSG(
             "[flb_aws_client_mock] %d mock responses provided. Attempting to call %d "
             "times. Aborting.",
@@ -176,11 +179,13 @@ static struct flb_http_client *flb_aws_client_mock_vtable_request(
             int header_found = FLB_FALSE;
             /* Search for header in request */
             for (int h = 0; h < dynamic_headers_len; ++h) {
-                if (strncmp(dynamic_headers[h].key, (char *)val1,
-                            dynamic_headers[h].key_len) == 0) {
+                ret = strncmp(dynamic_headers[h].key, (char *)val1,
+                              dynamic_headers[h].key_len);
+                if (ret == 0) {
                     /* Check header value */
-                    TEST_CHECK(strncmp(dynamic_headers[h].val, (char *)val2,
-                                       dynamic_headers[h].val_len + 1) == 0);
+                    ret = strncmp(dynamic_headers[h].val, (char *)val2,
+                                  dynamic_headers[h].val_len + 1);
+                    TEST_CHECK(ret == 0);
                     TEST_MSG("[aws_mock_client] Expected Header: (%s: %s)", (char *)val1,
                              (char *)val2);
                     TEST_MSG("[aws_mock_client] Received Header: (%s: %s)", (char *)val1,
@@ -218,7 +223,8 @@ static struct flb_http_client *flb_aws_client_mock_vtable_request(
                      (int)(uintptr_t)dynamic_headers_len);
         }
         else if (response_config->config_parameter == FLB_AWS_CLIENT_MOCK_EXPECT_URI) {
-            TEST_CHECK(strncmp(uri, (char *)val1, strlen((char *)val1) + 1) == 0);
+            ret = strncmp(uri, (char *)val1, strlen((char *)val1) + 1);
+            TEST_CHECK(ret == 0);
             TEST_MSG("[aws_mock_client] Expected URI: %s", (char *)val1);
             TEST_MSG("[aws_mock_client] Received URI: %s", uri);
         }
xxxxxxx:bin falamatt$

matthewfala · 2021-09-22T22:22:40Z

Here's the diff from the second forced push where c7e97e3 is the last commit before the forced push

falamatt$ git diff c7e97e306
diff --git a/src/aws/flb_aws_imds.c b/src/aws/flb_aws_imds.c
index d0193292b..fbba13b99 100644
--- a/src/aws/flb_aws_imds.c
+++ b/src/aws/flb_aws_imds.c
@@ -31,7 +31,7 @@
 static struct flb_aws_header imds_v2_token_ttl_header = {
     .key = "X-aws-ec2-metadata-token-ttl-seconds",
     .key_len = 36,
-    .val = "21600",  // 6 hours (ie maximum ttl)
+    .val = "21600",  /* 6 hours (ie maximum ttl) */
     .val_len = 5,
 };
 
@@ -39,8 +39,8 @@ static struct flb_aws_header imds_v2_token_ttl_header = {
 const static struct flb_aws_header imds_v2_token_token_header_template = {
     .key = "X-aws-ec2-metadata-token",
     .key_len = 24,
-    .val = "",     // Replace with token value
-    .val_len = 0,  // Replace with token length
+    .val = "",     /* Replace with token value */
+    .val_len = 0,  /* Replace with token length */
 };
 
 /* Declarations */
@@ -130,7 +130,7 @@ int flb_aws_imds_request_by_key(struct flb_aws_imds *ctx, const char *metadata_p
 
     /* Abort on version detection failure */
     if (imds_version == FLB_AWS_IMDS_VERSION_EVALUATE) {
-        // Exit gracefully allowing for retrys
+        /* Exit gracefully allowing for retrys */
         flb_warn("[imds] unable to evaluate IMDS version");
         return -1;
     }
@@ -148,7 +148,7 @@ int flb_aws_imds_request_by_key(struct flb_aws_imds *ctx, const char *metadata_p
         ec2_imds_client, FLB_HTTP_GET, metadata_path, NULL, 0, &token_header,
         (imds_version == FLB_AWS_IMDS_VERSION_1) ? 0 : 1);
     if (!c) {
-        // Exit gracefully allowing for retrys
+        /* Exit gracefully allowing for retrys */
         flb_warn("[imds] failed to retrieve metadata");
         return -1;
     }
falamatt$

Signed-off-by: Matthew Fala <[email protected]>

…truct Signed-off-by: Matthew Fala <[email protected]>

PettitWesley · 2021-09-30T00:56:47Z

@edsiper I merged #4114 so you don't need to backport this one

github-actions bot added the docs-required label Sep 13, 2021

matthewfala force-pushed the aws-imdsv2-r branch from 0b39316 to ad01ac8 Compare September 13, 2021 23:45

matthewfala commented Sep 14, 2021

View reviewed changes

src/aws/flb_aws_credentials_ec2.c Show resolved Hide resolved

PettitWesley reviewed Sep 14, 2021

View reviewed changes

matthewfala force-pushed the aws-imdsv2-r branch from ad01ac8 to 69ad9ea Compare September 14, 2021 02:11

matthewfala commented Sep 14, 2021

View reviewed changes

src/aws/flb_aws_imds.c Show resolved Hide resolved

src/aws/flb_aws_imds.c Show resolved Hide resolved

matthewfala changed the title ~~[aws] Include IMDSv2 support for EC2 instances~~ aws: Include IMDSv2 support for EC2 instances Sep 14, 2021

PettitWesley reviewed Sep 14, 2021

View reviewed changes

zhonghui12 reviewed Sep 14, 2021

View reviewed changes

matthewfala commented Sep 15, 2021

View reviewed changes

src/aws/flb_aws_credentials_ec2.c Outdated Show resolved Hide resolved

edsiper added AWS Issues with AWS plugins or experienced by users running on AWS and removed docs-required labels Sep 15, 2021

github-actions bot added the docs-required label Sep 18, 2021

matthewfala force-pushed the aws-imdsv2-r branch 4 times, most recently from 2a2f31f to c90a41f Compare September 20, 2021 22:03

zhonghui12 reviewed Sep 20, 2021

View reviewed changes

include/fluent-bit/aws/flb_aws_imds.h Outdated Show resolved Hide resolved

tests/internal/aws_client_mock.c Outdated Show resolved Hide resolved

matthewfala force-pushed the aws-imdsv2-r branch 3 times, most recently from fa844ec to d9b623c Compare September 21, 2021 01:46

zhonghui12 previously approved these changes Sep 21, 2021

View reviewed changes

matthewfala mentioned this pull request Sep 21, 2021

Credentials are not retrieved from AWS IMDSv2 when running on EC2 #2840

Closed