Metricbeat AWS module S3 bucket misinformation

[aschifman]

When setting up Metricbeat with the AWS module, to collect S3 bucket information, there are inconsistencies and misrepresented data.

There are two issues:

1. Not all buckets are displayed in Kibana. In our AWS lab, we have 361 buckets, yet only 60 are actually able to be displayed in the S3 Overview dashboard.

Screen shot take from AWS S3 Management Console:

Yet, these do not show in Kibana.

2. Secondly, every bucket that is displayed, show as having zero objects, but in AWS console, most all have multiple objects.

Screenshot of two bucket sizes and yet, the second screenshot highlights them not showing in Kibana:

The buckets with Cloudwatch metrics do not show as having any objects in Kibana.

For confirmed bugs, please report:

• Version: Cloud deployment running 7.9.3 with Metricbeat 7.9.2 and later upgraded to 7.10 still with same problem
• Operating System: Mac
• Steps to Reproduce: Standard Metricbeat setup has been seen in two different environments, one using Elastic AWS lab, and the other with AWS using their own lab systems.

My AWS counterpart experiences the following...

He has six S3 buckets, yet only two show in Kibana. All six have metrics in Cloudwatch and many objects, yet the two buckets that Kibana sees shows as zero.

Thanks

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

Hi @aschifman, sorry for the delay. I would like to figure out one thing first: are we missing collecting data from some specific buckets or just not displaying some of them in Kibana dashboard?

1. In TSVB visualizations, there is a limit on how many buckets can be displayed we set in order to not fill one chart with too much data:

2. For the buckets showing no object, could you take a look at the specific field aws.s3_daily_storage.number_of_objects in Kibana Discover to see what actual value it's collected? My guess is, it is a bug in the visualization. If that's the case, I can create a PR to migrate these visualizations from using TSVB to lens, which should help.

[aschifman]

Hey @kaiyan-sheng
There are not any buckets showing objects which is why this whole things is just odd...names appear, though some are missing.

The second screenshot above is taken after adjusting the Top setting in visualizations...trying it at 50, then to 500, but no difference.

The aws.s3_daily_storage.number_of_objects field is not available, so you are on to something...again strange because some information for S3 is making it in.

I do adjust the time filter from 15 minutes to the past one year in case that is something on the mind...I had also deleted all metricbeat index patterns, though I attempted this with a 7.9 and 7.10 deployment as well.

Discover aws metrics missing the aws.s3 fields:

I am also attaching my aws.yml config file.
aws.yml.txt

All metrics use the same IAM role, which has the following permissions...basically full/admin on everything.

The role:

The policy permissions:

I can show you this setup if you like...let me know and thanks!

[aschifman]

I have found something very interesting...I am not sure what to make of this, but this has to do with the visualizations as you were already leaning towards. It may not be a bug but a default setting behavior that should be adjusted.

If you change the Data timerange mode within the S3 Daily Storage Number of Objects [Metricbeat AWS] visualization - setting for Panel options - from the default Last value to Entire time range things show up.

It is then a matter of saving that change and updating the dashboard...the dashboard did not automatically update. I had to edit the dashboard object, and then edit the visualization from there...saving then shows what was missing.

Now I do not know if this is correct to change, but does point to something. I still do not understand, however, why there is no aws.s3 metrics to filter on within Discover.

[kaiyan-sheng]

Thanks for looking into this! Sorry what do you mean by no aws.s3 metrics? There will only be metrics with name aws. s3_daily_storage.* and aws.s3_request.* because there is no metricset name s3.

[aschifman]

I mean there is no aws.s3* anything...If you see the screenshot I took above, you can see the metrics go from aws.natgateway.* to aws.tags.*.

[kaiyan-sheng]

Gotcha! Thanks! What's the time range did you set for checking metrics? aws. s3_daily_storage.* reports only once per day so maybe try to change the time range in Kibana Discover to 1 day :)

[aschifman]

@kaiyan-sheng I set it at different levels...15 minutes all the way to over a year...no difference, though when I did go out a year, things like RDS metrics then showed up.

[kaiyan-sheng]

hmm sorry let me make sure I understand this correctly. So there is no metrics from s3_daily_storage.* or aws.s3_request.* at all?

[aschifman]

Not showing in Discover...

On Fri, Dec 4, 2020 at 20:20 kaiyan-sheng ***@***.***> wrote: hmm sorry let me make sure I understand this correctly. So there is no metrics from s3_daily_storage.* or aws.s3_request.* at all? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22582 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APYSUHG7UUQGBVUL7HAV7B3STGDGTANCNFSM4TU4SBBQ> .

-- ------ Aaron M. Schifman Partner Integration Engineer [email protected] Search. Observe. Protect.

[kaiyan-sheng]

Let me just put some notes here for what @aschifman and I talked about offline:

1. s3_daily_storage metricset collects metrics from AWS CloudWatch so if there are data missing in CloudWatch, s3_daily_storage will not be able to get them.
2. Filter created by Controls visualization seems to have a hard limit on how many values can be displayed on the drop down menu (10 is the value we see). We are waiting on confirmation from Kibana team to figure out what to do here.

3. I do believe we should migrate this dashboard to use lens at some point.

[kaiyan-sheng]

@aschifman Reponse from kibana team about the s3 bucket names in the drop down menu:

This depends on the "dynamic options" setting. if you're having it on we'll load the top 10 and when the user types we'll refine the search results. If dynamic options is off, we're not reloading more data when the user types, therefore you'll get a size control where you can say how many entries should be loaded.

I tested it with Dynamic Options off and it does show all s3 bucket names. But it requires a hardcoded Size value:

The problem is we have no idea what this hard code size value should be when creating the visualization and it varies in different AWS environments too. So my suggestion is to keep the Dynamic Options on.