Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[8.15](backport #40467) x-pack/metricbeat/module/gcp: Fix missing ECS labels when using exclude_labels: true #40596

Merged
merged 4 commits into from
Sep 18, 2024
Merged

[8.15](backport #40467) x-pack/metricbeat/module/gcp: Fix missing ECS labels when using exclude_labels: true #40596

merged 4 commits into from
Sep 18, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Aug 23, 2024 •
edited by zube bot
Loading

Proposed commit message

The GCP metrics metricset's exclude_labels: true option causes an issue in the groupTimeSeries function. The default metadata collector is created as a value, preventing the function from initializing metadataService with the correct time series data. This results in missing important labels like cloud.instance.id in the events. To fix this, metadataService should be initially set to nil, allowing initialization based on the time series parameter.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Compute Service Before Fix:

{
  "_index": ".ds-metricbeat-8.16.0-2024.08.12-000001",
  "_id": "zWdtRpEBVgxWUufvh8Eh",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2024-08-12T11:46:00.000Z",
    "metricset": {
      "name": "compute",
      "period": 60000
    },
    "gcp": {
      "labels": {},
      "compute": {
        "instance": {
          "uptime_total": {
            "sec": 965207
          },
          "memory": {
            "balloon": {
              "ram_size": {
                "value": 4115619840
              },
              "swap_in": {
                "bytes": 0
              },
              "swap_out": {
                "bytes": 0
              },
              "ram_used": {
                "value": 1657262080
              }
            }
          },
          "uptime": {
            "sec": 60.000000000116415
          },
          "cpu": {
            "usage": {
              "pct": 0.11856303568638395
            },
            "usage_time": {
              "sec": 7.113782141183037
            }
          },
          "network": {
            "egress": {
              "packets": {
                "count": 1217
              },
              "bytes": 883471
            },
            "ingress": {
              "packets": {
                "count": 980
              },
              "bytes": 230375
            }
          },
          "disk": {
            "read": {
              "bytes": 0
            },
            "write_ops_count": {
              "value": 1230
            },
            "read_ops_count": {
              "value": 0
            },
            "write": {
              "bytes": 12386800
            }
          }
        }
      }
    },
    "service": {
      "type": "gcp"
    },
    "agent": {
      "name": "EPINPUNW06A8",
      "type": "metricbeat",
      "version": "8.16.0",
      "ephemeral_id": "6882ed99-deee-4fae-a6ee-af617960c5ea",
      "id": "f49fbc8d-cc6a-4629-89b1-454f33f48d82"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {},
    "event": {
      "duration": 684338677,
      "dataset": "gcp.compute",
      "module": "gcp"
    }
  }
}

Compute Service After fix:

{
  "_index": ".ds-metricbeat-8.16.0-2024.08.12-000001",
  "_id": "uGc7RpEBVgxWUufvzb4e",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2024-08-12T10:52:00.000Z",
    "metricset": {
      "name": "compute",
      "period": 60000
    },
    "gcp": {
      "labels": {},
      "compute": {
        "instance": {
          "uptime_total": {
            "sec": 1980
          },
          "cpu": {
            "usage_time": {
              "sec": 0.13933221539459595
            },
            "usage": {
              "pct": 0.011611017949549662
            }
          },
          "uptime": {
            "sec": 60
          }
        }
      }
    },
    "service": {
      "type": "gcp"
    },
    "cloud": {
      "region": "us-central1",
      "instance": {
        "id": "8449630262774775859"
      },
      "account": {
        "id": "elastic-obs-integrations-dev",
        "name": "elastic-obs-integrations-dev"
      },
      "provider": "gcp",
      "availability_zone": "us-central1-a"
    },
    "host": {},
    "ecs": {
      "version": "8.0.0"
    },
    "agent": {
      "ephemeral_id": "c1f3c9a5-ff55-4d65-a586-e751805d1347",
      "id": "f49fbc8d-cc6a-4629-89b1-454f33f48d82",
      "name": "EPINPUNW06A8",
      "type": "metricbeat",
      "version": "8.16.0"
    },
    "event": {
      "duration": 706928630,
      "dataset": "gcp.compute",
      "module": "gcp"
    }
  }
}

Spanner Service Before Fix:

{
  "_index": ".ds-metricbeat-8.16.0-2024.08.12-000001",
  "_id": "CTKeRpEBGeZDOp8bEh6T",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2024-08-12T12:40:46.200Z",
    "event": {
      "duration": 921850497,
      "dataset": "gcp.metrics",
      "module": "gcp"
    },
    "metricset": {
      "name": "metrics",
      "period": 60000
    },
    "gcp": {
      "metrics": {
        "instance": {
          "leader_percentage_by_region": {
            "value": 1
          }
        }
      },
      "labels": {}
    },
    "service": {
      "type": "gcp"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {},
    "agent": {
      "type": "metricbeat",
      "version": "8.16.0",
      "ephemeral_id": "4227a7c2-941d-45b0-ab3d-0328c7a3ee97",
      "id": "f49fbc8d-cc6a-4629-89b1-454f33f48d82",
      "name": "EPINPUNW06A8"
    }
  }
}

Spanner Service After Fix:

{
  "_index": ".ds-metricbeat-8.16.0-2024.08.12-000001",
  "_id": "fDK1RpEBGeZDOp8bDx-J",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2024-08-12T13:06:00.000Z",
    "host": {},
    "agent": {
      "ephemeral_id": "00deca2b-9a95-42af-809c-730759f69cdf",
      "id": "f49fbc8d-cc6a-4629-89b1-454f33f48d82",
      "name": "EPINPUNW06A8",
      "type": "metricbeat",
      "version": "8.16.0"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "cloud": {
      "provider": "gcp",
      "instance": {
        "id": "spanner-1"
      },
      "account": {
        "id": "elastic-obs-integrations-dev",
        "name": "elastic-obs-integrations-dev"
      }
    },
    "event": {
      "dataset": "gcp.metrics",
      "module": "gcp",
      "duration": 955200812
    },
    "metricset": {
      "name": "metrics",
      "period": 60000
    },
    "gcp": {
      "metrics": {
        "instance": {
          "cpu": {
            "utilization_by_operation_type": {
              "value": 0
            }
          }
        }
      },
      "labels": {
        "metrics": {
          "priority": "medium",
          "database": "db-1",
          "category": "misc",
          "is_system": "false"
        },
        "resource": {
          "instance_config": "regional-europe-west1",
          "location": "europe-west1"
        }
      }
    },
    "service": {
      "type": "gcp"
    }
  }
}
This is an automatic backport of pull request #40467 done by [Mergify](https://mergify.com).

@Linu-Elias @mergify
x-pack/metricbeat/module/gcp: Fix missing ECS labels when using exclu…
1f45d71 
…de_labels: true (#40467)

* Issue fix

* Change.log

* resolved comments

* fix

(cherry picked from commit 973af49)
@mergify mergify bot requested a review from a team as a code owner August 23, 2024 06:57
@mergify mergify bot added the backport label Aug 23, 2024
@mergify mergify bot requested review from a team as code owners August 23, 2024 06:57
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 23, 2024
@botelastic
Copy link

botelastic bot commented Aug 23, 2024

This pull request doesn't have a Team:<team> label.

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Aug 26, 2024

This pull request has not been merged yet. Could you please review and merge it @Linu-Elias? 🙏

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Aug 27, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.15/pr-40467 upstream/mergify/bp/8.15/pr-40467
git merge upstream/8.15
git push upstream mergify/bp/8.15/pr-40467

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 2, 2024

This pull request has not been merged yet. Could you please review and merge it @Linu-Elias? 🙏

2 similar comments
@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 9, 2024

This pull request has not been merged yet. Could you please review and merge it @Linu-Elias? 🙏

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 16, 2024

This pull request has not been merged yet. Could you please review and merge it @Linu-Elias? 🙏

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 17, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.15/pr-40467 upstream/mergify/bp/8.15/pr-40467
git merge upstream/8.15
git push upstream mergify/bp/8.15/pr-40467

@Linu-Elias Linu-Elias merged commit 8c80b65 into 8.15 Sep 18, 2024
20 checks passed
@Linu-Elias Linu-Elias deleted the mergify/bp/8.15/pr-40467 branch September 18, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees

@Linu-Elias Linu-Elias

Labels
backport needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kaiyan-sheng @Linu-Elias