This repository has been archived by the owner on Sep 17, 2024. It is now read-only.
chore: update branch to 6.8. code #584
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪Test errors
Expand to view the tests failures> Show only the first 10 test failures
|
|
@jsoriano I don't think we can continue with this maintenance branch, as it seems the Beats project did not backport the I envision two options:
wdyt? |
I don't think that copying and adjusting the Maybe a compromise solution is to add the adjusted |
You mean from e2e codebase? I see it totally doable: cloning Beats in a specific branch (7.x instead of 6.8) and use the compose files in there. See ee8e79f#diff-2577dc7dd20ba506a083636a88fc3a51d1b3e7b60bc642adcd1b94f882441711R86. It would be replacing the branch |
v1v
approved these changes
Jan 13, 2021
mdelapenya
commented
Jan 13, 2021
mdelapenya
commented
Jan 14, 2021
cachedout
approved these changes
Jan 14, 2021
|
@jsoriano the errors that are shown in the CI are related to not being able to create the index for metricbeat:
Do you think we need to adapt the code in the tests to define the index name in another manner? I can see this log in metricbeat container logs: where we expect the index name to be |
|
@mdelapenya is this happening with all integrations, or only with Ceph? We would need to review the index settings we are using, there can be differences between 6.x and 7.x. Since 7.0 ILM is enabled by default, so indexes created can be different. Maybe the issue is related to that. If this is the case I can think on two possible solutions:
I would prefer first option, as this is the default in 6.8. If this is only happening with some integrations, the issue could be that without ILM indexes are not created till an event is sent. Modules like Ceph can take some time till some event is generated, what could delay the creation of the index. |
It happens to any integration. |
An example output:
{
"took": 25,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 181,
"max_score": 1.0,
"hits": [
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "As4VF3cB7TFTnNGpbCcl",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T20:00:14.909Z",
"metricset": {
"module": "system",
"rtt": 219,
"name": "memory"
},
"event": {
"dataset": "system.memory",
"duration": 219000
},
"system": {
"memory": {
"actual": {
"free": 6021242880,
"used": {
"pct": 0.279,
"bytes": 2330558464
}
},
"swap": {
"free": 2147479552,
"total": 2147479552,
"used": {
"bytes": 0,
"pct": 0
}
},
"hugepages": {
"free": 0,
"reserved": 0,
"surplus": 0,
"default_size": 2097152,
"total": 0,
"used": {
"pct": 0,
"bytes": 0
}
},
"total": 8351801344,
"used": {
"bytes": 6198669312,
"pct": 0.7422
},
"free": 2153132032
}
},
"beat": {
"hostname": "02974131b615",
"version": "6.8.14",
"name": "02974131b615"
},
"host": {
"name": "02974131b615"
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "A84VF3cB7TFTnNGpbCcl",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T20:00:14.909Z",
"system": {
"cpu": {
"nice": {
"pct": 0
},
"user": {
"pct": 0.2088
},
"irq": {
"pct": 0
},
"steal": {
"pct": 0
},
"total": {
"pct": 0.3494
},
"cores": 4,
"system": {
"pct": 0.1406
},
"idle": {
"pct": 3.6485
},
"iowait": {
"pct": 0.0021
},
"softirq": {
"pct": 0
}
}
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
},
"host": {
"name": "02974131b615"
},
"event": {
"dataset": "system.cpu",
"duration": 259300
},
"metricset": {
"module": "system",
"rtt": 259,
"name": "cpu"
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "BM4VF3cB7TFTnNGpbCcl",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T20:00:14.910Z",
"metricset": {
"rtt": 329,
"name": "network",
"module": "system"
},
"event": {
"duration": 329900,
"dataset": "system.network"
},
"system": {
"network": {
"name": "ip6tnl0",
"in": {
"errors": 0,
"dropped": 0,
"bytes": 0,
"packets": 0
},
"out": {
"errors": 0,
"dropped": 0,
"packets": 0,
"bytes": 0
}
}
},
"host": {
"name": "02974131b615"
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "js4UF3cB7TFTnNGpgiYP",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:14.975Z",
"event": {
"dataset": "system.load",
"duration": 111200
},
"system": {
"load": {
"15": 0.4,
"norm": {
"15": 0.1,
"1": 0.18,
"5": 0.165
},
"cores": 4,
"1": 0.72,
"5": 0.66
}
},
"metricset": {
"name": "load",
"module": "system",
"rtt": 111
},
"beat": {
"hostname": "02974131b615",
"version": "6.8.14",
"name": "02974131b615"
},
"host": {
"name": "02974131b615"
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "j84UF3cB7TFTnNGpgiYP",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:14.978Z",
"system": {
"network": {
"name": "ip6tnl0",
"in": {
"packets": 0,
"errors": 0,
"dropped": 0,
"bytes": 0
},
"out": {
"errors": 0,
"dropped": 0,
"packets": 0,
"bytes": 0
}
}
},
"host": {
"name": "02974131b615"
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
},
"metricset": {
"module": "system",
"rtt": 198,
"name": "network"
},
"event": {
"dataset": "system.network",
"duration": 198800
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "kc4UF3cB7TFTnNGpgiYP",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:14.978Z",
"metricset": {
"name": "network",
"module": "system",
"rtt": 255
},
"event": {
"duration": 255300,
"dataset": "system.network"
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
},
"host": {
"name": "02974131b615"
},
"system": {
"network": {
"out": {
"bytes": 0,
"errors": 0,
"dropped": 0,
"packets": 0
},
"name": "tunl0",
"in": {
"packets": 0,
"errors": 0,
"dropped": 0,
"bytes": 0
}
}
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "lM4UF3cB7TFTnNGpgiYQ",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:14.978Z",
"event": {
"duration": 620100,
"dataset": "system.memory"
},
"system": {
"memory": {
"actual": {
"free": 6026010624,
"used": {
"pct": 0.2785,
"bytes": 2325790720
}
},
"swap": {
"total": 2147479552,
"used": {
"bytes": 0,
"pct": 0
},
"free": 2147479552
},
"hugepages": {
"surplus": 0,
"default_size": 2097152,
"total": 0,
"used": {
"pct": 0,
"bytes": 0
},
"free": 0,
"reserved": 0
},
"total": 8351801344,
"used": {
"bytes": 6188392448,
"pct": 0.741
},
"free": 2163408896
}
},
"metricset": {
"rtt": 620,
"name": "memory",
"module": "system"
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
},
"host": {
"name": "02974131b615"
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "oc4UF3cB7TFTnNGpqSYe",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:24.975Z",
"beat": {
"hostname": "02974131b615",
"version": "6.8.14",
"name": "02974131b615"
},
"host": {
"name": "02974131b615"
},
"metricset": {
"name": "load",
"module": "system",
"rtt": 261
},
"event": {
"duration": 261600,
"dataset": "system.load"
},
"system": {
"load": {
"15": 0.39,
"norm": {
"1": 0.1525,
"5": 0.16,
"15": 0.0975
},
"cores": 4,
"1": 0.61,
"5": 0.64
}
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "os4UF3cB7TFTnNGpqSYe",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:24.978Z",
"event": {
"dataset": "system.network",
"duration": 382500
},
"system": {
"network": {
"in": {
"packets": 0,
"errors": 0,
"dropped": 0,
"bytes": 0
},
"out": {
"errors": 0,
"dropped": 0,
"packets": 0,
"bytes": 0
},
"name": "ip6tnl0"
}
},
"beat": {
"hostname": "02974131b615",
"version": "6.8.14",
"name": "02974131b615"
},
"host": {
"name": "02974131b615"
},
"metricset": {
"name": "network",
"module": "system",
"rtt": 382
}
}
},
{
"_index": "metricbeat-6.8-ceph-master-97985eb-nautilus-centos-7-x86_64-oky6rooz-2021.01.18-000001",
"_type": "doc",
"_id": "ps4UF3cB7TFTnNGpqSYe",
"_score": 1.0,
"_source": {
"@timestamp": "2021-01-18T19:59:24.978Z",
"metricset": {
"name": "memory",
"module": "system",
"rtt": 459
},
"event": {
"duration": 459500,
"dataset": "system.memory"
},
"system": {
"memory": {
"swap": {
"used": {
"bytes": 0,
"pct": 0
},
"free": 2147479552,
"total": 2147479552
},
"hugepages": {
"total": 0,
"used": {
"bytes": 0,
"pct": 0
},
"free": 0,
"reserved": 0,
"surplus": 0,
"default_size": 2097152
},
"total": 8351801344,
"used": {
"pct": 0.7422,
"bytes": 6198444032
},
"free": 2153357312,
"actual": {
"free": 6016856064,
"used": {
"pct": 0.2796,
"bytes": 2334945280
}
}
}
},
"beat": {
"name": "02974131b615",
"hostname": "02974131b615",
"version": "6.8.14"
},
"host": {
"name": "02974131b615"
}
}
}
]
}
} |
mdelapenya
commented
Jan 18, 2021
| @@ -444,7 +444,7 @@ func (mts *MetricbeatTestSuite) thereAreEventsInTheIndex() error { | |||
| "must": []map[string]interface{}{ | |||
| { | |||
| "match": map[string]interface{}{ | |||
| "event.module": mts.Query.EventModule, | |||
| "event.dataset": "system.network", | |||
There was a problem hiding this comment.
@jsoriano this is a system metricseat, as I was not able to fetch any module-specific one.
There was a problem hiding this comment.
Does it mean that we cannot test modules? Or this is a temporary fix?
There was a problem hiding this comment.
I'm not sure how to retrieve the module specific metrics. The above JSON response represents a search cURL on the index, and I do not see any event related to the module. Could it be because the 6.8 branch does not understand 7.7 config file? We are fetching 7.7 branch's config because it's the first version including supported_versions files.
As a workaround, we could fetch config files from 6.8 branch on test time, instead of when preparing the test runtime.
There was a problem hiding this comment.
Could it be because the 6.8 branch does not understand 7.7 config file?
Do you see activity from modules in logs?
It could happen in some modules that some option has been added in 7.7 that is not available in 6.8, but in general I think that configuration files should be quite similar.
As a workaround, we could fetch config files from 6.8 branch on test time, instead of when preparing the test runtime.
Yes, I think we should use configuration from 6.8 to test this version.
There was a problem hiding this comment.
I checked both, 6.8 and 7.7, and they are equals
mdelapenya
commented
Jan 18, 2021
| @@ -480,7 +480,7 @@ func (mts *MetricbeatTestSuite) thereAreNoErrorsInTheIndex() error { | |||
| "must": []map[string]interface{}{ | |||
| { | |||
| "match": map[string]interface{}{ | |||
| "event.module": mts.Query.EventModule, | |||
| "event.dataset": "system.network", | |||
mdelapenya
commented
Jan 26, 2021
e2e/Makefile
Outdated
| @@ -85,7 +85,7 @@ notice: | |||
|
|
|||
| .PHONY: sync-integrations | |||
| sync-integrations: | |||
| OP_LOG_LEVEL=${LOG_LEVEL} ./op sync integrations --delete | |||
| OP_LOG_LEVEL=${LOG_LEVEL} ./op sync integrations --delete --remote "elastic:7.7" | |||
There was a problem hiding this comment.
This is the first version having the majority of the integrations the supported-versions.yml file.
8 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Why is it important?
We want to run e2e tests for metricbeat's integrations
Checklist
make noticein the proper directory)Author's Checklist
Related issues
Use cases
Screenshots
Logs
Follow-ups