Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: GKE test failure Receiving flows from Hubble Relay: received unknown message: &{"source:OBSERVER_EVENTS_QUEUE #2806

Closed
michi-covalent opened this issue Sep 20, 2024 · 0 comments · Fixed by #2815
Closed

CI: GKE test failure Receiving flows from Hubble Relay: received unknown message: &{"source:OBSERVER_EVENTS_QUEUE #2806

michi-covalent opened this issue Sep 20, 2024 · 0 comments · Fixed by #2815
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests.

Comments

@michi-covalent
Copy link
Contributor

gke test started failing: https://github.com/cilium/cilium-cli/actions/workflows/gke.yaml?query=event%3Aschedule

looks like hubble is dropping events more frequently. for example: https://github.com/cilium/cilium-cli/actions/runs/10948047808/job/30398192114#step:13:1339

🟥 Receiving flows from Hubble Relay: received unknown message: &{"source:OBSERVER_EVENTS_QUEUE  num_events_lost:1  cpu:{}"}

which causes flow validation to fail 💭
@michi-covalent michi-covalent added area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests. labels Sep 20, 2024
This was referenced Sep 20, 2024
Merged
Merged
michi-covalent added a commit that referenced this issue Sep 27, 2024
@michi-covalent
github: Increase hubble-event-queue-size
1849d75 
Fixes: #2806

Signed-off-by: Michi Mutsuzaki <[email protected]>
@michi-covalent michi-covalent mentioned this issue Sep 27, 2024
Merged
michi-covalent added a commit that referenced this issue Oct 1, 2024
@michi-covalent
multicluster: Increase hubble-event-queue-size
da87c57 
Multicluster workflow started hitting the same issue as #2806 [^1].

[^1]: https://github.com/cilium/cilium-cli/actions/runs/11133691166/job/30941326039?pr=2822#step:17:1261

Signed-off-by: Michi Mutsuzaki <[email protected]>
@michi-covalent michi-covalent mentioned this issue Oct 1, 2024
Merged
michi-covalent added a commit that referenced this issue Oct 2, 2024
@michi-covalent
multicluster: Increase hubble-event-queue-size
e70c039 
Multicluster workflow started hitting the same issue as #2806 [^1].

[^1]: https://github.com/cilium/cilium-cli/actions/runs/11133691166/job/30941326039?pr=2822#step:17:1261

Signed-off-by: Michi Mutsuzaki <[email protected]>
michi-covalent added a commit that referenced this issue Oct 2, 2024
@michi-covalent
multicluster: Increase hubble-event-queue-size
3cde1b7 
Multicluster workflow started hitting the same issue as #2806 [^1].

[^1]: https://github.com/cilium/cilium-cli/actions/runs/11133691166/job/30941326039?pr=2822#step:17:1261

Signed-off-by: Michi Mutsuzaki <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

github: Increase hubble-event-queue-size cilium/cilium-cli
1 participant
@michi-covalent