Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to disable writing k8s events at workflow level #13042

Closed
tooptoop4 opened this issue May 13, 2024 · 3 comments · Fixed by #13746
Closed

Option to disable writing k8s events at workflow level #13042

tooptoop4 opened this issue May 13, 2024 · 3 comments · Fixed by #13746
Labels
area/controller Controller issues, panics solution/suggested A solution to the bug has been suggested. Someone needs to implement it. type/feature Feature request

Comments

@tooptoop4
Copy link
Contributor

Summary

Environment variable/config to stop writing k8s events

Use Cases

When would you use this? When i want to stop etcd filling up

These parts of the code are writing k8s events which end up being stored in etcd:

case wfv1.WorkflowRunning:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeNormal, "WorkflowRunning", "Workflow Running")
case wfv1.WorkflowSucceeded:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeNormal, "WorkflowSucceeded", "Workflow completed")
case wfv1.WorkflowFailed, wfv1.WorkflowError:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeWarning, "WorkflowFailed", message)

woc.recordNodePhaseChangeEvents(woc.orig.Status.Nodes, woc.wf.Status.Nodes)

With a lot of workflows being churned through these events add a lot of storage pressure to etcd. At the same time i have not found these events useful as i can always look at logs in the controller pod

@tooptoop4 tooptoop4 added the type/feature Feature request label May 13, 2024
@terrytangyuan
Copy link
Member

terrytangyuan commented May 13, 2024

# Whether or not to emit events on node completion. These can take a up a lot of space in
# k8s (typically etcd) resulting in errors when trying to create new events:
# "Unable to create audit event: etcdserver: mvcc: database space exceeded"
# This config item allows you to disable this.
# (since v2.9)
nodeEvents: |
enabled: true

@terrytangyuan

This comment was marked as duplicate.

@tooptoop4
Copy link
Contributor Author

tooptoop4 commented May 13, 2024

@terrytangyuan i already have that as false but it still sends events!! seems i am getting events at workflow level

@agilgur5 agilgur5 added the area/controller Controller issues, panics label May 13, 2024
tooptoop4 added a commit to tooptoop4/argo-workflows that referenced this issue May 30, 2024
@agilgur5 agilgur5 changed the title Option to disable writing k8s events Option to disable writing k8s events at workflow level Oct 7, 2024
@agilgur5 agilgur5 added the solution/suggested A solution to the bug has been suggested. Someone needs to implement it. label Oct 7, 2024
@agilgur5 agilgur5 added this to the v3.6.0 milestone Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/controller Controller issues, panics solution/suggested A solution to the bug has been suggested. Someone needs to implement it. type/feature Feature request
Projects
None yet
3 participants