Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: 'apiVersion' in Kopf 1.37.x with Kubernetes 1.30.x #1144

Open
drexlar5 opened this issue Dec 5, 2024 Discussed in #1143 · 5 comments · May be fixed by #1151
Open

KeyError: 'apiVersion' in Kopf 1.37.x with Kubernetes 1.30.x #1144

drexlar5 opened this issue Dec 5, 2024 Discussed in #1143 · 5 comments · May be fixed by #1151
Labels
bug Something isn't working

Comments

@drexlar5
Copy link

drexlar5 commented Dec 5, 2024 •
edited by nolar
Loading

Discussed in #1143

Originally posted by drexlar5 December 3, 2024
Summary
I’m encountering a recurring KeyError: 'apiVersion' error in Kopf when it attempts to post events. This happens despite ensuring compatibility between Kopf, the Kubernetes Python client, and the Kubernetes server version.

Environment Details

  • Kopf Version: 1.37.3
  • Kubernetes Python Client Version: 30.5.0
  • Kubernetes Server Version: v1.30.5-gke.1443001
  • Kubernetes Client (kubectl) Version: v1.30.6-dispatcher

I’m running Kopf in a containerized environment, and the same issue is reproducible across different deployments.

Error Logs

Below are the logs I consistently see whenever that is a change to the custom applicationSet resource that kopf is watching:

> Traceback (most recent call last):
  File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/api/.venv/lib/python3.12/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/clients/events.py", line 35, in post_event
    if ref['apiVersion'] == 'v1' and ref['kind'] == 'Event':
       ~~~^^^^^^^^^^^^^^
KeyError: 'apiVersion'
Traceback (most recent call last):
  File "/api/.venv/bin/kopf", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/kopf/cli.py", line 60, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/kopf/cli.py", line 109, in run
    return running.run(
           ^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 81, in run
    asyncio.run(coro)
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 138, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 419, in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 226, in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
    ^^^^^^^^^^^^^
  File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
    await coro
  File "/api/.venv/lib/python3.12/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/clients/events.py", line 35, in post_event
    if ref['apiVersion'] == 'v1' and ref['kind'] == 'Event':
       ~~~^^^^^^^^^^^^^^
KeyError: 'apiVersion'

This error occurs during the posting of Kubernetes events, specifically when Kopf tries to access the apiVersion field of the ref object.

What I have tried

  • Updated Versions for Compatibility:

    • Confirmed compatibility between Kopf, the Kubernetes Python client, and the Kubernetes server:
      • Kopf: 1.37.3
      • Kubernetes Python Client: 30.5.0
      • Kubernetes Server: v1.30.5
    • Upgraded Kopf and the Kubernetes Python client to align with Kubernetes server 1.30.x. However, the issue persists.

  • Tested on Local KIND Environment:

    • Deployed the same operator on a local KIND cluster running Kubernetes server v1.31.x. The error does not occur in this setup, which suggests the issue might be specific to GKE or how it handles events.

Expected Behavior

Kopf should gracefully handle situations where the apiVersion field is missing from the ref object, or ensure that it is always populated during event posting.

Additional Notes

  • The issue appears more prominently in GKE (Google Kubernetes Engine) than in KIND.
  • This might be related to GKE-specific configurations or event structures.
@nolar nolar linked a pull request Dec 13, 2024 that will close this issue
Draft
@nolar
Copy link
Owner

nolar commented Dec 13, 2024

Hello. Thanks for reporting. A preliminary fix is in #1151, but I would like to request some clarification.

To which operator does this applicationSet belong? Is it Argo (the only thing I could find)?

I need this information in order to reproduce the issue locally, and to decide what to do in this case: either accept the ref "as is" without apiVerison in it (that would be strange and suspicious); or assume the ref is broken/unusable, thus skip the event posting entirely.

@nolar nolar added the bug Something isn't working label Dec 13, 2024
@NGTOne
Copy link

NGTOne commented Dec 16, 2024

Colleague of the original poster here. Yes, ApplicationSet belongs to ArgoCD; in our installation we have Argo 2.12.3 installed using chart version 7.6.2.

@drexlar5
Copy link
Author

drexlar5 commented Jan 6, 2025

@nolar any progress on this?

@raaymax
Copy link

raaymax commented Jan 16, 2025 •
edited
Loading

I have the same error when trying this example https://github.com/nolar/kopf/tree/main/examples/04-events

Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.5

python = "^3.12"
kopf = "^1.37.4"

@raaymax
Copy link

raaymax commented Jan 17, 2025

I applied #1151 locally and it seems to be working now without crashing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle event refs with no apiVersion/kind fields at all nolar/kopf
4 participants
@nolar @raaymax @NGTOne @drexlar5