Daemons & timers for background processing and reconciliation

[nolar]

What do these changes do?

Add background daemons & timers for resources.

Description

Previously, the handlers were event-driven — i.e. when something changes on the resource. If nothing changes, no handlers are called. To have reconciliation or other activities happening during the lifetime of a resource, the developers had to do their own task/thread orchestration.

With this PR, the task/thread orchestration is generalised to a new framework feature to perform background reconciliation with external systems, locally managed applications, or even local children K8s resources not the event-driven way:

import kopf

@kopf.daemon('zalando.org', 'v1', 'kopfexamples', initial_backoff=5.0, cancellation_backoff=10.0)
async def background_async(spec, logger, **_):
    while True:  # async handlers will be cancelled: instantly or with a graceful delay.
        logger.info(f"Ping from an async daemon: field={spec['field']!r}")
        await asyncio.sleep(2.5)

@kopf.timer('zalando.org', 'v1', 'kopfexamples', idle=15, interval=2, sharp=True)
def every_few_seconds_sync(spec, logger, **_):
    logger.info(f"Ping from a sync timer: field={spec['field']!r}")

The daemons run as long as the resource exists. They are stopped/cancelled when the resource is deleted (or when the operator exits/restarts). The daemons can be automatically restarted or left dead if exited (restart_backoff).

The timers are also triggered as long as the resource exists, but by schedule. They can be postponed by few seconds since the last change (idle), or since the resource creation / operator startup (initial_backoff), can have sharp or soft intervals (see the docs), etc.

For more examples & options, see /examples/ folder and the doc updates in this PR/branch.

Issues/PRs

Issues: closes #19

Related: #150 #271 #317 #122

Type of changes

• New feature (non-breaking change which adds functionality)

Checklist

• The code addresses only the mentioned problem, and this problem only
• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes
• If you provide code modification, please add yourself to CONTRIBUTORS.txt

---

WARNING: This PR is a preview. It is runnable and usable, but few minor things are not finished yet:

• Tests
  • For the stopper synchronization primitive.
  • For daemon existing reason signalling (DaemonStopperReason).
  • For daemon & timer decorators.
  • For async daemon cancellation.
  • For sync daemon cancellation.
  • For timer guarding task cancellation.
• A sync-daemon did not exit on operator stopping in a try-run (it did exit before, something is broken? — check & fix.).

[zincr]

🤖 zincr found 0 problems , 1 warning

ℹ️ Large Commits
✅ Approvals
✅ Specification
✅ Dependency Licensing

Details on how to resolve are provided below

---

Large Commits

Checks all commits for large additions to a single file. Large commits should be reviewed more carefully for potential copyright and licensing issues

This file contains a substantial change, please review to determine if the change comes from an external source and if there are any copyright or licensing issues to be aware of

• ℹ️ docs/daemons.rst had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ docs/timers.rst had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ kopf/reactor/daemons.py had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ kopf/reactor/processing.py had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ kopf/structs/primitives.py had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ tests/handling/daemons/conftest.py had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
• ℹ️ tests/handling/daemons/test_daemon_errors.py had +100 lines of code changed in a single commit
  Please review this file to determine the source of this change
   

[lgtm-com]

This pull request introduces 2 alerts when merging 6505cb4 into de23abe - view on LGTM.com

new alerts:

• 2 for Module imports itself

[nolar]

The tests are rudimentary, only for the smoke-testing of daemons & timers. This PR took already too much time, and is too big.

The low-level primitives are tested indirectly through the existing tests, and manually by running examples with Minikube (including the examples 14 & 15 for daemons & timers).

The framework itself will be tested for backward compatibility by deployed a release candidate to our own clusters and utilising the new features (incl. daemons & timers).

More detailed tests for all little aspects will be added later as separate PRs — with lower priority (time-wise).

[mnarodovitch]

This is not readable. Maybe from unittest.mock import call and

assert ...call_args_list == [call(*args, **kwargs)]

is better. Applies to the other stuff also. In the current situation, there might be obscure out of index exceptions if the test breaks.

[nolar]

Breaking the test in that case is intended. The first index is "secured" by a preceding check for the number of calls.

But you are right, making tests more readable is a good idea — but should be done for all the tests. Now, the tests in overall are in a clumsy state, grown over the year, and they were never refactored. Some of them are duplicating each other, some aspects of the code are just missing.

I've created #338 to collect the ideas on the tests, and added this one there.