How to periodly poll something?

[kopf-archiver]

An issue by Kiddinglife at 2019-06-28 05:30:49+00:00
Original URL: zalando-incubator/kopf#122
 

Hi folks,

for example,
kubectl autoscale deployment xxx --cpu-percent=50 --min=1 --max=10
but autoscale only supports k8s-built-in api resources like replicaset, deployment and statefulset.

I need my controller to periodly poll the premetheus server for some metrics so that my controller can automatically scale up or scale down pods.

thanks.

---

Commented by nolar at 2019-07-01 11:25:29+00:00
 

Kiddinglife Sorry, I didn't get the part with replicasets.

But for polling something in Kopf you can use the @kopf.on.resume handlers to start a background thread or an asyncio task:

• https://kopf.readthedocs.io/en/latest/handlers/?highlight=resum#cause-handlers

It is better if resuming is used on combination with @kopf.on.create: either the object was created while the operator is up (creation), or the operator was started while the object exists (resuming) — it is usually either-either.

@kopf.on.resume('zalando.org', 'v1', 'kopfexamples')
@kopf.on.create('zalando.org', 'v1', 'kopfexamples')
def start_poller(**kwargs):
    threading.Thread(target=poller, kwargs=kwargs).start()

def poller(**kwargs):
    while shouldstop:
        check_it()
        change_it()
        time.sleep(60)

Thread deduplication can be additionally secured with uid kwarg and by keeping a dict{uid=>thread} structure in the module:

threads = {}

@kopf.on.resume('zalando.org', 'v1', 'kopfexamples')
@kopf.on.create('zalando.org', 'v1', 'kopfexamples')
def start_poller(**kwargs):
    uid = kwargs['uid']
    if uid not in threads or not threads[uid].isAlive():
        shouldstop = threading.Event()
        thread = threading.Thread(target=poller, args=(shouldstop,), kwargs=kwargs)
        threads[uid] = (thread, shouldstop)
        thread.start()
    else:
        pass  # already running

@kopf.on.delete('zalando.org', 'v1', 'kopfexamples')
def stop_polling(uid, **kwargs):
    thread, shouldstop = threads[uid]
    shouldstop.set()
    thread.join()  # can take ~60 seconds
    del threads[uid]

def poller(shouldstop, **kwargs):
    while not shouldstop.is_set():
        ......
        time.sleep(60)

---

Commented by eshepelyuk at 2020-05-17 03:36:23+00:00
 

nolar is there any advantage of using dedicated threads for polling ? Versus using asyncio based libraries, like https://github.com/gawel/aiocron ?

---

Commented by nolar at 2020-05-26 20:21:09+00:00
 

eshepelyuk Cron-based functions can be sufficient for some cases. But they are unaware of Kubernetes resources in any form.

First of all, you might have multiple resource to serve, parametrised with their own values in spec. This can be probably solved by aiocron's "objects" in memo. Which is not so much different from having your own thread/task already.

Second, the polling is supposed to stop once the resource is gone. You would need to explicitly stop that aiocron object. Additionally, error handling and retrying and backoff intervals and retry-limiting should be implemented somehow.

I believe, one way or another, it is doable on a case-by-case basis. You can also start and cancel an asyncio task from creation+resuming or deletion handlers accordingly — and that will be enough. The difference is only in the amount of boilerplate code needed for this extra non-domain logic.

Disclaimer: I never used aiocron before, so I just assume what it is — based on its README and my own general expectation what a library with that name can do.

---

On the other hand, Kopf's daemons & timers are fully Kubernetes-specialised and resource-aware: they run individually for each resource (and access its spec, metadata, etc); and they stop being invoked when the resource is deleted; and all the Kopf-native retrying logic is there.

Kopf does not have anything with cron-like schedules right now (e.g. */5 8-20 * * *), only the basic time intervals. But this can be added as a new feature, once a good cron-spec parser is found (aiocron/croniter seem promising) — now, when we have a concept of time at all, not only event-driven reactions, this is doable.

---

Commented by eshepelyuk at 2020-05-27 07:27:30+00:00
 

nolar the entire question was about different topic

1. Since all kopf framework is based on asyncio, but the example you've given is using threads, the quesion pops up immediately - is there any advantage of using threads and not asyncio for solving the periodical poll task ?
   There were not intention neither to ask about any particular library, nor about cron expressions itself, etc

2. Also, the author of the task asked about polling prometheus server, so it's not even about thread \ task management bound to k8s resources.

---

Commented by nolar at 2020-05-27 08:45:46+00:00
 

eshepelyuk Ah, I see. Sorry for misunderstanding.

There is no big advantage of using threads over asyncio tasks. Actually, it is a disadvantage — threads are non-stoppable from outside, while asyncio tasks are cancellable.

However, as I have learned the hard way, asyncio, asynchronous programming, and cooperative concurrency are difficult to understand for many people. But they know threads, because it is an old and proven tool, explained in many books, and considered a must-have for software engineering skillset. So, by default, I usually explain ideas in plain-and-simple tech tools, in order to not bring complicated side-topics like asyncio to the table.

Not to mention that if people start using synchronous API client libraries in asynchronous tasks, this will be a disaster (in runtime). And most of the K8s API client libraries are still synchronous, especially the official one.

A side-note: for the same reason, Kopf supports both sync and async handlers, with usual def handlers being "easy", while async def being "native", both as 1st-class citizens. Back in July 2019, Kopf also used synchronous API client libraries under the hood, so it would be hard to say that Kopf was fully async inside (not the case anymore).

Nowadays, Kopf is fully async inside indeed. And there are daemons! I will come back to this question once 0.27 is released in a few days.

---

Commented by eshepelyuk at 2020-05-27 14:20:32+00:00
 

nolar thnx

your intention of explaning via threads is clear now.

one more question to continue - daemons and timers are bound to some K8S resource now and are active only when that tracked resource exists.
As far as I understand request of the initiator - he needs a sort of daemon \ times to be run unbounded from any K8S resource.

I have a vision of how to overcome this - i.e. create a didicated CRD for this prometheus polling and associate KOPF timer or daemon to that resource.
But this approach should be described in docs, i think.

---

Commented by nolar at 2020-05-27 16:34:32+00:00
 

eshepelyuk For me, it is not clear if the requested polling is global, or per-resource. I would assume that the intention was to poll the metrics of specific pods of a specific app/deployment, in order to scale it — and repeat it for each of the apps/deployments. So, it would be resource-bound.

But if that is a global single-instance polling, then start a thread or a task explicitly, and poll from there. You can do it from @kopf.on.startup/@kopf.on.cleanup handlers (they didn't exit in July 2019), or from the core app. The mentioned aiocron seems like a good tool for this too. Kopf is not needed for such time-scheduled polling, and is not related to this kind of applications; having a global cluster object to run a single thread/task seems redundant.

---

Commented by eshepelyuk at 2020-05-27 17:59:59+00:00
 

nolar motivation is clearly described, reasonable and complete :)
thanks for your time

[nolar]

I assume this question is answered.

TL;DR (modernised for 2021): Use @kopf.timer or manage your own threads/tasks, which can be stored in memo (per-resource — since long time ago; operator's global — since kopf>=1.30.0.dev1).