help: force secrets upserting eventhough POLLER_INTERVAL_MILLISECONDS is set to higher interval

[avistech]

Hi guys, we have noticed that if we set POLLER_INTERVAL_MILLISECONDS=86400000 (24 hours) external secrets are not pulled from aws secrets manager and upserted to secrets immediately after deployment.

Use Case: Our secrets are db connection string and passwords which are not expected to change very often so we wanted to have poller interval once a day. we havent found way to force externalsecrets deployment to poll and upsert secrets immediately helm upgrade. it waits for 24 hours for even first time polling. please let us know if you need more details. thanks.

[jeffpearce]

HI @avistech - this makes sense. It's probably pretty easy to fix, let us know if you'd like to submit a PR to fix it :-)

[avistech]

@jeffpearce thanks for quick response. May i know if it is good idea to fetch the secrets immediately for the first time and wait for POLLER_INTERVAL for the subsequent pulls (or) control the behaviour with a configuration parameter like env.POLL_SECRETS_ON_START=TRUE. i would appreciate your inputs. thanks.

[jeffpearce]

I prefer fetching it the first time - adding another env variable just complicates the code, and I can’t think of a good reason not to always fetch it

[avistech]

@jeffpearce thanks again for prompt response. I just went through the code and noticed that adding forcePoll param to 'true' for ADDED case in daemon.js would suffice the requirement. Please let me know if this is good enough. thanks

    case 'ADDED': {
      this._addPoller(descriptor, true)
      break
    }

[jeffpearce]

Yes, that seems like the right fix

[Flydiverny]

Wasn't sure whether this was the best way to go or not (in #107), as this will upsert all secrets whenever the watch stream is disconnected or pod restarted or so, which might be fine. But then code can be simplified again as now the forcePull flag is unnecessary as its always set to true in all addPoller calls.

After #107 was merged it would however create the secrets right away if it doesn't find them as well as upsert as soon as the external secrets are modified. Might not really cover this use case, but I don't really see why one would need a forced upsert when kubernetes-external-secrets is updated, sure the poller intervall would be reset if the pod is restarted?

[jeffpearce]

@avistech, thoughts?

[avistech]

Hi, we are not aware of #107 changes but seems to be good for our use case. As we know, AWS pricing model is based on number of API calls per secret stored in secrets manager so we wanted the secrets to be pulled only once in a day or so. we have multiple microservices and if every service starts polling every 10 seconds its going to add costs to project because our secrets don't change so often. As @Flydiverny mentioned, we are good if secrets are pulled right away if controller doesn't find them and wait for POLLER_INTERVAL for next poll. thanks.

[Flydiverny]

I think we need to find some kind of improvement to implement as well though.
As say we have the 24h poll intervall, and the watch stream is disconnected within 24 hours everytime, we would never cause a new poll as the intervall would be reset everytime the watch stream is disconnected.

[Flydiverny]

Made a draft for a possible solution in #133 :)

[avistech]

Thanks. we will wait for next release to upgrade our helm chart. :)