Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Image metadata lookup and automation for push hooks #835

Merged
merged 4 commits into from
Nov 15, 2017

Conversation

squaremo
Copy link
Member

@squaremo squaremo commented Nov 13, 2017

  1. To process image push hooks, we need to be able to pre-empt the cache
    warming loop.

This commit introduces a priority channel, as a field of the warmer,
and changes the loop such that it looks up image data from the
priority channel in preference to refreshing image data from the
cluster.

It is possible to be notified about an image for which the loop does
not have credentials. This means the priority lookup will be skipped,
and is otherwise harmless. The credentials are only requested at the
start, and when the regular refresh is run. A refinement which
improves on this would be to plumb through a means of getting a
specific image's credentials from the cluster.

  1. Add a "new image notification" to the warmer, and plumb it through to the daemon so that it prompts the daemon to do an automation run.

This is done as a callback, and the daemon method (newly exported) AskForImagePoll supplied. This is the simplest way, given that we want to keep the image polling (and other things) running on a ticker as well.

The code that attempts to figure out if there's a new image comes with a couple of caveats:

  • it only checks tags as strings, so it won't notice if a tag has moved (but in general, we don't really deal with that anyway)
  • if there's no prior record of the image (no tags value in memcached), it'll fire the new image notification; if memcached has restarted, that means it'll be called for each image. This doesn't necessarily result in as many automation runs, though, since the AskForImagePoll will discard duplicate requests.

To process image push hooks, we need to be able to pre-empt the cache
warming loop.

This commit introduces a priority channel, as a field of the warmer,
and changes the loop such that it looks up image data from the
priority channel in preference to refreshing image data from the
cluster.

It is possible to be notified about an image for which the loop does
not have credentials. This means the priority lookup will be skipped,
and is otherwise harmless. The credentials are only requested at the
start, and when the regular refresh is run. A refinement which
improves on this would be to plumb through a means of getting a
specific image's credentials from the cluster.
When refreshing the metadata for an image (possibly at the prompting
of the priority channel), we can compare the stored set of tags with
those we have just fetched. If there are new tags in the latter,
that's a sign that it's worth doing an automation run -- the new tag
might well be an update for a controller.

Since we might miss a new image notification for whatever reason, we
still drive the automation on a ticker; this just pre-empts that
ticker.
 - this means we'll start refreshing things straight away rather than
   waiting for the first tick
 - the integration test assumes it works this way
@squaremo squaremo changed the title [WIP] Image metadata lookup and automation for push hooks Image metadata lookup and automation for push hooks Nov 15, 2017
// Otherwise, check whether there are any entries in the
// fetched tags that aren't in the cached tags, ignoring any
// in the cached tags that aren't in fetched tags.
sort.Strings(cacheTags)

This comment was marked as abuse.

This comment was marked as abuse.

Copy link
Contributor

@samb1729 samb1729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sufficient set implementation boilerplate

@squaremo squaremo merged commit d1ddc6f into master Nov 15, 2017
@squaremo squaremo deleted the priority-warming branch November 15, 2017 12:50
@rade rade mentioned this pull request Feb 3, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants