feat(kubernetes): Pruning #131

sh0rez · 2019-12-11T15:53:18Z

This PR is another take on implementing a garbage collection of resources, that are no longer present in the Jsonnet code.

@malcolmholmes already experimented with using a label and removing resources (#117) and kubectl apply --prune (#123). Both attempts had their shortcomings (performance, reliability).

Just out of interest I tried what I could come up with and so far I think this is robust enough to be used in production:

we label each resource with tanka.dev/environment: <relative-path-from-project-root>
on apply / diff we get all most common resources (the same list kubectl apply --prune uses) and check if there is anything labeled in there we don't know about
These will be marked as being deleted in diff
On apply, they are deleted.
This behavior defaults to enabled, as it ensures that the Jsonnet is declarative
It can be disabled by using --prune=false, labeling can be disabled using spec.injectLabels.environment.

Can I get some opinions on this @rfratto @gouthamve @tomwilkie?

Co-authored-by: Malcolm Holmes [email protected]

sh0rez · 2019-12-16T17:14:46Z

Todo

Make it respect --target: Currently pruning will always happen, there is no chance to say what to prune
Make it opt in: When enabled by default, it will cause trouble with users that work on branches that are not frequently rebased onto master. Instead, it is the better workflow to just run tk apply --prune once in a while and you are good to go.

Before, the name of the internally used Environment type was different depending on the argument passed on the command line, even though it resulted in the same environment after all (relative vs absolute paths, etc.) This fixes the issue by using the `jpath` package to compute baseDir and rootDir, using the relative path from rootDir to baseDir as the name. This results in stable names, regardless which directory representation is used.

The environment's name (relative path from rootDir) is now added into each created resource as a label, to be able to easily identify objects by their origin. Furthermore, the Kubernetes type now includes the entire Environment (had only the spec before), to be able to access the env's name in that package as well

Allows to query all known apiResources from the server

Adds support for removing orphaned objects from the cluster. They can be viewed in diff and removed in apply

Pruning defaults to enabled, but can be opted out using `--prune=false`

Broken in upstream, nothing we can do anything about

All of our current commands take environments, only the import analysis works on plain files ... this has no benefit, it actually makes the implementation harder, as we cannot use stable paths there. Switches to environments as the target.

making it opt out can result in dangerous moments when the local sources are not recent enough. As this could kill entire clusters, we are going to make it opt in

sh0rez · 2020-01-06T21:04:02Z

@rfratto @malcolmholmes I guess this is ready for another round of review.

Made it opt in
Made it respect targets

I think it is safe for production usage now. WDYT?

anguslees · 2020-01-11T22:19:29Z

How will CRDs work in this?
Does it work with manifests that include multiple namespaces? (I didn't read deeply enough)

(Basically: Why not just duplicate the logic used in kubecfg and descendents (ksonnet, qbec, etc)?)

malcolmholmes · 2020-01-12T15:43:20Z

I don't think CRDs will work with this particularly well. The Kubernetes API does not provide a "list all resources" API (well it does, but the 'all' that it lists is just a subset of resources). There is kubectl api-resources but that gives you a HUGE list, and you will have to check each and every one one at a time, which isn't too efficient. Maybe it is possible to support CRDs, and maybe @sh0rez 's latest code will do it fine, but you are right to ask your question - it is a tough thing.

sh0rez · 2020-01-12T15:47:35Z

I don't think CRDs will work with this particularly well. The Kubernetes API does not provide a "list all resources" API (well it does, but the 'all' that it lists is just a subset of resources). There is kubectl api-resources but that gives you a HUGE list, and you will have to check each and every one one at a time, which isn't too efficient. Maybe it is possible to support CRDs, and maybe @sh0rez 's latest code will do it fine, but you are right to ask your question - it is a tough thing.

Kind of. While implementing a similar feature in kubectl, it became an obvious limitation that we cannot check all kinds at once ... this is a design thing of the API and cannot be fixed easily. Instead, kubectl uses a whitelist of the most common objects – which obviously won't include CRD's.

Tanka uses the same list, but also supports checking the entire list, but this will be much slower.

Of course we could add a flag that allows you to extend the whitelist on the CLI.

So @anguslees to answer the question:

it handles CRD's just fine, as long as they are in the whitelist
they are not in the whitelist by default but you will be able to add them on the CLI (or we could even make this possible from Jsonnet directly)

malcolmholmes · 2020-01-12T16:52:38Z

presumably you could also add the whitelist in spec.json? Then, if there are specific resource types that aren't in the default list, you know they will ALWAYS be checked for the relevant environments?

sh0rez · 2020-01-12T16:56:30Z

presumably you could also add the whitelist in spec.json? Then, if there are specific resource types that aren't in the default list, you know they will ALWAYS be checked for the relevant environments?

I think it is the wrong place, because such a thing applies per library (libraries introduce CRD's after all) and not per environment. Otherwise, you would need to track the whitelist across all JSON files, which leads to config drift, which is the entire goal of Tanka to prevent this.

Thus said, we could instead introduce a special tanka.dev object in the Jsonnet output, which everyone can write to:

{
  "tanka.dev": {
    "additionalPruneKinds": [ ... ],
  }
}

This also means the library that introduces a CRD can specify it directly, users won't even notice it

malcolmholmes · 2020-01-12T17:34:50Z

That could work too. It would be ignored as regarding kubernetes resources, which is good.

I would suggest:

{
  "tanka.dev": {
    "prune": {
      "additionalKinds": [ ... ],
    },
  }
}

Or better still:

{
  "tanka.dev": {
    "prune": {
      "kinds"+: [ ... ],
    },
  }
}

"additional" implied by jsonnet construct.

sh0rez · 2020-01-12T17:36:30Z

That could work too. It would be ignored as regarding kubernetes resources, which is good.

I would suggest:
{
  "tanka.dev": {
    "prune": {
      "additionalKinds": [ ... ],
    },
  }
}
Or better still:
{
  "tanka.dev": {
    "prune": {
      "kinds"+: [ ... ],
    },
  }
}
"additional" implied by jsonnet construct.

Like that! Depends on whether we can preseed the initial Jsonnet object though ... @sbarzowski is this possible? And are there any drawbacks?

anguslees · 2020-01-19T22:40:07Z

(Basically: Why not just duplicate the logic used in kubecfg and descendents (ksonnet, qbec, etc)?)

Repeating ^. I think the authors/reviewers of this PR should be aware that kubecfg/ksonnet/qbec follow an algorithm different to kubectl/helm, and I think it addresses a number of weaknesses in the kubectl-prune approach (disclaimer: I wrote the initial version in kubecfg). In particular they walk all the GroupKinds (using api-versions introspection) and work just fine with CRDs since they contain no hard-coded list.

Happy to swap notes further if it's useful - mostly just wanted to ensure you're choosing the proposed whitelist-approach with knowledge of alternatives.

sh0rez · 2020-01-20T09:29:08Z

@anguslees To be honest, we did not consider other approaches in the ecosystem when designing this ... we took kubectl as a reference.

Could you please share some details on the approach you suggest?

does it allow to avoid having to maintain a whitelist?
how does it look performance wise? how long does it take to compute the pruned objects?
what about edge cases? for example the current one does not handle api groups (apps/v1 vs extensions/v1beta1) very well
how well does it fit the Tanka codebase? For example, we are not using client-go (instead kubectl) and we are not using any of the apimachinery (though we can change if it is beneficial).

We are not fixed to what is done in this PR, if there are better ways to address the issue I am happy to abandon this one and use something else :D

sh0rez · 2020-02-29T20:54:36Z

In the meantime, this PR has drifted off too much from the codebase. Furthermore, the taken approach is not the best, we should reconsider as part of our Roadmap (#217).

That being said, closing this now. Keep an eye on #88 for future efforts.

anguslees · 2020-03-09T01:27:25Z

Replying here so that when this feature is resumed, it can be discovered:

Could you please share some details on the approach you suggest?

The relevant bit of kubecfg's code is https://github.com/bitnami/kubecfg/blob/f07144e10f55b35eed18e5fd0e1335be91105a1a/pkg/kubecfg/update.go#L338

From memory, qbec implements (mostly) the same algorithm using different code, if you want another source. qbec's gc docs are https://qbec.io/reference/garbage-collection/ and are a good summary of the issues involved.

does it allow to avoid having to maintain a whitelist?

Correct, there is no whitelist.

how does it look performance wise? how long does it take to compute the pruned objects?

It takes as long as it takes to do an API list for each groupkind. In my experience this is dominated by the round trip time to the server (assuming they're done sequentially) - for a "local" apiserver, the entire garbage collection process takes 10-ish seconds for my 100-ish-pod cluster.

what about edge cases? for example the current one does not handle api groups (apps/v1 vs extensions/v1beta1) very well

Works fine. The key point here is to track objects by uid, not by apigroup+namespace+name. The "same" resource willl have the same uid, even when viewed through multiple apigroups/versions.

how well does it fit the Tanka codebase? For example, we are not using client-go (instead kubectl) and we are not using any of the apimachinery (though we can change if it is beneficial).

Don't know the tanka codebase well enough to answer that. I suspect the hardest part will be building the list of apigroups, since the output from kubectl api-resources is not as machine-friendly as the "real" REST API response.

sh0rez added kind/feature Something new should be added component/kubernetes Working with Kubernetes clusters labels Dec 11, 2019

sh0rez requested a review from rfratto December 11, 2019 15:53

sh0rez self-assigned this Dec 11, 2019

sh0rez changed the title ~~feat(kubernetes): Pruning~~ WIP feat(kubernetes): Pruning Dec 16, 2019

sh0rez added the keepalive label Jan 2, 2020

sh0rez added 7 commits January 6, 2020 21:07

fix: schemaError format

7393208

feat(kubernetes/client): apiResources

c341b86

Allows to query all known apiResources from the server

feat(kubernetes): Pruning

48f6d67

Adds support for removing orphaned objects from the cluster. They can be viewed in diff and removed in apply

feat(cli): configure pruning using CLI

0c5814d

Pruning defaults to enabled, but can be opted out using `--prune=false`

fix(kubernetes): ignore broken ComponentStatus resource

f8c7660

Broken in upstream, nothing we can do anything about

sh0rez force-pushed the prune branch from 57040f6 to bcc8991 Compare January 6, 2020 20:08

sh0rez added 4 commits January 6, 2020 21:08

style: gofmt strict

bcc8991

fix(kubernetes): prune opt-in

068ede9

making it opt out can result in dangerous moments when the local sources are not recent enough. As this could kill entire clusters, we are going to make it opt in

feat(kubernetes): make --prune respect --target

2f61573

sh0rez changed the title ~~WIP feat(kubernetes): Pruning~~ feat(kubernetes): Pruning Jan 6, 2020

sh0rez mentioned this pull request Jan 11, 2020

tk delete #155

Closed

malcolmholmes mentioned this pull request Jan 13, 2020

Access to Tanka metadata #161

Closed

sh0rez mentioned this pull request Jan 13, 2020

feat(jsonnet): expose Environment at runtime #163

Merged

sh0rez mentioned this pull request Jan 23, 2020

fix: Stable environment name #182

Merged

sh0rez closed this Feb 29, 2020

This was referenced Apr 3, 2020

Pruning resources #88

Closed

feat: Prune old resources #251

Merged

sh0rez deleted the prune branch July 1, 2020 13:40

Mikulas mentioned this pull request Sep 27, 2020

Environment name set in spec.json is ignored #393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kubernetes): Pruning #131

feat(kubernetes): Pruning #131

sh0rez commented Dec 11, 2019 •

edited

Loading

sh0rez commented Dec 16, 2019 •

edited

Loading

sh0rez commented Jan 6, 2020

anguslees commented Jan 11, 2020

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020 •

edited

Loading

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020

anguslees commented Jan 19, 2020

sh0rez commented Jan 20, 2020 •

edited

Loading

sh0rez commented Feb 29, 2020

anguslees commented Mar 9, 2020

feat(kubernetes): Pruning #131

feat(kubernetes): Pruning #131

Conversation

sh0rez commented Dec 11, 2019 • edited Loading

sh0rez commented Dec 16, 2019 • edited Loading

Todo

sh0rez commented Jan 6, 2020

anguslees commented Jan 11, 2020

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020 • edited Loading

malcolmholmes commented Jan 12, 2020

sh0rez commented Jan 12, 2020

anguslees commented Jan 19, 2020

sh0rez commented Jan 20, 2020 • edited Loading

sh0rez commented Feb 29, 2020

anguslees commented Mar 9, 2020

sh0rez commented Dec 11, 2019 •

edited

Loading

sh0rez commented Dec 16, 2019 •

edited

Loading

sh0rez commented Jan 12, 2020 •

edited

Loading

sh0rez commented Jan 20, 2020 •

edited

Loading