Containers graph refresh

[cben]

A tracking issue for things to do (in all repos).
ManageIQ/manageiq#14337 PoC is now nearly 💯 % conversion but a lot of work remains.
This issue is for full refresh; targeted is out of scope => tracking separately in #114.

• 🙏 @zakiva's effort to add deletion tests Refresher_spec: Add deletions tests manageiq-providers-openshift#18.
• 🙏 Improve refresher test coverage, there are some things uncovered even by straightforward refresh from scratch.
  • I have a couple commits in [WIP] [PoC] Container graph refresh manageiq#14337 adding tests.
    => extracted to Add some missing refresher specs manageiq-providers-openshift#19
  • SecurityContext has no tests
  • PVC are not in cassette => they are in the deletions cassete Volume-related graph refresh fixes & VCR specs #74
  • Project labels have no tests?
  • 🐛 possibly found bug in linking pod -> build pod didn't look up by namespace (Erez fixing)
  • some untested things left unimplemented (variables starting with _) in Graph refresh: links between entities #52 ?
• 💄 Consider some renames for consistency in refresh parser e.g. s/:project/:container_project/.
  These would avoid some special casing in both save_inventory and refresh save. (I've tried to keep TODOs in the POC pr.)
• 🏗️ Rebase post repo split, find place to stick new code.
  • k8s refactoring Separate data_index lookups from parsing #46
  • k8s implement graph refresh Configurable graph refresh #30, Graph refresh: links between entities #52
  • openshift refactoring Separate data_index lookups from parsing manageiq-providers-openshift#30
  • openshift implement graph refresh Changes needed for openshift graph refresh #57, Openshift graph refresh manageiq-providers-openshift#34
  • Upgrade existing refresher from deprected parse_legacy_inventory to newer collect/parse separation.
• 🏗️ Don't replace save_inventory_container, add a setting to toggle.
  • Unhide setting.
  • enable by default! Enable graph refresh (batch strategy) by default #112, Enable graph refresh (batch strategy) by default manageiq-providers-openshift#48
• 🏗️ Implement disconnections (currently just deletes). Can set :delete_method but needs batching to be fast.
  • Enable Zahi's deletion tests for graph refresh too.
• 🏗️ 🏇 Persister Add persistor with configurable saver strategy #73, Add openshift persister and test also batch saver strategy manageiq-providers-openshift#39
  • setting for batched saver strategy
• 🏗️ 🐛 tag mapping with graph refresh #126 Implement label->tag mapping (currently not saving tags).
  • Tests skipped for now.
• Measure performance / memory ⏱ 📝

above list is surely incomplete.

RFE: https://bugzilla.redhat.com/show_bug.cgi?id=1470021

[cben]

Extracted some tests ManageIQ/manageiq-providers-openshift#19 ,
associations ManageIQ/manageiq#15298 [merged]

Adam nicely refactored & rebased much of the other changes into #30, plan is to finish and land that (behind a flag), then do similarly for openshift.

[cben]

• 🐛 component statuses tries to store a KubeClient object :message=>#<Kubeclient::ComponentStatus health="true">,
• 🐛 fixing some volume-related bugs, adding tests Volume-related graph refresh fixes & VCR specs #74

[cben]

• secondary index by blacklisted :namespace is bad — seems to work for full refresh but can't match records from DB, as those don't have namespace, only container_project_id.
  extracted to issue tracking targetted refresh RFE: targeted refresh #114

[cben]

• 🙏 ContainerImage creation event has no tests

[cben]

• 🐎 performance, probably very small impact: cross_link_node is done during parsing, consider batched

[cben]

• 🐛 there was tricky bug with save_unused_images: false mode deleting existing child custom_attributes records. Fixed (thanks Ladislav).
  • Have parent inventory collections as dependencies manageiq#15903
  • Add parent inventory collections to ensure proper disconnect order #134
  • Enable check metadata not deleted specs manageiq-providers-openshift#46

We might need more extensive deletion tests for other entities...

[moolitayer]

@cben are we keeping a list of known issue somewhere? Label Tag Mapping is all I could find (except for the save_unused_images issue that has it's own prs)

[cben]

Yes, this is the list :-)

Added emojis to categorize (approximately) correctness/unimplemeted 🏗️ / bugs 🐛 / desired tests 🙏 / performance 🐎 / code polish 💄.

Tag mapping (I'm working on it) and unused images are the only current known bugs.

[cben]

This is long done.