[WIP] [DEBUG] CONTAINER_INFLATE_FACTOR env var to duplicate most entities

[cben]

@miq-bot add-label providers/containers, performance, wip

This is silly and incomplete but already useful to stress container refresh performance.
Plus more logging (TODO: separate PR with logging)

Usage:

env CONTAINER_INFLATE_FACTOR=100 bundle exec rspec ./spec/models/manageiq/providers/openshift/container_manager/refresher_spec.rb:40

Currently fails on wrong @container_build_pod.container_build but never mind.

• [untested] Should also work refreshing a live environment.
  Be ready to reset your DB, especially if you vary CONTAINER_INFLATE_FACTOR...

• TODO: does not inflate nodes & images.

Trying to understand what the customer may be hitting in https://bugzilla.redhat.com/show_bug.cgi?id=1436176. However, they have huge number of images, which this does not inflate :-(

First findings

https://gist.github.com/cben/4b8404346f42c917fd1aead993db895e

• speed becomes especially abysmal when it goes into swap.
  E.g. CONTAINER_INFLATE_FACTOR=1000 sends my 12G laptop deep into swap...

using https://github.com/jvns/ruby-stacktrace to spy on the ruby stack:

-parsing:

• during parser, I see 60~90% time in OpenStruct new_ostruct_member. It seems that if Kubeclient left things as hashes and our parser accessed foo[:bar] or foo["bar"] instead of foo.bar, we could gain a lot.

• parser is not main bottleneck.

• container_groups is ~30% of total parsing time - OK, container_templates is ~50% - surprising!

• I'd expect RAM usage to reach approximate peak during parsing, but I do see significant increase later while saving. TODO: study. (especially plausible with one huge table — like images in customer's case)

• saving:

  • before Optimize store_ids_for_new_records by getting rid of the O(n^2) lookups #14542, during save a lot of time is indeed stuck in store_ids_for_new_records.
    rest of time it's spread around postgresql_adapter, active_record etc...
  • after Optimize store_ids_for_new_records by getting rid of the O(n^2) lookups #14542, save is nicely faster and spread around postgres, AR etc all time.
    eg. for 11000 replicators, save became 3x faster.
    however, could still take hours in absolute time...

[miq-bot]

Some comments on commits cben/manageiq@29786af~...2bcff0b

app/models/ems_refresh/save_inventory_container.rb

• ⚠️ - 13 - Detected puts. Remove all debugging statements.

app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb

• ⚠️ - 10 - Detected puts. Remove all debugging statements.

app/models/manageiq/providers/openshift/container_manager/refresher.rb

• 💣 💥 🔥 🚒 - 62 - Detected byebug. Remove all debugging statements.
• 💣 💥 🔥 🚒 - 65 - Detected byebug. Remove all debugging statements.
• ⚠️ - 40 - Detected puts. Remove all debugging statements.
• ⚠️ - 72 - Detected puts. Remove all debugging statements.

[miq-bot]

Checked commits cben/manageiq@29786af~...2bcff0b with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
4 files checked, 8 offenses detected

app/models/ems_refresh/save_inventory_container.rb

• ❗ - Line 13, Col 7 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.

app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb

• ❗ - Line 10, Col 7 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
• ❗ - Line 360, Col 78 - Style/MultilineMethodCallBraceLayout - Closing method call brace must be on the line after the last argument when opening brace is on a separate line from the first argument.

app/models/manageiq/providers/openshift/container_manager/refresher.rb

• ⚠️ - Line 62, Col 17 - Lint/Debugger - Remove debugger entry point byebug.
• ⚠️ - Line 65, Col 15 - Lint/Debugger - Remove debugger entry point byebug.
• ❗ - Line 40, Col 44 - Style/ExtraSpacing - Unnecessary spacing detected.
• ❗ - Line 40, Col 9 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.
• ❗ - Line 72, Col 11 - Rails/Output - Do not write to stdout. Use Rails's logger if you want to log.

[miq-bot]

This pull request is not mergeable. Please rebase and repush.