Report Widget

[kbrock]

High level

Widgets are reports that are pre-run for users.
They are updated every hour and run in the user's group context and timezone.
In summary, each report is run many times (# groups x # timezones x # times per day).
So any performance issue will be felt.

A customer complained that three reports were taking too long to run.

Before

• While we do embed virtual attributes directly into SQL where possible ( Allow cpu_usagemhz_rate_average_max_over_time_period and derived_memory_used_max_over_time_period to be used in reports as SQL #13101 ) we are still bringing back the tables for these columns.
• Top Storage Consumers report uses a non-SQL-friendly attribute Vm#provisioned_storage. This forces a download of the hardwares and disks tables.

After

• Tables for SQL-friendly virtual attributes are no longer downloaded. This is done by removing them from the includes list.
• Vm#provisioned_storage is now SQL-friendly so we can take advantage of the above includes optimization.

numbers

ms	bytes	objects	queries	query (ms)	rows	comments
6,119.8	103,342,161*	7,534,105	1,194	1,497.9	6,768	before
4,958.1	121,689,623*	5,639,577	1,190	1,197.1	1,148	after
19.0%	n/a	25%	0.3%	20.1%	83%	diff

* Memory usage does not reflect 7,169,463 freed objects.
* Memory usage does not reflect 2,051,300 freed objects.
GC cleaned up too many objects to truly reflect the memory improvement

related to:

• Speed up widget generation #14224
• https://bugzilla.redhat.com/show_bug.cgi?id=1251259

---

reproduction details

profile_methods("MiqWidget::ContentGenerator#generate_content", "MiqWidget#generate_one_content_for_group", "MiqWidget#generate_one_content_for_user")
opts = ["MiqGroup", "EMS_OPS_BU", nil, ["Pacific Time (US & Canada)", "UTC"]] # w.group_options.last
w = MiqWidget.find_by(description: 'report_top_storage_consumers')
w.reload ; bookend("master", gc: true) { w.send(:content_generator).generate(w, *opts) }

ms	bytes	objects	queries	query (ms)	rows	comments
11,311.3	120,445,839*	16,006,917	1,194	2,529.8	6,768	master-1
6,180.1	7,873,235*	7,534,025	1,194	1,418.4	6,768	master-1
6,195.6	104,498,675*	7,516,979	1,194	1,528.1	6,768	master-1
6,119.8	103,342,161*	7,534,105	1,194	1,497.9	6,768	master-1

* Memory usage does not reflect 7,169,463 freed objects.
* Memory usage does not reflect 3,134,060 freed objects.
* Memory usage does not reflect 3,117,013 freed objects.
* Memory usage does not reflect 3,134,141 freed objects.

ms	bytes	objects	queries	query(ms)	rows	comments
8,915.0	44,824,308*	12,650,749	1,190	2,322.9	1,148	widget-1
4,912.0	124,986,592*	5,598,635	1,190	1,230.9	1,148	widget-1
5,074.4	9,322,759*	5,426,925	1,190	1,381.1	1,148	widget-1
4,958.1	121,689,623*	5,639,577	1,190	1,197.1	1,148	widget-1

* Memory usage does not reflect 5,280,780 freed objects.
* Memory usage does not reflect 2,010,357 freed objects.
* Memory usage does not reflect 1,838,645 freed objects.
* Memory usage does not reflect 2,051,300 freed objects.

[kbrock]

/cc @NickLaMuro

[gtanzillo]

Should this be klass.respond_to?(:virtual_attribute?)

[miq-bot]

Checked commits kbrock/manageiq@4357b8d~...26de928 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
4 files checked, 0 offenses detected
Everything looks good. 🍪

[gtanzillo]

Looks good 👍

[simaishi]

@kbrock Please resolve conflict and create an Euwe-specific PR (referencing this one) or suggest other PRs to backport.

$ git diff
diff --cc app/models/miq_report/generator.rb
index b118111,f340ead..0000000
--- a/app/models/miq_report/generator.rb
+++ b/app/models/miq_report/generator.rb
@@@ -85,7 -97,11 +85,15 @@@ module MiqReport::Generato
      if klass.nil?
        klass = db_class
        result = {}
++<<<<<<< HEAD
 +      cols.each { |c| result.merge!(c.to_sym => {}) if klass.virtual_attribute?(c) || klass == LiveMetric } if cols
++=======
+       if cols && klass.respond_to?(:virtual_attribute?)
+         cols.each do |c|
+           result[c.to_sym] = {} if klass.virtual_attribute?(c) && !klass.attribute_supported_by_sql?(c)
+         end
+       end
++>>>>>>> 832c5a3... Merge pull request #14285 from kbrock/widget_2
      end
  
      if includes.kind_of?(Hash)
@@@ -99,12 -115,16 +107,20 @@@
            assoc_klass = assoc_reflection.nil? ? nil : (assoc_reflection.options[:polymorphic] ? k : assoc_reflection.klass)
  
            if v.nil? || v["include"].blank?
 -            result[k] = {}
 -          elsif assoc_klass
 -            result[k] = include_as_hash(v["include"], assoc_klass)
 +            result.merge!(k => {})
 +          else
 +            result.merge!(k => get_include_for_find(v["include"], assoc_klass)) if assoc_klass
            end
  
++<<<<<<< HEAD
 +          v["columns"].each { |c| result[k].merge!(c.to_sym => {}) if assoc_klass.virtual_attribute?(c) } if assoc_klass && assoc_klass.respond_to?(:virtual_attribute?) && v["columns"]
++=======
+           if assoc_klass && assoc_klass.respond_to?(:virtual_attribute?) && v["columns"]
+             v["columns"].each do |c|
+               result[k][c.to_sym] = {} if assoc_klass.virtual_attribute?(c) && !assoc_klass.attribute_supported_by_sql?(c)
+             end
+           end
++>>>>>>> 832c5a3... Merge pull request #14285 from kbrock/widget_2
          end
        end
      elsif includes.kind_of?(Array)

[kbrock]

Thanks @simaishi - removing euwe label