Properly generate report yml include with virtual attributes

[kbrock]

Problem

Before

If a report.yml file had virtual attributes in the :cols but did not have an :include entry,
it was not properly generating the includes().references() hash.

REMINDER: report.yml files are used to represent every data view in our product.

After

It is now generating the correct :includes from :col_order when :include is not defined,
regardless of whether :col contains virtual attributes or not.

Impact

We are constantly tuning the app/model classes. Sometimes the relations change (e.g.: pictures no longer points to binary_blobs) and other times the virtual attributes are embeddable in sql. So this means the correct includes()/references() values change.

Since the report.yml files are in a different git repo, they are often not updated. This means extra data gets downloaded, or in some cases, just plain break. Which happened with ServiceTemplate.yml.

The frustrating part is the includes entries can just be derived from the app/model class relationships, and shouldn't even need to be declared in the report.yml files in the first place.

Conclusion

Properly generating the includes() allows us to remove more of the :include entries from the reports, and that means we have fewer incorrect reports / views. And it means faster views.

/cc @jrafanie @martinpovolny

[Fryguy]

I love this change so much...can we remove the "includes" sections from the report.yml files after this? I would love to kill an entire section

[Fryguy]

My question in #18205 (comment) is really asking, can we remove include.presence from this section entirely?

[kbrock]

users tend to copy paste the report templates, and most of the report views have a blank include block. Of our 180 reports, 95 are a blank hash.

$ ag -l '^include:\n\n' product/views/ | wc -l
      95
$ ag -l '^include:\n' product/views/ | wc -l
     170
$ find product/views | wc -l
     180

[kbrock]

@Fryguy Full disclaimer.

I tried to do this in the past #13675
And it blew up / so I had to revert a bunch of them #14439

So there may still be dragons here.
I'm pretty sure introducing this change works now. but will do my best to track down the trouble makers.

[kbrock]

@Fryguy Sweet. This is working.

For OrchestrationStack.yml - which has ext_management_system.name and total_cloud_networks (non-sql friendly virtual attributes) show the same values and perform the same queries after I remove the columns and includes values.

Unlike the last time I tried to make this change, the columns on associations (e.g.: ext_management_systems.name) show up in the final screen.

[kbrock]

@Fryguy changed some comments to actual tests. Are you good to go with this PR?

[miq-bot]

Checked commit kbrock@1b31bd7 with ruby 2.3.3, rubocop 0.52.1, haml-lint 0.20.0, and yamllint 1.10.0
2 files checked, 0 offenses detected
Everything looks fine. 👍

[kbrock]

Ugh. Rebasing onto red master strikes again

[Fryguy]

This one has the potential to break the UI...can you verify that the UI tests pass with this PR?

cc @himdel (This is where I would have liked to do @miq-bot test-plugin manageiq-ui-classic 😉 ...btw, that test framework is in the process of being built)

[Fryguy]

LGTM, pending UI test verification.

[kbrock]

What this effects:

1. Have no includes: defined in the report (8 of the 180 view reports)
2. Have cols defined (all of our views)
3. Have virtual attributes that are ruby only (sql virtual attributes are ok) (3 of those 8)
4. Reference other tables (dots) in the col_order (0 of those 8)

And this makes sense, because if we did have any of those, they would be broken already.
This PR simply fixes reports that fall under that case (which we avoided since it is broken)

Hmm. Little confused this bug bit us so badly, since if we tend to remove the cols and the includes at the same time. I guess it only comes into play when we want to bring down a column that is not displayed e.g.: picture.id