Fix struct flattening to add a validity column only when the input column has null element

[ttnghia]

Currently, struct flattening adds a validity column when the input column has a null mask (by calling to nullable()). In the situation when comparing two structs columns having no null but one column has a null mask, flattening them will result in two tables with different numbers of columns.

This PR fix that problem by using has_nulls() instead of nullable(). As a result, the validity column will be added to the flattening result only when the input structs column has null.

Note that when comparing two structs columns in which one column has null while the other doesn't, we must check for (nested) null existence and pass in column_nullability::FORCE for flattening both columns. This makes sure the flattening results are tables having the same number of columns.

Closes #8187.

[abellina]

Locally this fixes the issue reported by @gerashegalov

[mythrocks]

LGTM!

[harrism]

@gpucibot merge

[gerashegalov]

Can you add a test case (ok if in another PR) so we don't defer to spark-rapids to catch these issues

[ttnghia]

Can you add a test case (ok if in another PR) so we don't defer to spark-rapids to catch these issues

Sure. But since we are about to do code freeze, can that test be targeted to branch 21.08?

[gerashegalov]

Sure. But since we are about to do code freeze, can that test be targeted to branch 21.08?

I think it's reasonable since @abellina confirmed the fix with a Spark test run

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.06@e97fc1c). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 447b8fe differs from pull request most recent head 224eb13. Consider uploading reports for the commit 224eb13 to get more accurate results

@@               Coverage Diff               @@
##             branch-21.06    #8374   +/-   ##
===============================================
  Coverage                ?   82.86%           
===============================================
  Files                   ?      106           
  Lines                   ?    17874           
  Branches                ?        0           
===============================================
  Hits                    ?    14811           
  Misses                  ?     3063           
  Partials                ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e97fc1c...224eb13. Read the comment docs.