[BUG] Null Policy of EqualJoin on StructType

[sperlingxx]

Describe the bug
Currently, cuDF flatten all nested columns before performing the join. After flattening, validity masks of nested type columns are lost. It means that cuDF will regard join key composed by non-null struct who only contains single null child value as a null key by mistake.

For instance,
If the schema of join key is StructType([("a", LongType(nullable=True))], nullable=False), the non-null record Struct([("a", null)]) will be regarded as a null record because of the flattening of nested columns.

[jrhemstad]

After flattening, validity masks of nested type columns are lost.

No they aren't. The child validity masks are projected into a BOOL column.

will regard join key composed by non-null struct who only contains single null child value as a null key

This is precisely why we project the validity mask into a BOOL column to avoid this.

{NULL} and NULL will not compare as equal because it will expand to TRUE, NULL and FALSE, NULL.

[sperlingxx]

Hi @jrhemstad, thanks for correcting me! And I filed another issue(#7934) to describe my problem.