[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting

[bdrillard]

What changes were proposed in this pull request?

This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged in #18075.

How was this patch tested?

The same test provided in #18075 is included in this patch.

[maor121]

This is a very important fix for a very common use case. In our company we had to create a workaround in many different cases. Are there any plans to merge this into 2.0.x as well?
We cannot upgrade to 2.1, and we really need this fix on 2.0.x as well :)

[bdrillard]

@maor121 Yeah, I should be able to backpatch to 2.0.x as well in a separate PR.

[dongjoon-hyun]

Hi, @bdrillard .
Did you make a backport for branch-2.2?

[bdrillard]

@dongjoon-hyun I had not yet. I hadn't realized master was already on a 2.3.0-SNAPSHOT. I can also write a backport for branch-2.2.

[dongjoon-hyun]

Yep. To backport, we must land this on branch-2.2 first.
In addition, you can use the same title except branch name like the following.

• [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting
• [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting

[bdrillard]

@dongjoon-hyun Great, thanks for the heads up. I'll prepare a branch-2.2 backport PR and circle back to this PR with the link as confirmation.

[bdrillard]

@dongjoon-hyun I've prepared the PR for the branch-2.2 backport, #18377.

[cloud-fan]

ok to test

[SparkQA]

Test build #78394 has finished for PR 18354 at commit 5685072.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #78401 has finished for PR 18354 at commit 51a59cb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to 2.1!

[HyukjinKwon]

@bdrillard, sounds properly backported. Would you close this?

[manojlds]

Finding it difficult to track. Is this available in a 2.1 release?

[cloud-fan]

it was backported and then reverted because it's too risky, we can close this PR.

[manojlds]

@cloud-fan - thanks. Is there any known workaround for this?

[cloud-fan]

Unfortunately there is no workaround... You can upgrade to spark 2.3 or do the backport and make a new release in your company. cc @rxin any ideas for this case?