[SPARK-31203][BUILD] Upgrade derby to 10.14.2.0 from 10.12.1.1

[Udbhav30]

What changes were proposed in this pull request?

This PR(SPARK-31203) aims to upgrade derby.

Some major changes from this upgrade are
DERBY-6987 The default Network Server security policy file could be trimmed down somewhat.
DERBY-6986 Network Server COMMAND_TESTCONNECTION need not try to open a database
DERBY-6726 NPE from trigger

Why are the changes needed?

To bring some bug fixes.

Does this PR introduce any user-facing change?

no

How was this patch tested?

manual build

[Udbhav30]

cc @dongjoon-hyun

[maropu]

ok to test

[maropu]

Could you leave some major changes from 10.12.1.1 to 10.14.2.0 in the PR descritpion? e.g., #27860 (comment)

[SparkQA]

Test build #120094 has finished for PR 27970 at commit 515f3a0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @Udbhav30 . Thank you for making a PR. BTW, in a production environment, Apache Hive Metastore is used instead of Derby. I don't think this PR is a security related patch. Also, those network features of Derby are not used by Apache Spark, are they?

Also, this increases the inconsistency from Hive modules which use 10.10.2.0 in hive-1.2 and hive-2.3 profiles. We don't use Hive 3.0+.

Apache Hive 3.0.0 ~ 4.0.0-SNAPSHOT
<derby.version>10.14.1.0</derby.version>

Apache Hive 1.2.0 ~ 2.3.7-SNAPSHOT
<derby.version>10.10.2.0</derby.version>

cc @wangyum

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #120107 has finished for PR 27970 at commit 515f3a0.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #120137 has finished for PR 27970 at commit 515f3a0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Yeah, Derby isn't for production purpose - separate Metastore is supposed to be used. I don't think that security issue actually matters too. Also, yes, I wouldn't upgrade derby but rather try to match with Hive's side.

Furthermore, different Derby can be picked up (see #20944). Sharing it will potentially cause an issue; therefore, technically upgrading does not fully address the issue you pointed out.

-1 from me.

[Udbhav30]

Hi @dongjoon-hyun , i opened the PR because of the security fix. If you suggest this PR is not required then i will close it.
Please provide your suggestions thanks.

[HyukjinKwon]

If the security issue doesn't directly affect the project, we dont necessarily have to upgrade it. Given that we don't use Derby in production and the upgrade doesn't look completely handling the issue even if it affects Spark, seems not worthwhile upgrading for now.

Can you describe how Spark is affected by the security issue you mentioned?

[dongjoon-hyun]

Thank you all! This is not required for now for that security reason. For the other bug fixes, we can consider later. I'll close the JIRA as Later, too.