HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown #3768
Conversation
|
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Not very experienced with the hbase integration, goes little bit out of my zone. if it gets stuck by any chance. Do bother the relevant mailing list.
@jojochuang has a comment on the jira, so we need an agreement from him as well.
@jojochuang if you could spare some time and give a feedback, IIRC you have some Hbase context as well. :-)
| <hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version> | ||
| <hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version> |
There was a problem hiding this comment.
how is this upgrade downgrading the hadoop version? Earlier on a lower version we were good with 3.x and now to upgrade we are coming down.
Out of curiosity, why 2.8.5, nothing from 2.10 line or so?
There was a problem hiding this comment.
Valid question. The earlier hbase version 2.0 is not only not so stable but also it just worked with Hadoop 3 version magically but as we upgrade HBase 2.x versions, they won't work by default with Hadoop 3 because as of today, hbase has only published hadoop 2 compatible hbase 2 artifacts by default. If we want to make hbase 2 work with hadoop 3, first we need to compile hbase 2 with hadoop-3.0 profile, only then we get hadoop 3 compatible hbase 2 jars. Hence, we should have anyways kept hbase-compatible-hadoop.version on Hadoop's 2.x release line but somehow we didn't and this worked with HBase 2.0.
I have provided detailed explanation of this also on HADOOP-17668. Comment here (hope I covered the right link, Jira interface seems to have been updated recently)
And yes @jojochuang's review would also be great, thanks for the reminder (my bad, I missed).
There was a problem hiding this comment.
hmm, I need to go through this and try, considering @aajisaka is convinced. I will back off for now.
You guys can proceed. Nothing blocking from my side
There was a problem hiding this comment.
Out of curiosity, why 2.8.5, nothing from 2.10 line or so?
To provide in-depth dependency view for this question: https://github.com/apache/hbase/blob/rel/2.2.4/pom.xml#L1373-L1378
<hadoop-two.version>2.8.5</hadoop-two.version>
<hadoop-three.version>3.1.2</hadoop-three.version>
<!-- These must be defined here for downstream build tools that don't look at profiles.
They ought to match the values found in our default hadoop profile, which is
currently "hadoop-2.0". See HBASE-15925 for more info. -->
<hadoop.version>${hadoop-two.version}</hadoop.version>
There was a problem hiding this comment.
Ah i see. Yes by default HBase 2 is built with Hadoop 2 profile. It is not until HBase 3.0 (being developed) that Hadoop 2 profile got removed.
|
Merged. Thank you @virajjasani and @ayushtkn. |
…own (apache#3768) Signed-off-by: Akira Ajisaka <[email protected]>
…own (apache#3768) Signed-off-by: Akira Ajisaka <[email protected]>
Description of PR
As mentioned on HADOOP-17668, we can't upgrade hbase2 profile version beyond 2.2.4 until we either have hbase 2 artifacts available that are built with hadoop 3 profile by default or hbase 3 is rolled out (hbase 3 is compatible with hadoop 3 versions only).
This PR is to upgrade hbase2 profile version to 2.2.4 and also fix TestTimelineWriterHBaseDown to create connection only after mini cluster is up.
How was this patch tested?
Local testing.
For unit testing, here is the test result with hbase2 profile:
For code changes: