Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown #3768

Merged
merged 1 commit into from
Dec 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions hadoop-project/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@
<swagger-annotations-version>1.5.4</swagger-annotations-version>
<snakeyaml.version>1.26</snakeyaml.version>
<hbase.one.version>1.7.1</hbase.one.version>
<hbase.two.version>2.0.2</hbase.two.version>
<hbase.two.version>2.2.4</hbase.two.version>
<junit.version>4.13.2</junit.version>
<junit.jupiter.version>5.5.1</junit.jupiter.version>
<junit.vintage.version>5.5.1</junit.vintage.version>
Expand Down Expand Up @@ -2428,9 +2428,10 @@
</activation>
<properties>
<hbase.version>${hbase.two.version}</hbase.version>
<hbase-compatible-hadoop.version>3.0.0</hbase-compatible-hadoop.version>
<hbase-compatible-hadoop.version>2.8.5</hbase-compatible-hadoop.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this upgrade downgrading the hadoop version? Earlier on a lower version we were good with 3.x and now to upgrade we are coming down.
Out of curiosity, why 2.8.5, nothing from 2.10 line or so?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid question. The earlier hbase version 2.0 is not only not so stable but also it just worked with Hadoop 3 version magically but as we upgrade HBase 2.x versions, they won't work by default with Hadoop 3 because as of today, hbase has only published hadoop 2 compatible hbase 2 artifacts by default. If we want to make hbase 2 work with hadoop 3, first we need to compile hbase 2 with hadoop-3.0 profile, only then we get hadoop 3 compatible hbase 2 jars. Hence, we should have anyways kept hbase-compatible-hadoop.version on Hadoop's 2.x release line but somehow we didn't and this worked with HBase 2.0.

I have provided detailed explanation of this also on HADOOP-17668. Comment here (hope I covered the right link, Jira interface seems to have been updated recently)

And yes @jojochuang's review would also be great, thanks for the reminder (my bad, I missed).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I need to go through this and try, considering @aajisaka is convinced. I will back off for now.
You guys can proceed. Nothing blocking from my side

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why 2.8.5, nothing from 2.10 line or so?

To provide in-depth dependency view for this question: https://github.com/apache/hbase/blob/rel/2.2.4/pom.xml#L1373-L1378

    <hadoop-two.version>2.8.5</hadoop-two.version>
    <hadoop-three.version>3.1.2</hadoop-three.version>
    <!-- These must be defined here for downstream build tools that don't look at profiles.
         They ought to match the values found in our default hadoop profile, which is
         currently "hadoop-2.0". See HBASE-15925 for more info. -->
    <hadoop.version>${hadoop-two.version}</hadoop.version>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah i see. Yes by default HBase 2 is built with Hadoop 2 profile. It is not until HBase 3.0 (being developed) that Hadoop 2 profile got removed.

<hbase-compatible-guava.version>11.0.2</hbase-compatible-guava.version>
<hbase-server-artifactid>hadoop-yarn-server-timelineservice-hbase-server-2</hbase-server-artifactid>
<hbase-compatible-jetty.version>9.3.27.v20190418</hbase-compatible-jetty.version>
</properties>
<dependencyManagement>
<dependencies>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -515,6 +515,30 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<scope>test</scope>
<version>${hbase-compatible-jetty.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<scope>test</scope>
<version>${hbase-compatible-jetty.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-webapp</artifactId>
<scope>test</scope>
<version>${hbase-compatible-jetty.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
<version>${hbase-compatible-jetty.version}</version>
</dependency>
</dependencies>
</profile>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@ public void testTimelineWriterHBaseDown() throws Exception {
HBaseTestingUtility util = new HBaseTestingUtility();
HBaseTimelineWriterImpl writer = new HBaseTimelineWriterImpl();
try {
util.startMiniCluster();
Configuration c1 = util.getConfiguration();
c1.setLong(TIMELINE_SERVICE_READER_STORAGE_MONITOR_INTERVAL_MS, 5000);
writer.init(c1);
writer.start();

util.startMiniCluster();
DataGeneratorForTest.createSchema(util.getConfiguration());

TimelineStorageMonitor storageMonitor = writer.
Expand Down