HDFS-15201 SnapshotCounter hits MaxSnapshotID limit #1870

karthikhw · 2020-03-02T19:25:50Z

Jira: https://issues.apache.org/jira/browse/HDFS-15201

Users reported that they are unable to take HDFS snapshots and their snapshotCounter hits MaxSnapshotID limit. MaxSnapshotID limit is 16777215.

SnapshotManager.java

private static final int SNAPSHOT_ID_BIT_WIDTH = 24;

/**
 * Returns the maximum allowable snapshot ID based on the bit width of the
 * snapshot ID.
 *
 * @return maximum allowable snapshot ID.
 */
 public int getMaxSnapshotID() {
 return ((1 << SNAPSHOT_ID_BIT_WIDTH) - 1);
}

I think, SNAPSHOT_ID_BIT_WIDTH is too low. May be good idea to increase SNAPSHOT_ID_BIT_WIDTH to 31? to aline with our CURRENT_STATE_ID limit (Integer.MAX_VALUE - 1).

/**
 * This id is used to indicate the current state (vs. snapshots)
 */
public static final int CURRENT_STATE_ID = Integer.MAX_VALUE - 1;

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java

hadoop-yetus · 2020-03-02T21:57:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 26s	trunk passed
+1 💚	compile	1m 8s	trunk passed
+1 💚	checkstyle	0m 45s	trunk passed
+1 💚	mvnsite	1m 15s	trunk passed
+1 💚	shadedclient	16m 3s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 47s	trunk passed
+0 🆗	spotbugs	2m 50s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 48s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s	the patch passed
+1 💚	compile	1m 1s	the patch passed
+1 💚	javac	1m 1s	the patch passed
+1 💚	checkstyle	0m 37s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
+1 💚	mvnsite	1m 8s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 58s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 42s	the patch passed
+1 💚	findbugs	2m 58s	the patch passed
		_ Other Tests _
-1 ❌	unit	80m 30s	hadoop-hdfs in the patch passed.
+0 🆗	asflicense	0m 40s	ASF License check generated no output?
		146m 50s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots
	hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
	hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
	hadoop.hdfs.server.namenode.TestCacheDirectives
	hadoop.hdfs.server.namenode.snapshot.TestSnapshot
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
	hadoop.hdfs.server.namenode.TestReencryptionWithKMS
	hadoop.hdfs.server.namenode.TestNestedEncryptionZones
	hadoop.hdfs.server.namenode.TestFSImage
	hadoop.hdfs.server.namenode.TestReencryption
	hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
	hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
	hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
	hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands
	hadoop.hdfs.server.namenode.TestFileTruncate
	hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
	hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean

Subsystem	Report/Notes
Docker	Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/1/artifact/out/Dockerfile
GITHUB PR	#1870
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 45d4fd00ddea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `edc2e9d`
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/1/testReport/
Max. process+thread count	4152 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

karthikhw · 2020-03-02T22:18:39Z

Submitted new PR with the changes requested.

hadoop-yetus · 2020-03-03T01:10:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 53s	trunk passed
+1 💚	compile	1m 9s	trunk passed
+1 💚	checkstyle	0m 44s	trunk passed
+1 💚	mvnsite	1m 18s	trunk passed
+1 💚	shadedclient	16m 3s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 46s	trunk passed
+0 🆗	spotbugs	2m 49s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 48s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s	the patch passed
+1 💚	compile	1m 0s	the patch passed
+1 💚	javac	1m 0s	the patch passed
+1 💚	checkstyle	0m 38s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
+1 💚	mvnsite	1m 7s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 39s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s	the patch passed
+1 💚	findbugs	2m 54s	the patch passed
		_ Other Tests _
-1 ❌	unit	92m 36s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s	The patch does not generate ASF License warnings.
		159m 57s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestEncryptionZonesWithKMS
	hadoop.hdfs.TestEncryptionZones

Subsystem	Report/Notes
Docker	Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/2/artifact/out/Dockerfile
GITHUB PR	#1870
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 828dfedc29e0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `edc2e9d`
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/2/testReport/
Max. process+thread count	4480 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

...-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java

jojochuang · 2020-03-03T01:27:52Z

@karthikhw can you double check the failed tests? especially the snapshot tests.

jojochuang · 2020-03-03T01:29:33Z

I've gone through all the usage of the snapshot id. The only concern i had was bitwise operations on the snapshot id, but i didn't find any. Widening the allowed range shouldn't be a problem.

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java

karthikhw

Done!

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java

hadoop-yetus · 2020-03-05T00:04:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 2s	trunk passed
+1 💚	compile	1m 6s	trunk passed
+1 💚	checkstyle	0m 46s	trunk passed
+1 💚	mvnsite	1m 12s	trunk passed
+1 💚	shadedclient	15m 58s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s	trunk passed
+0 🆗	spotbugs	2m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 49s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 6s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	checkstyle	0m 39s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
+1 💚	mvnsite	1m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 52s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 40s	the patch passed
+1 💚	findbugs	2m 53s	the patch passed
		_ Other Tests _
-1 ❌	unit	92m 45s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 40s	The patch does not generate ASF License warnings.
		158m 10s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.TestEncryptionZonesWithKMS
	hadoop.hdfs.TestEncryptionZones

Subsystem	Report/Notes
Docker	Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/3/artifact/out/Dockerfile
GITHUB PR	#1870
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux bb84c0e515d8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 3afd4cb
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/3/testReport/
Max. process+thread count	4515 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1870/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

karthikhw · 2020-03-05T00:17:14Z

It looks test case failure is un-related to this issue.

karthikhw · 2020-03-09T18:20:03Z

@arp7 @jojochuang Can you please review this change if you get sometime?

arp7

+1

@szetszwo do you want to take a look too?

szetszwo · 2020-03-10T15:15:09Z

The changes look good. Just a question: why changing getMaxSnapshotID() from -1 to -2?

karthikhw · 2020-03-10T19:44:30Z

@szetszwo

Changed getMaxSnapshotID() from -1 to -2 because of CURRENT_STATE_ID(Integer.MAX_VALUE - 1) that have -1 already.

If SNAPSHOT_ID_BIT_WIDTH is 28 then we are ok with -1 but later change in SNAPSHOT_ID_BIT_WIDTH to 31 then user have to update getMaxSnapshotID() from -1 to -2.

szetszwo · 2020-03-12T23:45:24Z

Let's keep it "-1" since we are using 28 for the moment. If there still a problem later on, we should think about what to do. We do not necessarily change it to 31 at that time.

karthikhw · 2020-03-13T00:05:46Z

Thank you @szetszwo I changed back to -1.

szetszwo · 2020-03-16T00:15:05Z

+1 the latest change looks good. Thanks @karthikhw

lokeshj1703

@karthikhw Thanks for working on this! The changes look good to me. +1.

lokeshj1703 · 2020-03-24T09:47:09Z

@karthikhw Thanks for the contribution! @jojochuang @mukul1987 @arp7 @szetszwo Thanks for reviewing the PR! I have committed it to master branch.

shenxingwuying · 2020-05-09T06:32:58Z

Submitted new PR with the changes requested.

@karthikhw

At the beginning, SNAPSHOT_ID_BIT_WIDTH was setted 31, but later was setted 28.

Why change SNAPSHOT_ID_BIT_WIDTH from 31 to 28? I think maybe:

because of the lots of unittest cases failed ?
SNAPSHOT_ID_BIT_WIDTH=31 will make java oom? (Now I have not understand why 31 make java oom but 28 not)

(cherry picked from commit 5250cd6) Change-Id: Ibf48916c28f35e866d8b441af65de1a0b92b1733 (cherry picked from commit 20ea94d4a940cef35f0ff873dfaea19c6e5a7b83)

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit

15929f6

arp7 reviewed Mar 2, 2020

View reviewed changes

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java Outdated Show resolved Hide resolved

Set SNAPSHOT_ID_BIT_WIDTH to 28

558f954

jojochuang reviewed Mar 3, 2020

View reviewed changes

...-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java Show resolved Hide resolved

mukul1987 reviewed Mar 3, 2020

View reviewed changes

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java Outdated Show resolved Hide resolved

karthikhw and others added 2 commits March 4, 2020 13:12

Review fix

a189ef1

Update TestSnapshotManager.java

3f6830c

karthikhw commented Mar 4, 2020

View reviewed changes

...doop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java Outdated Show resolved Hide resolved

arp7 approved these changes Mar 9, 2020

View reviewed changes

Changed -1 from -2 for getMaxSnapshotID

b9f05e2

lokeshj1703 approved these changes Mar 24, 2020

View reviewed changes

lokeshj1703 merged commit 5250cd6 into apache:trunk Mar 24, 2020

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (apache#1870)

545d5d9

zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (apache#1870)

5ffcdde

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit #1870

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit #1870

karthikhw commented Mar 2, 2020 •

edited by lokeshj1703

Loading

hadoop-yetus commented Mar 2, 2020

karthikhw commented Mar 2, 2020

hadoop-yetus commented Mar 3, 2020

jojochuang commented Mar 3, 2020

jojochuang commented Mar 3, 2020

karthikhw left a comment

hadoop-yetus commented Mar 5, 2020

karthikhw commented Mar 5, 2020

karthikhw commented Mar 9, 2020

arp7 left a comment

szetszwo commented Mar 10, 2020

karthikhw commented Mar 10, 2020

szetszwo commented Mar 12, 2020

karthikhw commented Mar 13, 2020

szetszwo commented Mar 16, 2020

lokeshj1703 left a comment

lokeshj1703 commented Mar 24, 2020

shenxingwuying commented May 9, 2020 •

edited

Loading

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit #1870

HDFS-15201 SnapshotCounter hits MaxSnapshotID limit #1870

Conversation

karthikhw commented Mar 2, 2020 • edited by lokeshj1703 Loading

hadoop-yetus commented Mar 2, 2020

karthikhw commented Mar 2, 2020

hadoop-yetus commented Mar 3, 2020

jojochuang commented Mar 3, 2020

jojochuang commented Mar 3, 2020

karthikhw left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 5, 2020

karthikhw commented Mar 5, 2020

karthikhw commented Mar 9, 2020

arp7 left a comment

Choose a reason for hiding this comment

szetszwo commented Mar 10, 2020

karthikhw commented Mar 10, 2020

szetszwo commented Mar 12, 2020

karthikhw commented Mar 13, 2020

szetszwo commented Mar 16, 2020

lokeshj1703 left a comment

Choose a reason for hiding this comment

lokeshj1703 commented Mar 24, 2020

shenxingwuying commented May 9, 2020 • edited Loading

karthikhw commented Mar 2, 2020 •

edited by lokeshj1703

Loading

shenxingwuying commented May 9, 2020 •

edited

Loading