speeds up fate lock acquisition #5262
Conversation
Stores the lock data for fate locks in the zookeeper node name instead of the zookeeper data for the node. Ran some local performance test with hundreds of fate operations and saw lock times go from 750ms to 15ms. fixes apache#5181
| } | ||
| } | ||
|
|
||
| // TODO change data arg from byte[] to String.. in the rest of the code its always a String. |
There was a problem hiding this comment.
I will open a follow on issue for this and remove the TODO before merging.
There was a problem hiding this comment.
Removed the TODO and #5264. In the TODO I was thinking of using String, but realized using more concrete types would be better.
| List<FateLock.FateLockNode> lockNodes = | ||
| FateLock.validateAndWarn(fLockPath, zr.getChildren(fLockPath.toString())); | ||
|
|
||
| lockNodes.sort(Comparator.comparingLong(ln -> ln.sequence)); |
There was a problem hiding this comment.
Should the return type be a sorted set instead of a list that you sort later?
There was a problem hiding this comment.
Updated to a sorted set in 3f25ce0
core/src/main/java/org/apache/accumulo/core/fate/zookeeper/DistributedReadWriteLock.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/accumulo/core/fate/zookeeper/FateLock.java
Outdated
Show resolved
Hide resolved
…eLock.java Co-authored-by: Christopher Tubbs <[email protected]>
…tributedReadWriteLock.java Co-authored-by: Christopher Tubbs <[email protected]>
|
|
||
| List<TabletId> tabletIds; | ||
| // start a compaction on each tablet | ||
| try (var tablets = client.tableOperations().getTabletInformation(table1, new Range())) { |
There was a problem hiding this comment.
I wrote this test initially to check performance, but realized its a good test of many compaction on the same table w/ different config also.
I did not initially see a performance difference w/ this test. Looking into it the reason was the compaction were finishing too quickly, so only 8 or 9 compaction fate operations were active at a time. For the performance bug to be observed needed many more active fate operations. Locally I modified the test to kill the compactor processes and start 1K compaction fate operations. With those changes I could see orders of magnitude diffs for fate lock acquisition time.
| Preconditions.checkArgument(nodeName.startsWith(PREFIX) && nodeName.charAt(len - 11) == '#', | ||
| "Illegal node name %s", nodeName); | ||
| sequence = Long.parseLong(nodeName.substring(len - 10)); | ||
| lockData = nodeName.substring(PREFIX.length(), len - 11); |
There was a problem hiding this comment.
So, while this is probably unlikely to happen, the sequential numbering can overflow causing a negative to occur, which I think would cause a problem with this parsing instead of splitting on #.
There was a problem hiding this comment.
Added handing and test for negative seq numbers in 2309b9b
Stores the lock data for fate locks in the zookeeper node name instead of the zookeeper data for the node. Ran some local performance test with hundreds of fate operations and saw lock times go from 750ms to 15ms.
fixes #5181