Flaky tests in TestPhoenixConnectorTest#testRowLevelDelete: NotServingRegionException: hbase:meta,,1.1588230740 is closing

[joshthoward]

TestPhoenixConnectorTest#testRowLevelDelete is flaky.

Error:  Tests run: 229, Failures: 1, Errors: 0, Skipped: 54, Time elapsed: 471.792 s <<< FAILURE! - in TestSuite
Error:  io.trino.plugin.phoenix5.TestPhoenixConnectorTest.testRowLevelDelete  Time elapsed: 1.12 s  <<< FAILURE!
java.lang.RuntimeException: 
Failed after attempts=4, exceptions:
2021-10-26T16:57:21.382Z, RpcRetryingCaller{globalStartTime=1635267441379, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1.1588230740 is closing
	at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8455)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2972)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2967)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2961)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2642)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2569)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42274)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

[joshthoward]

Hit by #9715

[kokosing]

Failed after attempts=4, exceptions:

Can we try to increase the number of attempts?

[losipiuk]

Different test method but same cause: https://github.com/trinodb/trino/runs/4328474265?check_suite_focus=true

[ebyhr]

I guess the number of attempts comes from the value of HBASE_CLIENT_RETRIES_NUMBER (=hbase.client.retries.number) in TestingPhoenixServer. I will send a PR.

(hbase.client.retries.number * hbase.client.serverside.retries.multiplier) + ConnectionUtils.retries2Attempts increment

• hbase.client.retries.number is 1 in TestingPhoenixServer
• hbase.client.serverside.retries.multiplier is 3 by default

[findepi]

Thanks @ebyhr for looking into this!

can this issue be closed now?

[ebyhr]

@findepi I think so. Thanks for the reminder.