FISH-796 Fix clustered singleton bugs and added test

[lprimak]

Description

As I upgraded the code to Hazelcast 4, one of my tests was clustered singleton.
In the process of doing this, I found a few bugs, and re-added test

Important Info

Relates to #5014 and #5013

Testing

New tests

added back tests in Payara root to test clustered singleton

Testing Performed

automated and manual tests, as well as suite

Testing Environment

JDK 8, Mac

Notes for Reviewers

The PR seems big, but it's mostly new test files added

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[dmatej]

I'm curious if this would help my current issue with Hazelcast vs. docker node start, where is some race condition blocking the docker node instance start for one minute, because hazelcast on das starts some migration before node finishes it's startup -> it fails again and again each 5 seconds, after one minute it gives up and node finishes startup. Whole test finishes in cca 135 seconds.
This situation is 5 times more often than finishing test in cca 68 seconds where is no blocking.

[lprimak]

Not sure, but my gut says no, but Hazelcast 4.1 migration could

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[lprimak]

jenkins test

[MattGill98]

Looks broadly good, just a couple of questions

[lprimak]

jenkins test

[lprimak]

jenkins test

[Pandrex247]

I haven't gone through it with anything like a fine-toothed comb, but the tests pass and no obvious purple elephants from a skim over the code.

[AndrewG10i]

Hello there! @lprimak may I kindly share with you some early feedback as per this fix.

Yesterday I have downloaded the new release of Payara Micro 5.2021.1 (which includes this fix) and started with testing persistent schedules functionality (we use both ways of creating them with @Schedule and via TimerService), as from the fix topic it seems that persistent timers are going to be affected.

Please see below my findings considering the Payara Micro Community documentation for persistent timers -> How EJB timers are persisted.

In my tests we are considering two scenarios (by using cluster with two nodes launched with arguments: --name node1 and --name node2):

Scenario 1: instance (node1) that created persistent timer goes DOWN.

In pre-5.2021.1 versions (e.g. 5.2020.7):
The following statement from the documentation:
"If that instance goes down, the timer will be recreated on another instance with the same name once it joins the cluster."
actually worked as:
"If that instance goes down, the timer will be recreated (running) on another instance in the cluster."
So timer was actually migrating to the another node (node2) in the cluster and keep running on it. Yep, we were aware that it didn't follow the documentation but that behavior we actually liked: we need a timer running on the single node in the cluster with automatic failover to another node in case the one which created it goes down.

Since 5.2021.1 seems that was fixed, and now persistent timer doesn't run in the cluster (what basically corresponds to the documentation "Until that time, the timer becomes inactive.").
Just to note here that actually my tests shown that persistent timer runs once on the another node (node2) in the cluster, but after that single run - node2 shows message like:
[2021-02-11T13:16:34.957+0800] [] [INFO] [] [javax.enterprise.system.container.ejb.fish.payara.ejb.timer.hazelcast] The timer (1@@1613020572690@@node1@@clusterDev) is now owned by (node1). Removing from local cache

Seems all works in compliance with the documentation (spec) now, except that timer managed to run once on another node, but okay, in general - no action needed.

Scenario 2: instance (node1) that created persistent timer goes UP after it was down.

In pre-5.2021.1 versions (e.g. 5.2020.7) persistent timer was keep running on the another node (node2).

As per 5.2021.1: According to the documentation the following should happen: "If that instance goes down, the timer will be recreated on another instance with the same name once it joins the cluster."
But in my tests once I launch again node1 (which has originally created the timer) - timer is not re-created on it. Basically I do not see timer running at all in the cluster anymore.

This is our preliminary observations on the persistent timers behavior in Payara Micro 5.2021.1 with the fix according to this ticket.
I would like to emphasize that currently we are not registering separate issue for this as we are still working on checking and reconfirming our observations, and also adapting our application to the new behavior of persistent timers (as we need to migrate to the new release due to fixes for the other bugs).

We will do our best to prepare and publish test applications and test scenarios for demonstrating the reported behavior and once it is done and fully confirmed in our tests - we will register it as a separate issue.

Thank you!