FISH-105 Migrate EJB Timers from Live Instances

[Pandrex247]

Description

This is a new feature or improvement.

This change allows you to migrate EJB timers from live instances, rather than only down ones. This is done using the migrate-timers command.

Important Info

Blockers

None

Testing

New tests

None

Testing Performed

Database
From a clean domain:

• Configure __TimerPool JDBC Connection Pool to point to a non-relative location by editing the sole existing property to something like C:\Users\pandr\Downloads\Test rather than ${com.sun.aas.instanceRoot}/lib/databases/ejb-timer
• Create a Deployment Group with two instances and start them
• Deploy test app to the deployment group: https://github.com/payara/patched-src-javaee7-samples/tree/master/ejb/timer
• Check the instance logs to ensure the EJB Timer service started and that timers are executing
• Run the following command to migrate timers, using your own instance names: asadmin migrate-timers --target Insty1 Insty2
  • For clarification, target is where the timers are being migrated to, the primary parameter (in the above case Insty2) is where they're being migrated from
• Check the logs to ensure timers are no longer executing on Insty2 (should see some messages of timers no longer being owned by it), and that they're executing on Insty1
• Run the following command to move them back: asadmin migrate-timers --target Insty2 Insty1
• Check logs...
• And migrate again...: asadmin migrate-timers --target Insty1 Insty2
• Check logs...
• Migrate again without giving a target to let the DAS decide: asadmin migrate-timers Insty1
• Check logs...
• Kill the instance where the timers currently live to check automatic migration (they should be on Insty2)
• Check logs...

Hazelcast
From a clean domain:

• Create a Deployment Group with two instances and don't start them
• Edit the configs of both instances and the DAS to use DataGrid for their EJB Timers (configs > EJB Container > EJB Timers)
• Start instances and restart the DAS
• Deploy test app to the deployment group: https://github.com/payara/patched-src-javaee7-samples/tree/master/ejb/timer
• Check the instance logs to ensure the EJB Timer service started and that timers are executing
• Run the following command to migrate timers, using your own instance names: asadmin migrate-timers --target Insty1 Insty2
  • For clarification, target is where the timers are being migrated to, the primary parameter (in the above case Insty2) is where they're being migrated from
• Check the logs to ensure timers are no longer executing on Insty2 (should see some messages of timers no longer being owned by it), and that they're executing on Insty1
• Run the following command to move them back: asadmin migrate-timers --target Insty2 Insty1
• Check logs...
• And migrate again...: asadmin migrate-timers --target Insty1 Insty2
• Check logs...
• Migrate again without giving a target to let the DAS decide: asadmin migrate-timers Insty1
• Check logs...
• Kill the instance where the timers currently live to check automatic migration (they should be on Insty2)
• Check logs...

Testing Environment

Windows 10, Zulu JDK 8u275

Documentation

Pending...

Notes for Reviewers

There are separate issues pertaining to EJB timers which you may encounter, which as far as I can tell are unrelated to these changes. These are:

• The list-timers asadmin command is woeful, particularly when it comes to Hazelcast.
• Particularly when using Hazelcast it seems to be possible for the timers in the test app to register twice, sometimes "half" registering to one instance whilst "fully" registering to the second, causing NPEs whenever they trigger on the "half" registered instance.
  • To properly notice this, you will need to change the output string of the timers in the linked test app, since most of them output the same text.
  • When you start migrating timers it fixes the NPE, but it still seems to trigger twice - presumably this lets the instances figure out where the timers are (but you still have a duplicate).

Summary of changes:
Not much seemed to be necessary to get timer migration to work against the traditional database - just removing the check for if the instances were alive seemed to be enough! When the now migrated timer triggers on the original instance (where it should no longer be), it detects that it no longer owns the timer and removes it from its local cache.

Against a DataGrid proved a bit more troublesome, due to the "half-local, half distributed" nature of it. I settled on pinging an event across the Hazelcast DataGrid for the EJB Timer service to listen to, purging the local timer cache of the instance from whence they came.

[Pandrex247]

Jenkins test please