@Asynchronous Response in context of FaultTolerance hangs infinitely

[dalexandrov]

Environment Details

• Helidon Version: 3.1.1
• Helidon MP
• JDK version: 17
• OS: MacOSX or Linux
• Docker version (if applicable):

---

A RestClient:

@RegisterRestClient(baseUri = "https://localhost:8080/entityFT", configKey = "EntityRestClientFT")
@RegisterProvider(EntityClientExceptionMapperFT.class)
public interface EntityRestClientFT {
  
    @Path("asyncHandler")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Asynchronous
    CompletionStage<List<TestEntity>> getAllEntitiesAsync();

Bean that Injects the RestClient (Apologies for so many println statements, added only to troubleshoot this further):

@Path("/entityBeanFT")
@ApplicationScoped
public class EntityBeanFT {

    @Inject
    @RestClient
    EntityRestClientFT client;

    @Path("asyncHandler")
    @GET
    public CompletionStage<List<TestEntity>> getAllEntitiesAsync() throws Exception {
        final AtomicReference<Throwable> throwable = new AtomicReference<>();
        List<TestEntity> entities = new ArrayList<>();
        BiConsumer<List<TestEntity>, Throwable> consumer = (r, t) -> {
            if (t != null) {
                throwable.set(t);
            }
            else {
                entities.addAll(r);
            }
        };
        CompletionStage<List<TestEntity>> cs = client.getAllEntitiesAsync()
                .whenCompleteAsync(consumer);
        CompletableFuture<List<TestEntity>> cf = cs.toCompletableFuture();
        try {
            cf.join();
        }
        catch (Exception e) {
            System.out.println("Exception happened!" + e.getStackTrace());
        }
        if (cf.isDone()) {
            try {
                List<TestEntity> el = cf.get();
                System.out.println("Completed successfully!");
                for (TestEntity te : el) {
                    System.out.println(te);
                }
            }
            catch (Exception e) {
                System.out.println("Exception happened!" + e.getStackTrace());
            }
        }
        return cf;
    }
..........

Exception Mapper:

public class EntityClientExceptionMapperFT implements ResponseExceptionMapper<EntityException> {

    @Override
    public EntityException toThrowable(Response response) {
        if (response.getStatus() >= 400 && response.getStatus() < 500) {
            return new EntityRequestException("Encountered Request error", response);
        }
        else if (response.getStatus() >= 500) {
            return new EntityEndpointException("Bad implementation for entity!", response);
        }
        return null;
    }

}

Service or Controller:

@Path("/entityFT")
@ApplicationScoped
public class EntityControllerFT {
private AtomicLong asyncCounter = new AtomicLong(0);

@Path("asyncHandler")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response asyncHandler() {
        long counter = asyncCounter.getAndIncrement();
        System.out.println("Counter is " + counter);
        if (counter % 2 == 0) {
            return ok(this.entityRepository.all()).build();
        }
        else {
            return serverError().build();
        }
    }

The service alternately hands over an erroneous response and valid response.
Curl command used to invoke the client Bean: curl -v http://localhost:8090/entityBeanFT/asyncHandler
I am noticing that after the 1st call (which is successful), the 2nd one (expected to be erroneous), just waits or hangs indefinitely with no result. Also, tried to use @Retry annotation with @asynchronous to recover from exception, but the calls just hang forever. Tried to use and follow as documented @ https://download.eclipse.org/microprofile/microprofile-fault-tolerance-3.0/microprofile-fault-tolerance-spec-3.0.html#_asynchronous_usage, but could not get it working.

Reproducer:

quickstart-mp-server.zip
quickstart-mp.zip

[spericas]

I'm not sure what was the intent of method getAllEntitiesAsync() and all that code there. In any case, I spent time reviewing (and simplifying) the attached server and client applications. After some analysis, the issue boils down to the use of @Asynchronous in a RestClient interface. There's really no need to do that since the RestClient specification states that any method that returns a CompletionStage is considered async.

https://download.eclipse.org/microprofile/microprofile-rest-client-2.0/microprofile-rest-client-spec-2.0.html#_asynchronous_methods

No need to use a FT annotation there, and it seems that doing so creates some strange "future" interactions (that I didn't analyze in great detail). I'm attaching a simplified version of the client app with changes that seems to work fine.

quickstart-mp-working.zip

Please verify and possibly close issue.

[nehamunjal]

Thanks! The whole idea of so much code getAllEntitiesAsync() had was only to troubleshoot because of the hangs seen. Also, I wanted to use the Asynchronous annotation in conjunction with @retries. So the endpoint alternates between a 200 response and 500 response status code that gets translated to GreetResourceEndpointException as we have registered the exception mapper GreetClientExceptionMapper as a provider to the Rest Client. We expect a retry to happen in case of GreetResourceEndpointException that should give back a 200 response (Ref to commented @Retry annotation in the RestClient method CompletionStage getMessageAsync();

The method planned to use with FT annotation is:

@Path("asyncHandler") @GET @Retry(maxRetries = 1, delay = 1, delayUnit = ChronoUnit.SECONDS, maxDuration = 5, durationUnit = ChronoUnit.SECONDS, retryOn = GreetResourceEndpointException.class) CompletionStage<Response> getMessageAsync();
I don't see the Retry happening with Asynchronous processing. Not sure if I am missing something here, but had initiated the thread to get pointers here.

[nehamunjal]

Using this as reference here: https://download.eclipse.org/microprofile/microprofile-fault-tolerance-3.0/microprofile-fault-tolerance-spec-3.0.html#_interactions_when_returning_a_code_completionstage_code

``If an exceptionally completed CompletionStage is returned, or if an incomplete CompletionStage is returned which later completes exceptionally, then this will cause other specified Fault Tolerance policies to be applied.''

[spericas]

@nehamunjal I get the idea behind the retries now. I will check that, but I can see what may be happening: with @Asynchronous there's a problem between FT and RestClient, without it, FT possibly does not regard the method as Async and fails to execute the retry logic correctly.

[spericas]

I can confirm there's a problem with @Asynchronous support in FT and RestClient. Even when it seems to work in the example, there are issues. At first glance, it seems that the RestClient proxy class calls the FT interceptor (all interceptors actually) and FT logic then calls the proxy method again as part of its async logic, entering a loop that uses all the threads in the FT thread pool and then blocks. More investigation coming ...

[spericas]

Follow-up is issue #6696.