2.4.1 causes strange HTTP failure in devmode #21282

snowe2010 · 2021-11-08T17:06:21Z

Describe the bug

When starting this example project, I receive repeating error messages, over and over again (without doing anything, literally just starting it up). The project was generated with quarkus.io, with the only modifications made to remove/add dependencies to duplicate the error.

2021-11-05 12:29:05,388 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-0) HTTP Request to /_lambda_/2018-06-01/runtime/init/error failed, error id: ba255bb1-447f-4e17-8e68-dacb2e2e42b6-1: java.util.concurrent.RejectedExecutionException
        at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
        at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2076)

It appears to happen as soon as I enable the kogito-quarkus dependency.

Expected behavior

Dev mode should start without errors

Actual behavior

dev mode starts and throws a continuous stream of the above error.

How to Reproduce?

demo1.zip

Start with mvn clean quarkus:dev
Enable/disable kogito-quarkus dependency

Output of `uname -a` or `ver`

Darwin SR-MB-300245 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

Output of `java -version`

java version is specified using asdf in the .tool-versions file: java graalvm-21.0.0+java11

GraalVM version (if different from Java)

same as above. specified using asdf java graalvm-21.0.0+java11

Quarkus version or git rev

2.4.1

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

also specified using asdf. maven 3.6.3

Additional information

logs-1636390731605.txt

The text was updated successfully, but these errors were encountered:

geoand · 2021-11-08T18:41:30Z

This has nothing to do with kogito-quarkus , you can get a similar problem by just adding quarkus-vertx-web.
When I was trying this, I consistently got:

ERROR [io.qua.run.Application] (Quarkus Main Thread) Port 8080 seems to be in use by another process. Quarkus may already be running or the port is used by another application.

This seems to be due to the fact that io.quarkus.amazon.lambda.runtime.MockEventServer is being started on port 8080 because of https://github.com/quarkusio/quarkus/blob/2.4.1.Final/extensions/amazon-lambda/common-deployment/src/main/java/io/quarkus/amazon/lambda/deployment/MockEventServerConfig.java#L15.

Now I have no idea how this is meant to work, so this is one for @patriot1burke

quarkus-bot · 2021-11-08T18:41:42Z

/cc @matejvasek, @patriot1burke

snowe2010 · 2021-11-08T21:18:25Z

@geoand great! I thought I had duplicated it with the vertx-web dependency as well, but then disabled it and it was still happening and had traced it to kogito-quarkus. Glad you were able to track down the real issue.

snowe2010 · 2021-11-10T16:58:52Z

In the meantime, is there anyway to 'unconfigure' that port?

patriot1burke · 2021-11-10T19:29:09Z

@geoand So, you cannot mix amazon-lambda plugin with quarkus-vertx-web. The Amazon Lambda dev service starts a MockEventServer under port 8080 so that you can invoke on your lambdas using http. See the docs for more info. You shouldn't be starting a web server within a lambda endpoint anyways!!!

Not sure what's going on with the kotlin-quarkus plugin. It may be trying to start its own web server?

patriot1burke · 2021-11-10T19:32:57Z

If you want to use HTTP libraries like resteasy, vertx-web, or servlet with lambda, you need the amazon-lambda-http or amazon-lambda-rest extensions. See docs

geoand · 2021-11-10T19:37:18Z

Thanks for the info @patriot1burke .

The quarkus-kotlin extension shouldn't be starting anything, but likely the kogito extension depends on Vertx

snowe2010 · 2021-11-10T21:22:58Z

@geoand So, you cannot mix amazon-lambda plugin with quarkus-vertx-web. The Amazon Lambda dev service starts a MockEventServer under port 8080 so that you can invoke on your lambdas using http. See the docs for more info. You shouldn't be starting a web server within a lambda endpoint anyways!!!

@patriot1burke well that would have been good to know. We've been writing our own endpoints and directing them at the lambda handler itself. Where is this documented? What is the exact endpoint to hit? just localhost:8080/? What's the data format?

@geoand so where should this get directed to?

snowe2010 · 2021-11-10T21:27:01Z

oh, I see it's in the guide. I thought those were all just tutorials, not that they would contain detailed info like that...

patriot1burke · 2021-11-10T21:32:11Z

@geoand So, you cannot mix amazon-lambda plugin with quarkus-vertx-web. The Amazon Lambda dev service starts a MockEventServer under port 8080 so that you can invoke on your lambdas using http. See the docs for more info. You shouldn't be starting a web server within a lambda endpoint anyways!!!

@patriot1burke well that would have been good to know. We've been writing our own endpoints and directing them at the lambda handler itself. Where is this documented? What is the exact endpoint to hit? just localhost:8080/? What's the data format?

https://quarkus.io/guides/amazon-lambda#live-coding-and-unitintegration-testing

Data format is whatever you want to POST. This is new stuff BTW introduced in 2.4.0.

snowe2010 · 2021-11-10T21:39:48Z

@patriot1burke ah, that explains it. I thought we had tried it last year and it didn't work. Ok, thanks!

evacchi · 2021-11-11T07:26:52Z

it looks like a it is a transitive dependency for quarkus-undertow, which is being addressed at https://github.com/kiegroup/kogito-runtimes/pull/1669

[INFO] |  +- io.quarkus:quarkus-undertow:jar:2.4.0.Final:compile
[INFO] |  |  +- io.quarkus:quarkus-vertx-http:jar:2.4.0.Final:compile
[INFO] |  |  |  +- io.quarkus:quarkus-security-runtime-spi:jar:2.4.0.Final:compile
[INFO] |  |  |  +- io.quarkus:quarkus-mutiny:jar:2.4.0.Final:compile
[INFO] |  |  |  |  +- io.quarkus:quarkus-smallrye-context-propagation:jar:2.4.0.Final:compile
[INFO] |  |  |  |  |  \- io.smallrye:smallrye-context-propagation:jar:1.2.2:compile
[INFO] |  |  |  |  |     +- io.smallrye:smallrye-context-propagation-api:jar:1.2.2:compile
[INFO] |  |  |  |  |     \- io.smallrye:smallrye-context-propagation-storage:jar:1.2.2:compile
[INFO] |  |  |  |  \- io.smallrye.reactive:mutiny-smallrye-context-propagation:jar:1.1.2:compile
[INFO] |  |  |  +- io.quarkus:quarkus-vertx-http-dev-console-runtime-spi:jar:2.4.0.Final:compile
[INFO] |  |  |  +- io.quarkus:quarkus-vertx:jar:2.4.0.Final:compile
[INFO] |  |  |  |  +- io.quarkus:quarkus-netty:jar:2.4.0.Final:compile
[INFO] |  |  |  |  +- io.netty:netty-codec-haproxy:jar:4.1.68.Final:compile
[INFO] |  |  |  |  +- io.quarkus:quarkus-jackson:jar:2.4.0.Final:compile
[INFO] |  |  |  |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.12.5:compile
[INFO] |  |  |  |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.5:compile
[INFO] |  |  |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-core:jar:2.14.1:compile
[INFO] |  |  |  |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-runtime:jar:2.14.1:compile
[INFO] |  |  |  |  |  \- io.smallrye.reactive:vertx-mutiny-generator:jar:2.14.1:compile
[INFO] |  |  |  |  |     \- io.vertx:vertx-codegen:jar:4.1.5:compile
[INFO] |  |  |  |  \- io.smallrye:smallrye-fault-tolerance-vertx:jar:5.2.1:compile
[INFO] |  |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-web:jar:2.14.1:compile
[INFO] |  |  |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-web-common:jar:2.14.1:compile
[INFO] |  |  |  |  +- io.smallrye.reactive:smallrye-mutiny-vertx-auth-common:jar:2.14.1:compile
[INFO] |  |  |  |  \- io.smallrye.reactive:smallrye-mutiny-vertx-bridge-common:jar:2.14.1:compile
[INFO] |  |  |  \- io.vertx:vertx-web:jar:4.1.5:compile
[INFO] |  |  |     +- io.vertx:vertx-web-common:jar:4.1.5:compile
[INFO] |  |  |     +- io.vertx:vertx-auth-common:jar:4.1.5:compile
[INFO] |  |  |     \- io.vertx:vertx-bridge-common:jar:4.1.5:compile

I think you can exclude quarkus-undertow from kogito-quarkus in the meantime

snowe2010 · 2021-11-16T23:23:19Z

@patriot1burke Creating an app on quarkus.io with just the kotlin, lambda, and rest-client options results in an unrunnable build due to the 8080 thing. It automatically adds all the rest assured/rest easy stuff. I think that's why I encountered this issue.

snowe2010 · 2021-11-18T02:46:52Z

@patriot1burke it also appears that kogito depends on using vertx regardless, so I'm not sure how I'm supposed to remove the dependency here. Following Edoardo's suggestion just results in more and more errors, the more dependencies I exclude. A config option to disable or change this port would probably resolve the problem until it can get figured out a different way.

snowe2010 · 2021-11-23T22:26:31Z

turns out you can do so with

quarkus.lambda.mock-event-server.dev-port=8082
quarkus.lambda.mock-event-server.test-port=8083

gsmet · 2022-01-11T14:03:17Z

@patriot1burke maybe we should have a proper message error when Vert.x/RESTEasy is around and using the same port?

The other option would be to document the error message and how to solve it.

patriot1burke · 2022-01-11T22:01:02Z

@gsmet The problem is that people are using extensions that start up an HTTP server within a vanilla AWS Lambda. Some of these extensions like Kogito require an HTTP server (at least I think it requires HTTP) and should not be used within a vanilla AWS Lambda. Other extensions like quarkus-oidc-client are incorrectly pulling in the vertx-http-deployment dependency and thus starting an HTTP server.

I can have Vert.x HTTP publish either an "vertx-http" feature or a marker build item. Check within lambda processor for this and give a warning or abort with message.

snowe2010 · 2022-01-27T02:41:29Z

@patriot1burke so how are you 'supposed' to be using kogito then? It's built to run on AWS lambda is it not? And I'm glad to see that I wasn't crazy when I was seeing the same issues with OIDC. Is this all a consequence of the devmode http handler so that you can hit lambdas locally?

evacchi · 2022-01-27T08:17:59Z

Kogito was requiring undertow to serve static resources, this should be no longer the case for 1.15.0. We also automatically generate rest endpoints, which may conflict with AWS. You should be able to opt-out from automated rest generation setting to your properties:

kogito.generate.rest=false

then you should be able to try the new experimental public API https://blog.kie.org/2021/11/the-road-towards-a-kogito-public-api.html

I would also advise to check with Kogito 1.16, which should be release together with Quarkus 2.7.0, for all latest improvements.

patriot1burke · 2022-01-27T13:24:23Z

@evacchi Yeah, Kogito was pulling in quarkus-vertx-http and thus starting an HTTP endpoint.

patriot1burke · 2022-01-27T13:26:52Z

this all a consequence of the devmode http handler so that you can hit lambdas locally?

@snowe2010 Its more that this devmode change uncovered the fact that there are a few extensions that SHOULD NOT be booting up an http server and that were implemented incorrectly.

evacchi · 2022-01-27T13:27:18Z

so I think this has been addressed with https://github.com/kiegroup/kogito-runtimes/pull/1757

patriot1burke · 2022-01-27T13:30:33Z

@evacchi I don't know... grep for 'quarkus-vertx-http' or quarkus-vertx-http-deployment in your poms. Last time I looked it was still pulling this in.

evacchi · 2022-01-27T13:34:23Z

currently this is all I am finding:

./quarkus/extensions/kogito-quarkus-extension-common/kogito-quarkus-common-deployment/pom.xml:            <artifactId>quarkus-vertx-http-deployment-spi</artifactId>

which should be safe

geoand · 2022-01-27T13:36:29Z

which should be safe

👍🏼

Let's close the issue. If it's still a problem when Quarkus 2.7 and the corresponding Kogito 1.16 releases are out, we can reopem

snowe2010 added the kind/bug Something isn't working label Nov 8, 2021

quarkus-bot bot added the triage/needs-triage label Nov 8, 2021

geoand added area/amazon-lambda and removed triage/needs-triage labels Nov 8, 2021

patriot1burke mentioned this issue Nov 30, 2021

Amazon Lambda tests fail with 404, but devmode works fine #21827

Open

geoand closed this as completed Jan 27, 2022

geoand added the triage/out-of-date This issue/PR is no longer valid or relevant label Jan 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.4.1 causes strange HTTP failure in devmode #21282

2.4.1 causes strange HTTP failure in devmode #21282

snowe2010 commented Nov 8, 2021

geoand commented Nov 8, 2021 •

edited

Loading

quarkus-bot bot commented Nov 8, 2021

snowe2010 commented Nov 8, 2021

snowe2010 commented Nov 10, 2021

patriot1burke commented Nov 10, 2021

patriot1burke commented Nov 10, 2021 •

edited

Loading

geoand commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

patriot1burke commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

evacchi commented Nov 11, 2021

snowe2010 commented Nov 16, 2021

snowe2010 commented Nov 18, 2021

snowe2010 commented Nov 23, 2021

gsmet commented Jan 11, 2022

patriot1burke commented Jan 11, 2022

snowe2010 commented Jan 27, 2022 •

edited

Loading

evacchi commented Jan 27, 2022 •

edited

Loading

patriot1burke commented Jan 27, 2022

patriot1burke commented Jan 27, 2022 •

edited

Loading

evacchi commented Jan 27, 2022

patriot1burke commented Jan 27, 2022

evacchi commented Jan 27, 2022

geoand commented Jan 27, 2022 •

edited

Loading

2.4.1 causes strange HTTP failure in devmode #21282

2.4.1 causes strange HTTP failure in devmode #21282

Comments

snowe2010 commented Nov 8, 2021

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

geoand commented Nov 8, 2021 • edited Loading

quarkus-bot bot commented Nov 8, 2021

snowe2010 commented Nov 8, 2021

snowe2010 commented Nov 10, 2021

patriot1burke commented Nov 10, 2021

patriot1burke commented Nov 10, 2021 • edited Loading

geoand commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

patriot1burke commented Nov 10, 2021

snowe2010 commented Nov 10, 2021

evacchi commented Nov 11, 2021

snowe2010 commented Nov 16, 2021

snowe2010 commented Nov 18, 2021

snowe2010 commented Nov 23, 2021

gsmet commented Jan 11, 2022

patriot1burke commented Jan 11, 2022

snowe2010 commented Jan 27, 2022 • edited Loading

evacchi commented Jan 27, 2022 • edited Loading

patriot1burke commented Jan 27, 2022

patriot1burke commented Jan 27, 2022 • edited Loading

evacchi commented Jan 27, 2022

patriot1burke commented Jan 27, 2022

evacchi commented Jan 27, 2022

geoand commented Jan 27, 2022 • edited Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

geoand commented Nov 8, 2021 •

edited

Loading

patriot1burke commented Nov 10, 2021 •

edited

Loading

snowe2010 commented Jan 27, 2022 •

edited

Loading

evacchi commented Jan 27, 2022 •

edited

Loading

patriot1burke commented Jan 27, 2022 •

edited

Loading

geoand commented Jan 27, 2022 •

edited

Loading