Implement Stork

Fixes #40
edeandrea · May 18, 2022 · 26388e5 · 26388e5
1 parent 2e98810
commit 26388e5
Show file tree

Hide file tree

Showing 16 changed files with 582 additions and 387 deletions.
diff --git a/rest-fights/README.md b/rest-fights/README.md
@@ -10,6 +10,9 @@
     - [Retries](#retries)
         - [Hero Client](#hero-client)
         - [Villain Client](#villain-client)
+- [Service Discovery and Load Balancing](#service-discovery-and-client-load-balancing)
+    - [Service Discovery](#service-discovery)
+    - [Client-side Load Balancing](#client-side-load-balancing)
 - [Testing](#testing) 
 - [Running the Application](#running-the-application)
 - [Running Locally via Docker Compose](#running-locally-via-docker-compose)
@@ -66,6 +69,23 @@ The [`VillainClient`](src/main/java/io/quarkus/sample/superheroes/fight/client/V
 - The downstream [Villain service](../rest-villains) returns a `404` if no random [`Villain`](src/main/java/io/quarkus/sample/superheroes/fight/client/Villain.java) is found. `VillainClient` handles this case and simulates the service returning nothing.
 - In the event the downstream [Villain service](../rest-heroes) returns an error, `VillainClient` adds 3 retries with a 200ms delay between each retry.
 
+## Service Discovery and Client Load Balancing
+The fight service implements service discovery and client-side load balancing when making downstream calls to the [`rest-heroes`](../rest-heroes) and [`rest-villains`](../rest-villains) services. The service discovery is implemented in Quarkus using [SmallRye Stork](https://quarkus.io/blog/smallrye-stork-intro).
+
+Stork [integrates directly with the Quarkus REST Client Reactive](http://smallrye.io/smallrye-stork/1.1.0/quarkus). This means that there is no additional code needed in order to take advantage of Stork's service discovery and client-side load balancing.
+
+> You could disable Stork completely for the `HeroRestClient` by setting `quarkus.rest-client.hero-client.url` to any non-Stork URL (i.e. something that doesn't start with `stork://`). Similarly, you could disable Stork completely for the `VillainClient` by setting `fight.villain.client-base-url` to any non-Stork URL.
+
+### Service Discovery
+In local development mode, as well as when running via Docker Compose, SmallRye Stork is configured using [static list discovery](https://github.com/smallrye/smallrye-stork/blob/main/docs/service-discovery/static-list.md). In this mode, the downstream URLs are statically defined in an address list. In [`application.properties`](src/main/resources/application.properties), see the `quarkus.stork.hero-service.service-discovery.address-list` and `quarkus.stork.villain-service.service-discovery.address-list` properties.
+
+When [running in Kubernetes](https://quarkus.io/blog/stork-kubernetes-discovery), Stork is configured to use the [Kubernetes Service Discovery](http://smallrye.io/smallrye-stork/1.1.0/kubernetes). In this mode, Stork will read the Kubernetes `Service`s for the [`rest-heroes`](../rest-heroes) and [`rest-villains`](../rest-villains) services to obtain the instance information. Additionally, the instance information has been configured to refresh every minute. See the `rest-fights-config` ConfigMap in [the Kubernetes deployment descriptors](deploy/k8s). Look for the `quarkus.stork.*` properties within the various `ConfigMap`s.
+
+All of the other Stork service discovery mechanisms ([Consul](http://smallrye.io/smallrye-stork/1.1.0/consul) and [Eureka](http://smallrye.io/smallrye-stork/1.1.0/eureka)) can be used simply by updating the configuration appropriately according to the Stork documentation.
+
+### Client-Side Load Balancing
+In all cases, the default load balancing algorithm used is [round robin](http://smallrye.io/smallrye-stork/1.1.0/round-robin). All of the other load balancing algorithms ([random](http://smallrye.io/smallrye-stork/1.1.0/random), [least requests](http://smallrye.io/smallrye-stork/1.1.0/least-requests), [least response time](http://smallrye.io/smallrye-stork/1.1.0/response-time), and [power of two choices](http://smallrye.io/smallrye-stork/1.1.0/power-of-two-choices)) are available on the application's classpath, so feel free to play around with them by updating the configuration appropriately according to the Stork documentation.
+
 ## Testing
 This application has a full suite of tests, including an [integration test suite](src/test/java/io/quarkus/sample/superheroes/fight/rest/FightResourceIT.java). 
 - The test suite uses [Wiremock](http://wiremock.org/) for [mocking http calls](https://quarkus.io/guides/rest-client-reactive#using-a-mock-http-server-for-tests) (see [`HeroesVillainsWiremockServerResource`](src/test/java/io/quarkus/sample/superheroes/fight/HeroesVillainsWiremockServerResource.java)) to the downstream [Hero](../rest-heroes) and [Villain](../rest-villains) services.
@@ -92,8 +112,8 @@ By default, the application is configured with the following:
 | Database password        | `QUARKUS_MONGODB_CREDENTIALS_PASSWORD`                        | `quarkus.mongodb.credentials.password`                        | `superfight`                             |
 | Kafka Bootstrap servers  | `KAFKA_BOOTSTRAP_SERVERS`                                     | `kafka.bootstrap.servers`                                     | `PLAINTEXT://localhost:9092`             |
 | Apicurio Schema Registry | `MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL` | `mp.messaging.connector.smallrye-kafka.apicurio.registry.url` | `http://localhost:8086/apis/registry/v2` |
-| Heroes Service URL       | `quarkus.rest-client.hero-client.url`                         | `quarkus.rest-client.hero-client.url`                         | `http://localhost:8083`                  |
-| Villains Service URL     | `fight.villain.client-base-url`                               | `fight.villain.client-base-url`                               | `http://localhost:8084`                  |
+| Heroes Service URL       | `QUARKUS_REST_CLIENT_HERO_CLIENT_URL`                         | `quarkus.rest-client.hero-client.url`                         | `stork://hero-service`                   |
+| Villains Service URL     | `FIGHT_VILLAIN_CLIENT_BASE_URL`                               | `fight.villain.client-base-url`                               | `stork://villain-service`                |
 
 ## Running Locally via Docker Compose
 Pre-built images for this application can be found at [`quay.io/quarkus-super-heroes/rest-fights`](https://quay.io/repository/quarkus-super-heroes/rest-fights?tab=tags). 

diff --git a/rest-fights/pom.xml b/rest-fights/pom.xml
@@ -132,6 +132,46 @@
       <groupId>io.quarkus</groupId>
       <artifactId>quarkus-liquibase-mongodb</artifactId>
     </dependency>
+    <dependency>
+      <groupId>io.quarkus</groupId>
+      <artifactId>quarkus-smallrye-stork</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-service-discovery-static-list</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-service-discovery-kubernetes</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-service-discovery-eureka</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-service-discovery-consul</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-load-balancer-random</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-load-balancer-least-requests</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-load-balancer-least-response-time</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.smallrye.stork</groupId>
+      <artifactId>stork-load-balancer-power-of-two-choices</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.quarkus</groupId>
+      <artifactId>quarkus-kubernetes-client</artifactId>
+    </dependency>
     <dependency>
       <groupId>io.quarkus</groupId>
       <artifactId>quarkus-junit5</artifactId>

diff --git a/rest-fights/src/main/docker-compose/java11.yml b/rest-fights/src/main/docker-compose/java11.yml
@@ -16,8 +16,8 @@
       QUARKUS_LIQUIBASE_MONGODB_MIGRATE_AT_START: false
       QUARKUS_MONGODB_CREDENTIALS_USERNAME: superfight
       QUARKUS_MONGODB_CREDENTIALS_PASSWORD: superfight
-      QUARKUS_REST_CLIENT_HERO_CLIENT_URL: http://rest-heroes:8083
-      FIGHT_VILLAIN_CLIENT_BASE_URL: http://rest-villains:8084
+      QUARKUS_STORK_HERO_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-heroes:8083
+      QUARKUS_STORK_VILLAIN_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-villains:8084
       MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL: http://apicurio:8086/apis/registry/v2
     restart: on-failure
     networks:

diff --git a/rest-fights/src/main/docker-compose/java17.yml b/rest-fights/src/main/docker-compose/java17.yml
@@ -16,8 +16,8 @@
       QUARKUS_LIQUIBASE_MONGODB_MIGRATE_AT_START: false
       QUARKUS_MONGODB_CREDENTIALS_USERNAME: superfight
       QUARKUS_MONGODB_CREDENTIALS_PASSWORD: superfight
-      QUARKUS_REST_CLIENT_HERO_CLIENT_URL: http://rest-heroes:8083
-      FIGHT_VILLAIN_CLIENT_BASE_URL: http://rest-villains:8084
+      QUARKUS_STORK_HERO_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-heroes:8083
+      QUARKUS_STORK_VILLAIN_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-villains:8084
       MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL: http://apicurio:8086/apis/registry/v2
     restart: on-failure
     networks:

diff --git a/rest-fights/src/main/docker-compose/native-java11.yml b/rest-fights/src/main/docker-compose/native-java11.yml
@@ -16,8 +16,8 @@
       QUARKUS_LIQUIBASE_MONGODB_MIGRATE_AT_START: false
       QUARKUS_MONGODB_CREDENTIALS_USERNAME: superfight
       QUARKUS_MONGODB_CREDENTIALS_PASSWORD: superfight
-      QUARKUS_REST_CLIENT_HERO_CLIENT_URL: http://rest-heroes:8083
-      FIGHT_VILLAIN_CLIENT_BASE_URL: http://rest-villains:8084
+      QUARKUS_STORK_HERO_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-heroes:8083
+      QUARKUS_STORK_VILLAIN_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-villains:8084
       MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL: http://apicurio:8086/apis/registry/v2
     restart: on-failure
     networks:

diff --git a/rest-fights/src/main/docker-compose/native-java17.yml b/rest-fights/src/main/docker-compose/native-java17.yml
@@ -16,8 +16,8 @@
       QUARKUS_LIQUIBASE_MONGODB_MIGRATE_AT_START: false
       QUARKUS_MONGODB_CREDENTIALS_USERNAME: superfight
       QUARKUS_MONGODB_CREDENTIALS_PASSWORD: superfight
-      QUARKUS_REST_CLIENT_HERO_CLIENT_URL: http://rest-heroes:8083
-      FIGHT_VILLAIN_CLIENT_BASE_URL: http://rest-villains:8084
+      QUARKUS_STORK_HERO_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-heroes:8083
+      QUARKUS_STORK_VILLAIN_SERVICE_SERVICE_DISCOVERY_ADDRESS_LIST: rest-villains:8084
       MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_APICURIO_REGISTRY_URL: http://apicurio:8086/apis/registry/v2
     restart: on-failure
     networks:

diff --git a/rest-fights/src/main/java/io/quarkus/sample/superheroes/fight/client/VillainClient.java b/rest-fights/src/main/java/io/quarkus/sample/superheroes/fight/client/VillainClient.java
@@ -12,6 +12,7 @@
 import org.eclipse.microprofile.faulttolerance.CircuitBreaker;
 import org.jboss.resteasy.reactive.client.impl.UniInvoker;
 
+import io.quarkus.logging.Log;
 import io.quarkus.sample.superheroes.fight.config.FightConfig;
 
 import io.smallrye.faulttolerance.api.CircuitBreakerName;
@@ -25,50 +26,58 @@
  */
 @ApplicationScoped
 public class VillainClient {
-	private final WebTarget villainClient;
+  private final WebTarget villainClient;
 
-	public VillainClient(FightConfig fightConfig) {
-		this.villainClient = ClientBuilder.newClient()
-			.target(fightConfig.villain().clientBaseUrl())
-			.path("api/villains/");
-	}
+  public VillainClient(FightConfig fightConfig) {
+    this.villainClient = ClientBuilder.newClient()
+      .target(fightConfig.villain().clientBaseUrl())
+      .path("api/villains/");
+  }
 
-	/**
-	 * Gets a Villain from the Villain service wrapped with a recovery on a {@code 404} error. Also wrapped in a {@link CircuitBreaker}.
-	 * @return The Villain
-	 */
-	@CircuitBreaker(requestVolumeThreshold = 8, failureRatio = 0.5, delay = 2, delayUnit = ChronoUnit.SECONDS)
-	@CircuitBreakerName("findRandomVillain")
-	CompletionStage<Villain> getRandomVillain() {
-		// Want the 404 handling to be part of the circuit breaker
-		// This means that the 404 responses aren't considered errors by the circuit breaker
-		return this.villainClient.path("random")
-			.request(MediaType.APPLICATION_JSON_TYPE)
-			.rx(UniInvoker.class)
-			.get(Villain.class)
-			.onFailure(Is404Exception.IS_404).recoverWithNull()
-			.subscribeAsCompletionStage();
-	}
+  /**
+   * Gets a Villain from the Villain service wrapped with a recovery on a {@code 404} error. Also wrapped in a {@link CircuitBreaker}.
+   * @return The Villain
+   */
+  @CircuitBreaker(requestVolumeThreshold = 8, failureRatio = 0.5, delay = 2, delayUnit = ChronoUnit.SECONDS)
+  @CircuitBreakerName("findRandomVillain")
+  CompletionStage<Villain> getRandomVillain() {
+    // Want the 404 handling to be part of the circuit breaker
+    // This means that the 404 responses aren't considered errors by the circuit breaker
+    var target =this.villainClient.path("random");
+    Log.debugf("Going to make request to %s", target.getUri());
+
+    return target
+      .request(MediaType.APPLICATION_JSON_TYPE)
+      .rx(UniInvoker.class)
+      .get(Villain.class)
+      .invoke(villain -> Log.debugf("Got villain back from %s: %s", target.getUri(), villain))
+      .onFailure(Is404Exception.IS_404).recoverWithNull()
+      .subscribeAsCompletionStage();
+  }
 
-	/**
-	 * Finds a random {@link Villain}. The retry logic is applied to the result of the {@link CircuitBreaker}, meaning that retries that return failures could trigger the breaker to open.
-	 * @return A random {@link Villain}
-	 */
-	public Uni<Villain> findRandomVillain() {
-		// The CompletionState is important so that on retry the Uni re-subscribes to a new
-		// CompletionStage rather than the original one (which has already completed)
-		return Uni.createFrom().completionStage(this::getRandomVillain)
-			.onFailure().retry().withBackOff(Duration.ofMillis(200)).atMost(3);
-	}
+  /**
+   * Finds a random {@link Villain}. The retry logic is applied to the result of the {@link CircuitBreaker}, meaning that retries that return failures could trigger the breaker to open.
+   * @return A random {@link Villain}
+   */
+  public Uni<Villain> findRandomVillain() {
+    // The CompletionState is important so that on retry the Uni re-subscribes to a new
+    // CompletionStage rather than the original one (which has already completed)
+    return Uni.createFrom().completionStage(this::getRandomVillain)
+      .onFailure().retry().withBackOff(Duration.ofMillis(200)).atMost(3);
+  }
 
   /**
    * Calls hello on the Villains service.
    * @return A "hello" from Villains
    */
   public Uni<String> helloVillains() {
-    return this.villainClient.path("hello")
+    var target =this.villainClient.path("hello");
+    Log.debugf("Going to make request to %s", target.getUri());
+
+    return target
       .request(MediaType.TEXT_PLAIN_TYPE)
       .rx(UniInvoker.class)
-      .get(String.class);
+      .get(String.class)
+      .invoke(hello -> Log.debugf("Got response back from %s: %s", target.getUri(), hello));
   }
 }
diff --git a/rest-fights/src/main/java/io/quarkus/sample/superheroes/fight/rest/FightResource.java b/rest-fights/src/main/java/io/quarkus/sample/superheroes/fight/rest/FightResource.java
@@ -88,7 +88,7 @@ public Uni<Response> getFight(@Parameter(name = "id", required = true) @PathPara
 				return Response.ok(f).build();
 			})
 			.onItem().ifNull().continueWith(() -> {
-				Log.debugf("No fight found with id %d", id);
+				Log.debugf("No fight found with id %s", id);
 				return Response.status(Status.NOT_FOUND).build();
 			});
 	}

diff --git a/rest-fights/src/main/kubernetes/knative.yml b/rest-fights/src/main/kubernetes/knative.yml
@@ -1,3 +1,16 @@
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: default_view
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: default
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -10,8 +23,12 @@ metadata:
 data:
   quarkus.liquibase-mongodb.migrate-at-start: false
   quarkus.mongodb.hosts: fights-db:27017
-  quarkus.rest-client.hero-client.url: http://rest-heroes
-  fight.villain.client-base-url: http://rest-villains
+  quarkus.stork.hero-service.service-discovery.type: kubernetes
+  quarkus.stork.hero-service.service-discovery.application: rest-heroes
+  quarkus.stork.hero-service.service-discovery.refresh-period: 1M
+  quarkus.stork.villain-service.service-discovery.type: kubernetes
+  quarkus.stork.villain-service.service-discovery.application: rest-villains
+  quarkus.stork.villain-service.service-discovery.refresh-period: 1M
   kafka.bootstrap.servers: PLAINTEXT://fights-kafka:9092
   mp.messaging.connector.smallrye-kafka.apicurio.registry.url: http://apicurio:8080/apis/registry/v2
 ---

diff --git a/rest-fights/src/main/kubernetes/kubernetes.yml b/rest-fights/src/main/kubernetes/kubernetes.yml
@@ -1,3 +1,16 @@
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: default_view
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: default
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -10,8 +23,12 @@ metadata:
 data:
   quarkus.liquibase-mongodb.migrate-at-start: false
   quarkus.mongodb.hosts: fights-db:27017
-  quarkus.rest-client.hero-client.url: http://rest-heroes
-  fight.villain.client-base-url: http://rest-villains
+  quarkus.stork.hero-service.service-discovery.type: kubernetes
+  quarkus.stork.hero-service.service-discovery.application: rest-heroes
+  quarkus.stork.hero-service.service-discovery.refresh-period: 1M
+  quarkus.stork.villain-service.service-discovery.type: kubernetes
+  quarkus.stork.villain-service.service-discovery.application: rest-villains
+  quarkus.stork.villain-service.service-discovery.refresh-period: 1M
   kafka.bootstrap.servers: PLAINTEXT://fights-kafka:9092
   mp.messaging.connector.smallrye-kafka.apicurio.registry.url: http://apicurio:8080/apis/registry/v2
 ---

diff --git a/rest-fights/src/main/kubernetes/minikube.yml b/rest-fights/src/main/kubernetes/minikube.yml
@@ -1,3 +1,16 @@
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: default_view
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: default
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -10,8 +23,12 @@ metadata:
 data:
   quarkus.liquibase-mongodb.migrate-at-start: false
   quarkus.mongodb.hosts: fights-db:27017
-  quarkus.rest-client.hero-client.url: http://rest-heroes
-  fight.villain.client-base-url: http://rest-villains
+  quarkus.stork.hero-service.service-discovery.type: kubernetes
+  quarkus.stork.hero-service.service-discovery.application: rest-heroes
+  quarkus.stork.hero-service.service-discovery.refresh-period: 1M
+  quarkus.stork.villain-service.service-discovery.type: kubernetes
+  quarkus.stork.villain-service.service-discovery.application: rest-villains
+  quarkus.stork.villain-service.service-discovery.refresh-period: 1M
   kafka.bootstrap.servers: PLAINTEXT://fights-kafka:9092
   mp.messaging.connector.smallrye-kafka.apicurio.registry.url: http://apicurio:8080/apis/registry/v2
 ---

diff --git a/rest-fights/src/main/kubernetes/openshift.yml b/rest-fights/src/main/kubernetes/openshift.yml
@@ -1,3 +1,16 @@
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: default_view
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: default
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -10,8 +23,12 @@ metadata:
 data:
   quarkus.liquibase-mongodb.migrate-at-start: false
   quarkus.mongodb.hosts: fights-db:27017
-  quarkus.rest-client.hero-client.url: http://rest-heroes
-  fight.villain.client-base-url: http://rest-villains
+  quarkus.stork.hero-service.service-discovery.type: kubernetes
+  quarkus.stork.hero-service.service-discovery.application: rest-heroes
+  quarkus.stork.hero-service.service-discovery.refresh-period: 1M
+  quarkus.stork.villain-service.service-discovery.type: kubernetes
+  quarkus.stork.villain-service.service-discovery.application: rest-villains
+  quarkus.stork.villain-service.service-discovery.refresh-period: 1M
   kafka.bootstrap.servers: PLAINTEXT://fights-kafka:9092
   mp.messaging.connector.smallrye-kafka.apicurio.registry.url: http://apicurio:8080/apis/registry/v2
 ---