Multi route support #638

Delawen · 2023-05-05T10:06:20Z

Backend has now an experimental v2 api that supports multiroutes. Not all endpoints are ported from v1 yet.

Integration kind is not fully supported yet. This PR focused on Camel Routes, Kamelets, and Bindings.
Mixing different DSLs on the same file is not supported.

Superseeds #633

Integration kind CRD is still not fully supported. Mixing different DSLs on the same file is not supported.

Delawen · 2023-05-05T10:06:37Z

@lordrip 👀

api/pom.xml

Delawen · 2023-05-05T10:09:41Z

api/src/main/java/io/kaoto/backend/api/resource/v2/IntegrationsResource.java

+    @Consumes(MediaType.APPLICATION_JSON)
+    @Produces("text/yaml")
+    @Path("/")
+    @CacheResult(cacheName = "api")


Caching (see above comment)

How would this cache behave? Is this the correct "doc" for this annotation?

Yes, exactly that one.

Delawen · 2023-05-05T10:14:18Z

api/src/main/java/io/kaoto/backend/api/resource/v2/IntegrationsResource.java

+            final @Parameter(description = "DSL to use. For example: "
+                    + "'Kamelet Binding'.")
+            @QueryParam("dsl") String dsl) {
+        List<Integration> integrations = new ArrayList<>();


We should move all this to some service and keep the resources classes clean, but better not add more changes, it is already very long.

codecov · 2023-05-05T10:15:09Z

Codecov Report

Merging #638 (a389ebd) into main (4c347fc) will decrease coverage by 0.96%.
The diff coverage is 67.09%.

@@             Coverage Diff              @@
##               main     #638      +/-   ##
============================================
- Coverage     78.54%   77.58%   -0.96%     
- Complexity      132      146      +14     
============================================
  Files            45       46       +1     
  Lines          2358     2503     +145     
  Branches        370      393      +23     
============================================
+ Hits           1852     1942      +90     
- Misses          339      384      +45     
- Partials        167      177      +10

Impacted Files	Coverage Δ
...arser/camelroute/IntegrationStepParserService.java	`73.17% <0.00%> (-10.17%)`	⬇️
...ployment/generator/DeploymentGeneratorService.java	`100.00% <ø> (ø)`
...end/api/service/step/parser/StepParserService.java	`50.00% <ø> (ø)`
...elroute/IntegrationDeploymentGeneratorService.java	`50.61% <10.00%> (-6.33%)`	⬇️
...kend/api/service/deployment/DeploymentService.java	`51.61% <48.00%> (-2.45%)`	⬇️
...melroute/CamelRouteDeploymentGeneratorService.java	`75.00% <66.66%> (-1.93%)`	⬇️
...ployment/generator/kamelet/KameletRepresenter.java	`82.14% <70.58%> (-3.29%)`	⬇️
.../backend/api/resource/v2/IntegrationsResource.java	`73.91% <73.91%> (ø)`
...elet/KameletBindingDeploymentGeneratorService.java	`67.79% <81.25%> (+1.77%)`	⬆️
...tor/kamelet/KameletDeploymentGeneratorService.java	`69.69% <83.33%> (+2.42%)`	⬆️
... and 4 more

Delawen · 2023-05-05T10:15:16Z

api/src/test/java/io/kaoto/backend/api/resource/v2/IntegrationsResourceTest.java

@@ -0,0 +1,393 @@
+package io.kaoto.backend.api.resource.v2;


I copied the v1 file here thinking at some point we will deprecate and remove v1 and its tests.

At the end the round trip function.

Delawen · 2023-05-05T10:15:32Z

api/src/test/java/io/kaoto/backend/api/resource/v2/IntegrationsResourceTest.java

+        var to = (Map<String, Object>) ((Map<String, Object>) steps.get(0)).get("to");
+        assertEquals("log:", to.get("uri"));
+    }
+


This is the fun part for this PR

Delawen · 2023-05-05T10:15:58Z

api/src/test/java/io/kaoto/backend/api/resource/v2/IntegrationsResourceTest.java

+
+        assertEquals(res0.extract().body().asString(), res1.extract().body().asString());
+
+        var flows = Arrays.asList(res1.extract().body().as(Integration[].class));


Note that now we return a list/array of Integrations.

Delawen · 2023-05-05T10:16:41Z

api/src/test/resources/io/kaoto/backend/api/resource/amq-amq-multi.json

@@ -0,0 +1,2798 @@
+[


JSON responses now are arrays/lists, not single objects. So I had to copy some resources files and added the multi string to distinguish.

Delawen · 2023-05-05T10:17:08Z

api/src/test/resources/io/kaoto/backend/api/resource/kamelet-multi.yaml

+            uploadMode: '{{uploadMode}}'
+            accessToken: '{{accessToken}}'
+
+---


😈 multi Kamelets on the same file MUAHAHAHA

...ckend/api/service/deployment/generator/camelroute/IntegrationDeploymentGeneratorService.java

Delawen · 2023-05-05T10:18:20Z

...ckend/api/service/deployment/generator/camelroute/IntegrationDeploymentGeneratorService.java

@@ -107,6 +121,12 @@ public boolean appliesTo(final List<Step> steps) {
                        .anyMatch(Predicate.isEqual(s.getKind().toUpperCase())));
    }

+    @Override
+    public boolean appliesToFlows(List<StepParserService.ParseResult<Step>> flows) {


If one of the flows in the list of flows is from this DSL we assume that we can handle it.

...in/java/io/kaoto/backend/api/service/step/parser/camelroute/CamelRouteStepParserService.java

Delawen · 2023-05-05T10:19:44Z

...n/java/io/kaoto/backend/api/service/step/parser/camelroute/IntegrationStepParserService.java

+    @Override
+    public List<ParseResult<Step>> getParsedFlows(String input) {
+        var res = new LinkedList<ParseResult<Step>>();
+        String[] splitCRDs = input.split(System.lineSeparator() + "---" + System.lineSeparator());


Maybe this is too weak? Not sure how to make it better, to be honest. This could be in the middle of a long string text. Evil people will do evil things, you know.

yes, haters gonna hate, I know... Would something like this help?

Let me try :)

Not really working as expected.

Delawen · 2023-05-05T10:20:32Z

kamelet-support/src/main/java/io/kaoto/backend/KamelPopulator.java

@@ -118,7 +118,7 @@ public void populateKamelet(
                def.setTitle(String.valueOf(map.getOrDefault("title", "")));
                def.setDescription(String.valueOf(map.getOrDefault("description", "")));
                def.setRequired((List<String>) map.getOrDefault("required", null));
-                def.setProperties((Map<String, KameletDefinitionProperty>) map.getOrDefault("required", null));


Caught this bug while adding more kamel examples. Oopsie.

Delawen · 2023-05-05T10:21:03Z

...ckend/api/service/deployment/generator/kamelet/KameletBindingDeploymentGeneratorService.java

@@ -104,7 +105,12 @@ public String parse(final List<Step> steps,
            spec.setSteps(null);
        }

-        KameletBinding binding = new KameletBinding(String.valueOf(metadata.getOrDefault("name", "")), spec);
+        var name = "";


Again, we were not very careful and just trusted that things were not null. Naive us.

Delawen · 2023-05-05T10:21:22Z

...ckend/api/service/deployment/generator/kamelet/KameletBindingDeploymentGeneratorService.java

+    public String parse(List<StepParserService.ParseResult<Step>> flows) {
+        StringBuilder sb = new StringBuilder();
+        flows.stream().forEachOrdered(stepParseResult -> {
+            if (!sb.isEmpty()) {


Simple but effective.

Delawen · 2023-05-05T10:21:37Z

.../main/java/io/kaoto/backend/api/service/deployment/generator/kamelet/KameletRepresenter.java

@@ -211,6 +212,36 @@ public Node representData(final Object data) {
                                DumperOptions.FlowStyle.BLOCK);
                    }
                });
+
+        this.multiRepresenters.put(KameletDefinitionProperty.class,
+                new RepresentMap() {


This was also missing, dumb us.

Delawen · 2023-05-05T10:21:59Z

.../main/java/io/kaoto/backend/api/service/deployment/generator/DeploymentGeneratorService.java

@@ -55,8 +56,20 @@ public interface DeploymentGeneratorService {
     * If applies, the name will be the name used on the integration deployed.
     */
    @WithSpan
-    String parse(List<Step> steps, Map<String, Object> metadata,
-                 List<Parameter> parameters);
+    @Deprecated


deprecated because we should discourage its use.

lordrip

LGTM 👍

Just to confirm:

1. Response shape

given the following YAML

- from:
    uri: null
    steps:
        - aggregate:
              aggregation-strategy: aggregate

- from:
    uri: null
    steps: []

- from:
    uri: null
    steps: []

we receive an array of routes, which is what I would expect to receive, but just wanted to confirm if this is correct or I should expect an object as you mentioned in another conversation?

2. YAML format

Sending 2 integrations in an array, we get the following routes:

would be possible / appropriate / useful to give an empty line between the routes? this is extremely 💅 (stylish)

api/pom.xml

lordrip · 2023-05-05T10:30:47Z

api/src/main/java/io/kaoto/backend/api/resource/v2/IntegrationsResource.java

+    @Consumes(MediaType.APPLICATION_JSON)
+    @Produces("text/yaml")
+    @Path("/")
+    @CacheResult(cacheName = "api")


How would this cache behave? Is this the correct "doc" for this annotation?

lordrip · 2023-05-05T10:37:28Z

...n/java/io/kaoto/backend/api/service/step/parser/camelroute/IntegrationStepParserService.java

+    @Override
+    public List<ParseResult<Step>> getParsedFlows(String input) {
+        var res = new LinkedList<ParseResult<Step>>();
+        String[] splitCRDs = input.split(System.lineSeparator() + "---" + System.lineSeparator());


yes, haters gonna hate, I know... Would something like this help?

lordrip · 2023-05-05T10:58:04Z

BTW, almost forgot, as soon as we get this merged, I'll switch the UI to use the new endpoints and interact only with the first route in the array. After that, I'll start incorporating individual functionality into the multi-route support.

of flows

Delawen · 2023-05-05T11:51:03Z

BTW, almost forgot, as soon as we get this merged, I'll switch the UI to use the new endpoints and interact only with the first route in the array. After that, I'll start incorporating individual functionality into the multi-route support.

You like danger 😸 Living on the edge!

Delawen · 2023-05-05T11:51:35Z

Added an object wrapper to return instead of a plain array of flows. You were right.

sonarcloud · 2023-05-05T11:51:37Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
8 Code Smells

No Coverage information
2.6% Duplication

lordrip · 2023-05-05T12:53:45Z

Alright, now it sends and receives an object like this:

{
  "flows": [ ... ],
  "properties: { ... }
}

Delawen · 2023-05-05T15:02:15Z

Adding an extra line between routes is breaking tests in an ugly way. I will leave that for a second PR.

With the new endpoints introduced by KaotoIO/kaoto-backend#638, we could start wiring up the source code editor and the related components to use the multiple flows approach. Changes This pull request replaces the previous v1/integrations endpoint with the new v2/integrations endpoint that supports multiple flows. All the changes were validated against the existing e2e tests to ensure that the previous functionality from the single flow is still in place. * chore: Fix settings.js e2e test * chore: Fix editing.cy.js e2e test * chore: Fix step_integration.cy.js e2e test * chore: Fix source_code.cy.js e2e test * chore: Fix deleting_steps.cy.js e2e test * chore: Fix catalog_actions.cy.js e2e test * chore: Fix code_editor_actions.cy.js e2e test * chore: Fix step_actions_from_canvas.cy.js e2e test * chore: Fix step_extension_actions_from_canvas.cy.js e2e test * chore: Fix branches_from_canvas.cy.js e2e test * chore: Fix branches_step_actions_from_canvas.cy.js e2e test * chore: Fix nested_branches_from_canvas.cy.js e2e test * chore: Fix nested_branches_step_actions_from_canvas.cy.js e2e test relates to: #801

lordrip and others added 2 commits May 3, 2023 13:19

Initial draft

634710f

feature: Support for multi-routes.

fc727ce

Integration kind CRD is still not fully supported. Mixing different DSLs on the same file is not supported.

Delawen requested a review from a team May 5, 2023 10:06

Delawen requested a review from lordrip May 5, 2023 10:06

Delawen mentioned this pull request May 5, 2023

Draft - Multiple routes support #633

Closed

Delawen commented May 5, 2023

View reviewed changes

api/pom.xml Show resolved Hide resolved

Delawen commented May 5, 2023

View reviewed changes

...ckend/api/service/deployment/generator/camelroute/IntegrationDeploymentGeneratorService.java Show resolved Hide resolved

Delawen commented May 5, 2023

View reviewed changes

...in/java/io/kaoto/backend/api/service/step/parser/camelroute/CamelRouteStepParserService.java Show resolved Hide resolved

Delawen commented May 5, 2023

View reviewed changes

fix: Removing @Inject from native test

0deb218

lordrip approved these changes May 5, 2023

View reviewed changes

feature: Return a wrapper object for flows instead of a plain array/list

a389ebd

of flows

lordrip self-requested a review May 5, 2023 12:53

lordrip approved these changes May 5, 2023

View reviewed changes

Delawen enabled auto-merge (rebase) May 5, 2023 14:59

lordrip mentioned this pull request May 7, 2023

feat(multiple-flows): Enable experimental support for multiple flows KaotoIO/kaoto-ui#1749

Merged

Delawen merged commit efa5105 into KaotoIO:main May 8, 2023

This was referenced May 19, 2023

Support multiple routes #502

Closed

feat/wire-up new backend endpoints to support multiple flows KaotoIO/kaoto-ui#1823

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi route support #638

Multi route support #638

Delawen commented May 5, 2023

Delawen commented May 5, 2023

Delawen May 5, 2023

lordrip May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

codecov bot commented May 5, 2023 •

edited

Loading

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

lordrip May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

Delawen May 5, 2023

lordrip left a comment

lordrip May 5, 2023

lordrip May 5, 2023

lordrip commented May 5, 2023

Delawen commented May 5, 2023 •

edited

Loading

Delawen commented May 5, 2023

sonarcloud bot commented May 5, 2023

lordrip commented May 5, 2023

Delawen commented May 5, 2023


		assertEquals(res0.extract().body().asString(), res1.extract().body().asString());

		var flows = Arrays.asList(res1.extract().body().as(Integration[].class));

Multi route support #638

Multi route support #638

Conversation

Delawen commented May 5, 2023

Delawen commented May 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 5, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lordrip left a comment

Choose a reason for hiding this comment

1. Response shape

2. YAML format

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lordrip commented May 5, 2023

Delawen commented May 5, 2023 • edited Loading

Delawen commented May 5, 2023

sonarcloud bot commented May 5, 2023

lordrip commented May 5, 2023

Delawen commented May 5, 2023

codecov bot commented May 5, 2023 •

edited

Loading

Delawen commented May 5, 2023 •

edited

Loading