DATAGO-64296: initial commit for terraform plugin #132

gregmeldrum · 2023-11-23T14:52:19Z

What is the purpose of this change?

This PR adds the new terraform plugin to the EMA. The plugin is added as a new module.
Additionally, a temporary REST API has been added to allow testing
The REST API follows the newly approved command model

How was this change implemented?

The terraformClient has been copied and modified from https://github.com/microsoft/terraform-spring-boot
The RESP API is added to the application module since this module takes care of the northbound interface
The terraform specific code is added to the terraform-plugin module.

Some work has been deferred to later stories such as:

logging support (some logging support is included but it will be enhanced in the future)
building the docker image containing the terraform executable and solace provider
adding a messaging API for Solace Cloud Event Portal
functional testing for all supported usecases (in this PR only simple adding/deleting of queues is tested)
negative functional testing
non-functional testing (performance and load)

How was this change tested?

Simple functional tests were run using the REST API via POSTMAN commands.
Integration tests were run using the TerraformClientRealTests test class in the terraform-plugin
Unit tests were added to test the basic terraform command handling using a mocked terraform client.

Is there anything the reviewers should focus on/be aware of?

Just re-iterating, some of the code was taken from open-source (MIT license) at https://github.com/microsoft/terraform-spring-boot
The following classes were copied and modified:
ProcessLauncher
TerraformClient
Our lawyer gave the thumbs up to use this code as long as we include a link to it's origin and a copy of the license under which it was released.

…ages

…each request requires a new client

gregmeldrum · 2023-11-23T14:58:23Z

.../src/main/java/com/solace/maas/ep/event/management/agent/command/rest/CommandController.java

@@ -0,0 +1,28 @@
+package com.solace.maas.ep.event.management.agent.command.rest;


This code will be removed once the messaging I/F is added.

gregmeldrum · 2023-11-23T14:59:09Z

...ation/src/main/java/com/solace/maas/ep/event/management/agent/logging/StreamingAppender.java

                sendLogsAsync(event,
                        event.getMDCPropertyMap().get(RouteConstants.SCAN_ID),
                        event.getMDCPropertyMap().get(RouteConstants.TRACE_ID),
                        event.getMDCPropertyMap().get(RouteConstants.ACTOR_ID),
                        event.getMDCPropertyMap().get(RouteConstants.SCAN_TYPE),
                        event.getMDCPropertyMap().get(RouteConstants.SCHEDULE_ID),
                        event.getMDCPropertyMap().get(RouteConstants.MESSAGING_SERVICE_ID));
+            } else if (StringUtils.isNotEmpty(event.getMDCPropertyMap().get(RouteConstants.COMMAND_CORRELATION_ID))) {
+                log.trace("This is a placeholder for DATAGO-64298");


This is the hook where we will add the terraform logs in the future.

gregmeldrum · 2023-11-23T15:00:05Z

...e/plugin/src/main/java/com/solace/maas/ep/event/management/agent/plugin/mop/MOPProtocol.java

@@ -3,7 +3,9 @@
 public enum MOPProtocol {
    scanData(2850),
    scanDataControl(2851),
-    EMAHeartbeat(2852);
+    EMAHeartbeat(2852),
+    commandProtocol(2853);


This was added for future considerations when we add the messaging API

gregmeldrum · 2023-11-23T15:00:23Z

service/pom.xml

@@ -237,22 +239,6 @@
                                </resources>
                            </configuration>
                        </execution>
-                        <execution>


Unused execution

gregmeldrum · 2023-11-23T15:01:55Z

.../java/com/solace/maas/ep/event/management/agent/plugin/terraform/client/ProcessLauncher.java

@@ -0,0 +1,149 @@
+package com.solace.maas.ep.event.management.agent.plugin.terraform.client;


This file comes from open-source with light modifications.

gregmeldrum · 2023-11-23T15:02:38Z

.../java/com/solace/maas/ep/event/management/agent/plugin/terraform/client/TerraformClient.java

@@ -0,0 +1,197 @@
+package com.solace.maas.ep.event.management.agent.plugin.terraform.client;


This file comes from open source with moderate modifications.

gregmeldrum · 2023-11-23T15:09:02Z

...olace/maas/ep/event/management/agent/plugin/terraform/configuration/TerraformProperties.java

+@Service
+@Data
+public class TerraformProperties {
+    @Value("${plugins.terraform.workingDirectoryRoot:/${HOME}/tfconfig}")


This is an unorthodox way of setting properties. @value is not the recommended way to populate config. In this case I wanted a convenient way to set a default value for this value without requiring a change to the application.yml file. Alternatively, we could add a backend story to EP to create a new plugins.terraform.workingDirectoryRoot property to the generated application.ylm file.

service/application/src/main/java/com/solace/maas/ep/common/messages/CommandMessage.java

sonarqube-solacecloud · 2023-11-23T21:56:43Z

SonarQube Quality Gate

4.0% Coverage on New Code (is less than 80%)

See analysis details on SonarQube

.../main/java/com/solace/maas/ep/event/management/agent/plugin/command/model/CommandBundle.java

service/terraform-plugin/pom.xml

.../java/com/solace/maas/ep/event/management/agent/plugin/terraform/client/TerraformClient.java

...e/maas/ep/event/management/agent/plugin/terraform/manager/TerraformLogProcessingService.java

moodiRealist · 2023-11-24T19:23:33Z

...ava/com/solace/maas/ep/event/management/agent/plugin/terraform/manager/TerraformManager.java

+        String traceId = MDC.get("traceId");
+        String spanId = MDC.get("spanId");
+
+        log.debug("Executing command {} for ms {} correlationId {} context {}", command.getCommand(), request.getServiceId(),


please change ms to some generic name like service since it won't work for schemaRegsitry configPush scenarios

...test/java/com/solace/maas/ep/event/management/agent/plugin/terraform/TerraformCommandIT.java

gregmeldrum added 12 commits November 9, 2023 13:12

First implementation of terraform plugin

8015469

Remove unneeded annotation

c25e365

Cleanup for poc

4319153

Fix up dockerfile

46cd6d1

Rework Command classes to match ADR

b498848

Add potential MOP message class for command request and response mess…

73489d0

…ages

Add some experiments with logging

8fdd126

rebase with main

13db44f

Cleanup pom files

31b1a29

Create factory for terraform client since the client is stateful and …

cf6c111

…each request requires a new client

Add mock logs

3ec624f

Improve test coverage

1eb41f3

gregmeldrum had a problem deploying to build_pr November 23, 2023 14:52 — with GitHub Actions Error

Removing terraform docker artifacts for first commit

310cece

gregmeldrum temporarily deployed to build_pr November 23, 2023 14:57 — with GitHub Actions Inactive

Simplify code in the Terraform Manager

fd3d1b4

gregmeldrum temporarily deployed to build_pr November 23, 2023 15:12 — with GitHub Actions Inactive

gregmeldrum commented Nov 23, 2023

View reviewed changes

moodiRealist reviewed Nov 23, 2023

View reviewed changes

service/application/src/main/java/com/solace/maas/ep/common/messages/CommandMessage.java Outdated Show resolved Hide resolved

Attempt to fix sonarqube

88240bd

gregmeldrum temporarily deployed to build_pr November 23, 2023 19:25 — with GitHub Actions Inactive

Change thread handling for terraform command

b96e19c

gregmeldrum temporarily deployed to build_pr November 23, 2023 19:45 — with GitHub Actions Inactive

Change thread handling for terraform command

8566602

gregmeldrum temporarily deployed to build_pr November 23, 2023 19:51 — with GitHub Actions Inactive

Change messagingServiceId to serviceId

8e90b88

gregmeldrum temporarily deployed to build_pr November 23, 2023 21:52 — with GitHub Actions Inactive

moodiRealist reviewed Nov 24, 2023

View reviewed changes

.../main/java/com/solace/maas/ep/event/management/agent/plugin/command/model/CommandBundle.java Outdated Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

service/terraform-plugin/pom.xml Outdated Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

service/terraform-plugin/pom.xml Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

.../java/com/solace/maas/ep/event/management/agent/plugin/terraform/client/TerraformClient.java Outdated Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

...e/maas/ep/event/management/agent/plugin/terraform/manager/TerraformLogProcessingService.java Outdated Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

...e/maas/ep/event/management/agent/plugin/terraform/manager/TerraformLogProcessingService.java Outdated Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

...e/maas/ep/event/management/agent/plugin/terraform/manager/TerraformLogProcessingService.java Show resolved Hide resolved

moodiRealist reviewed Nov 24, 2023

View reviewed changes

...e/maas/ep/event/management/agent/plugin/terraform/manager/TerraformLogProcessingService.java Outdated Show resolved Hide resolved

Addressing PR comments

803d459

gregmeldrum had a problem deploying to build_pr November 24, 2023 16:46 — with GitHub Actions Failure

Fixing a PR comment change

665dae7

gregmeldrum had a problem deploying to build_pr November 24, 2023 16:49 — with GitHub Actions Failure

gregmeldrum requested a review from moodiRealist November 24, 2023 16:52

moodiRealist reviewed Nov 24, 2023

View reviewed changes

Change log based on PR comment

008ef97

gregmeldrum had a problem deploying to build_pr November 24, 2023 19:31 — with GitHub Actions Failure

moodiRealist reviewed Nov 24, 2023

View reviewed changes

...test/java/com/solace/maas/ep/event/management/agent/plugin/terraform/TerraformCommandIT.java Outdated Show resolved Hide resolved

Update IT method scope to private

84b57d5

gregmeldrum had a problem deploying to build_pr November 24, 2023 19:38 — with GitHub Actions Failure

moodiRealist approved these changes Nov 24, 2023

View reviewed changes

gregmeldrum merged commit d19dfea into main Nov 24, 2023
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DATAGO-64296: initial commit for terraform plugin #132

DATAGO-64296: initial commit for terraform plugin #132

gregmeldrum commented Nov 23, 2023 •

edited

Loading

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

gregmeldrum Nov 23, 2023

sonarqube-solacecloud bot commented Nov 23, 2023

moodiRealist Nov 24, 2023

		@@ -0,0 +1,28 @@
		package com.solace.maas.ep.event.management.agent.command.rest;

		@@ -0,0 +1,149 @@
		package com.solace.maas.ep.event.management.agent.plugin.terraform.client;

		@@ -0,0 +1,197 @@
		package com.solace.maas.ep.event.management.agent.plugin.terraform.client;

DATAGO-64296: initial commit for terraform plugin #132

DATAGO-64296: initial commit for terraform plugin #132

Conversation

gregmeldrum commented Nov 23, 2023 • edited Loading

What is the purpose of this change?

How was this change implemented?

How was this change tested?

Is there anything the reviewers should focus on/be aware of?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqube-solacecloud bot commented Nov 23, 2023

Choose a reason for hiding this comment

gregmeldrum commented Nov 23, 2023 •

edited

Loading