DATAGO-57805: Add command line functionality #153

gregmeldrum · 2024-01-18T22:05:28Z

What is the purpose of this change?

The existing EMA user interface is too difficult. The ask is to add a command runner which takes arguments.

How was this change implemented?

Built on top of Julian's excellent POC. Added a new command runner that can be accessed by disabling tomcat using spring command line attributes.
Made the error handling more robust by introducing the new PluginClientException that camel can use to tell when the scan has failed and report an error.

How was this change tested?

Tested manually from the command line. Also regressed the existing standalone and cloud-connected scans to make sure nothing was broken.

Is there anything the reviewers should focus on/be aware of?

No

…ocessing in failure scenarios

gregmeldrum · 2024-01-18T22:23:47Z

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

+        if (args.length > 2) {
+
+            String type = args[0];
+


For now, only handle the scan command.

comment for next ticket, let's change this var to commandType or command or something else.

gregmeldrum · 2024-01-18T22:25:44Z

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

+            log.error("Scan request [{}]: Scan did not complete successfully.", scanId);
+        }
+    }
+


It completed successfully if there are dataCollectionFileEntitites for each scan type, and none of them are marked as "FAILED"

gregmeldrum · 2024-01-18T22:27:02Z

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

+        AtomicBoolean scanCompleted = new AtomicBoolean(false);
+
+        // The hard timeout is set to 30 minutes to prevent the scan from running forever
+        long hardTimeoutMillis = System.currentTimeMillis() + 30 * 60 * 1000;


This is the "paranoia" timeout just so the EMA doesn't run forever. It's currently set to 30 minutes since large configs can take a long time to scan.

gregmeldrum · 2024-01-18T22:28:17Z

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

+
+        // The initiated timout is set to 2 minutes to shortcut a scan that is stuck in the INITIATED state
+        // This typically occurs if there is an error in the broker URL or credentials
+        long initiatedTimeoutMillis = System.currentTimeMillis() + 2 * 60 * 1000;


This timeout is triggered if all of the scans are in "INITIATED" state for more than 2 minutes. It's meant to catch cases where we had trouble creating the client or running the command but the exception was not captured.

gregmeldrum · 2024-01-18T22:33:23Z

...ain/java/com/solace/maas/ep/event/management/agent/processor/RouteCompleteProcessorImpl.java

        String scanId = (String) exchange.getIn().getHeader(RouteConstants.SCAN_ID);
-


I pulled this code into a shared abstract class called RouteStateProcessor that's used by RouteCompleteProcessor and RouteFailedProcessor.

gregmeldrum · 2024-01-18T22:36:24Z

...ation/src/main/java/com/solace/maas/ep/event/management/agent/service/ScanStatusService.java

        this.idGenerator = idGenerator;
    }

    @Transactional
-    public ScanStatusEntity save(String name, String scanId) {


Adding the ability to set any state instead of hardcoding to COMPLETE.

gregmeldrum · 2024-01-18T22:37:15Z

.../com/solace/maas/ep/event/management/agent/plugin/manager/client/KafkaClientManagerImpl.java

@@ -39,17 +40,17 @@ public class KafkaClientManagerImpl implements MessagingServiceClientManager<Adm
    private final TimeUnit maxReconnectBackoffTimeUnit;



SaveActions plugin re-wrote this...

gregmeldrum · 2024-01-18T22:37:56Z

...n/java/com/solace/maas/ep/event/management/agent/plugin/exception/PluginClientException.java

@@ -0,0 +1,19 @@
+package com.solace.maas.ep.event.management.agent.plugin.exception;


This is a new Generic Plugin Exception that Camel will look for to know when a scan has failed.

gregmeldrum · 2024-01-18T22:39:05Z

...om/solace/maas/ep/event/management/agent/plugin/route/handler/base/AbstractRouteBuilder.java

@@ -40,7 +41,8 @@ protected AbstractRouteBuilder(ScanTypeDescendentsProcessor scanTypeDescendentsP
    public void configure() throws Exception {

        // NetworkClient Selectable network i/o exception + Kafka Future exceptions
-        onException(IOException.class, InterruptedException.class, ExecutionException.class, TimeoutException.class)
+        onException(IOException.class, InterruptedException.class, ExecutionException.class, TimeoutException.class,
+                PluginClientException.class)


Adding the PluginClientException that any plugin can throw. This signals to camel that an error has occurred and Camel can set the scan type status to FAILED.

gregmeldrum · 2024-01-18T22:39:26Z

...om/solace/maas/ep/event/management/agent/plugin/route/handler/base/AbstractRouteBuilder.java

@@ -59,6 +61,7 @@ public void configure() throws Exception {
                        RouteConstants.TRACE_ID + "}]: The status of [${header." +
                        RouteConstants.SCAN_TYPE + "}]" + " is: [${header." + RouteConstants.SCAN_STATUS + "}].")
                .to("direct:perRouteScanStatusPublisher?block=false&failIfNoConsumers=false")
+                .to("direct:processScanStatusAsFailed?block=false")


New route to set the scan status to failed for this type.

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

moodiRealist · 2024-01-19T19:24:13Z

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java

+        long initiatedTimeoutMillis = System.currentTimeMillis() + 2 * 60 * 1000;
+
+        while (System.currentTimeMillis() < hardTimeoutMillis && !scanCompleted.get()) {
+            Thread.sleep(2000);


I think this Thread.sleep increases CPU usage by a lot. Can we use Timer and TimerTask instead to run a scanComplete or hardTimeoutMillis check every 2 minutes?

If I'm guessing wrong on the cpu consumption, let's keep the code as is.

Everything I've read online says that the thread is released and does not consume CPU for a .sleep and .wait() so out of simplicity I'd like to keep it as is.

moodiRealist

approved with non-blocking comments.

…d off

sonarqube-solacecloud · 2024-01-19T22:17:16Z

SonarQube Quality Gate

56.1% Coverage on New Code (is less than 80%)

See analysis details on SonarQube

gregmeldrum added 2 commits January 18, 2024 15:59

Add CLI capability for enhanced user experience

cd6d6f3

Update error handling so that Camel can see client errors and stop pr…

3f732bc

…ocessing in failure scenarios

gregmeldrum had a problem deploying to build_pr January 18, 2024 22:05 — with GitHub Actions Failure

Minor code cleanup

e9487b3

gregmeldrum had a problem deploying to build_pr January 18, 2024 22:07 — with GitHub Actions Error

Remove unused code

ae10ec5

gregmeldrum had a problem deploying to build_pr January 18, 2024 22:08 — with GitHub Actions Failure

Update the way Exceptions are created

e4b598e

gregmeldrum temporarily deployed to build_pr January 18, 2024 22:17 — with GitHub Actions Inactive

gregmeldrum added 2 commits January 18, 2024 17:23

Additional code cleanup

24f080d

More code cleanup

467f9d3

gregmeldrum temporarily deployed to build_pr January 18, 2024 22:35 — with GitHub Actions Inactive

gregmeldrum commented Jan 18, 2024

View reviewed changes

Improve tests

c82f2fe

gregmeldrum had a problem deploying to build_pr January 19, 2024 16:22 — with GitHub Actions Failure

Suppress duplicate warnings in tests

78bdb58

gregmeldrum had a problem deploying to build_pr January 19, 2024 16:46 — with GitHub Actions Error

gregmeldrum marked this pull request as ready for review January 19, 2024 16:46

Merge branch 'main' into DATAGO-57805-command-line

7fcfd1f

gregmeldrum temporarily deployed to build_pr January 19, 2024 16:48 — with GitHub Actions Inactive

gregmeldrum requested a review from moodiRealist January 19, 2024 17:06

moodiRealist reviewed Jan 19, 2024

View reviewed changes

.../application/src/main/java/com/solace/maas/ep/event/management/agent/cli/EmaCommandLine.java Outdated Show resolved Hide resolved

moodiRealist reviewed Jan 19, 2024

View reviewed changes

moodiRealist approved these changes Jan 19, 2024

View reviewed changes

Make sure the EmaCommandLine is only run when the web server is turne…

6a19782

…d off

gregmeldrum temporarily deployed to build_pr January 19, 2024 21:57 — with GitHub Actions Inactive

Address PR comment for using builder pattern

ba06537

gregmeldrum temporarily deployed to build_pr January 19, 2024 22:12 — with GitHub Actions Inactive

gregmeldrum merged commit a59133c into main Jan 19, 2024
5 of 6 checks passed

gregmeldrum deleted the DATAGO-57805-command-line branch February 6, 2024 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DATAGO-57805: Add command line functionality #153

DATAGO-57805: Add command line functionality #153

gregmeldrum commented Jan 18, 2024 •

edited

Loading

gregmeldrum Jan 18, 2024

moodiRealist Jan 19, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

gregmeldrum Jan 18, 2024

moodiRealist Jan 19, 2024

moodiRealist Jan 19, 2024

gregmeldrum Jan 19, 2024

moodiRealist left a comment

sonarqube-solacecloud bot commented Jan 19, 2024

		String scanId = (String) exchange.getIn().getHeader(RouteConstants.SCAN_ID);

		@@ -39,17 +40,17 @@ public class KafkaClientManagerImpl implements MessagingServiceClientManager<Adm
		private final TimeUnit maxReconnectBackoffTimeUnit;

		@@ -0,0 +1,19 @@
		package com.solace.maas.ep.event.management.agent.plugin.exception;

DATAGO-57805: Add command line functionality #153

DATAGO-57805: Add command line functionality #153

Conversation

gregmeldrum commented Jan 18, 2024 • edited Loading

What is the purpose of this change?

How was this change implemented?

How was this change tested?

Is there anything the reviewers should focus on/be aware of?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moodiRealist left a comment

Choose a reason for hiding this comment

sonarqube-solacecloud bot commented Jan 19, 2024

gregmeldrum commented Jan 18, 2024 •

edited

Loading