NIFI-13008 - CLI command to upgrade all instances of a versioned flow #8611

pvillard31 · 2024-04-08T15:23:10Z

Summary

NIFI-13008 - CLI command to upgrade all instances of a versioned flow

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 21

Licensing

New dependencies are compatible with the Apache License 2.0 according to the License Policy
New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

Documentation formatting appears as expected in rendered files

pvillard31 · 2024-04-08T15:29:45Z

Example of command and output:

With force and scoped to a given process group

./cli.sh nifi pg-change-all-versions --baseUrl https://localhost:8443/nifi-api/ --truststore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/truststore.p12 --truststoreType PKCS12 --truststorePasswd 39dfd2623cc2f14e9c78b391ce2d8bee --keystore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/keystore.p12 --keystoreType PKCS12 --keystorePasswd 43823e27844853e450f895689d3d986f --keyPasswd 43823e27844853e450f895689d3d986f --flowIdentifier e76e8b1a-3a02-4fe9-ba4f-f559acb151f9 --force --processGroupId a9cd774d-018e-1000-3acc-544a3caaf5d5

#   Name              Id                                     Prev Version   New Version   Message
-   ---------------   ------------------------------------   ------------   -----------   ----------------------------------------------------------------------------------------------------
1   MyVersionedFlow   a9ced459-018e-1000-ab85-352acdd238f9   3              3             Process group already at desired version
2   MyVersionedFlow   a9ce7d38-018e-1000-14af-e9ff25e6d759   1              null          Failed to perform update flow request due to StandardProcessGroup[identifier=a9ce7d38-018e-1000-14af-e9ff25e6d759,name=MyVersionedFlow] cannot be updated to the proposed flow because the proposed flow does not contain a match for Connection[ID=5b0c4128-d73b-352d-50f6-a8d325a7be32, Source ID=6d6ef667-f560-3637-5fc3-1c26901ff799, Dest ID=48817384-4031-332e-837d-1bdc298325b6] and the connection currently has data in the queue.

Without force and without scoping to a process group

./cli.sh nifi pg-change-all-versions --baseUrl https://localhost:8443/nifi-api/ --truststore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/truststore.p12 --truststoreType PKCS12 --truststorePasswd 39dfd2623cc2f14e9c78b391ce2d8bee --keystore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/keystore.p12 --keystoreType PKCS12 --keystorePasswd 43823e27844853e450f895689d3d986f --keyPasswd 43823e27844853e450f895689d3d986f --flowIdentifier e76e8b1a-3a02-4fe9-ba4f-f559acb151f9

#   Name              Id                                     Prev Version   New Version   Message
-   ---------------   ------------------------------------   ------------   -----------   ----------------------------------------------------------------------------------------------------
1   MyVersionedFlow   a9ce9e63-018e-1000-59ed-2cb11e96d2df   2              null          Failed to perform update flow request due to StandardProcessGroup[identifier=a9ce9e63-018e-1000-59ed-2cb11e96d2df,name=MyVersionedFlow] cannot be updated to the proposed flow because the proposed flow does not contain a match for Connection[ID=d2122b84-ce83-3958-2de4-bcfb33f2a496, Source ID=944a1a3c-0d47-35de-5522-069527d99531, Dest ID=e5fde182-1ded-3fdb-725b-2294382a69f9] and the connection currently has data in the queue.

Downgrade by specifying the flow version

./cli.sh nifi pg-change-all-versions --baseUrl https://localhost:8443/nifi-api/ --truststore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/truststore.p12 --truststoreType PKCS12 --truststorePasswd 39dfd2623cc2f14e9c78b391ce2d8bee --keystore /Users/pierre/dev/github/nifi/nifi-assembly/target/nifi-2.0.0-SNAPSHOT-bin/nifi-2.0.0-SNAPSHOT/conf/keystore.p12 --keystoreType PKCS12 --keystorePasswd 43823e27844853e450f895689d3d986f --keyPasswd 43823e27844853e450f895689d3d986f --flowIdentifier e76e8b1a-3a02-4fe9-ba4f-f559acb151f9 --force --flowVersion 1

#   Name              Id                                     Prev Version   New Version   Message
-   ---------------   ------------------------------------   ------------   -----------   ----------------------------------------------------------------------------------------------------
1   MyVersionedFlow   a9ce9e63-018e-1000-59ed-2cb11e96d2df   2              null          Failed to perform update flow request due to StandardProcessGroup[identifier=a9ce9e63-018e-1000-59ed-2cb11e96d2df,name=MyVersionedFlow] cannot be updated to the proposed flow because the proposed flow does not contain a match for Connection[ID=0a4f1e3e-b062-381f-2838-05aac3d384e0, Source ID=944a1a3c-0d47-35de-5522-069527d99531, Dest ID=781e060f-6b00-37de-acd7-4e6cb68087b2] and the connection currently has data in the queue.
2   MyVersionedFlow   a9cdb690-018e-1000-9ab0-66411f5398d9   3              1             SUCCESS
3   MyVersionedFlow   a9ced459-018e-1000-ab85-352acdd238f9   3              1             SUCCESS
4   MyVersionedFlow   a9ce7d38-018e-1000-14af-e9ff25e6d759   1              1             Process group already at desired version

gresockj

Thanks @pvillard31. I have some minor stylistic comments for now, but I'd like to test this out and provide some runtime feedback.

gresockj · 2024-04-17T13:39:01Z

...oolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/NiFiCommandGroup.java

@@ -133,6 +134,7 @@ protected List<Command> createCommands() {
        commands.add(new PGGetVersion());
        commands.add(new PGStopVersionControl());
        commands.add(new PGChangeVersion());
+        commands.add(new PGChangeAllVersions());


The new command should be added to the toolkit guide as well

gresockj · 2024-04-17T13:48:31Z

...-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeAllVersions.java

+        final List<ProcessGroupDTO> processGroups = new ArrayList<>();
+        final Map<String, ChangeVersionResult> changeVersionResults = new HashMap<String, ChangeVersionResult>();
+
+        for (ProcessGroupDTO pgDTO : pgList) {


This can be final as well.

gresockj · 2024-04-17T13:49:04Z

...-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeAllVersions.java

+            parentPgId = flowClient.getRootGroupId();
+        }
+
+        PGList doPGList = new PGList();


We can have a few more final variables here, such as doPGList, pgList, and doPGChangeVersion.

gresockj · 2024-04-17T13:49:29Z

...-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeAllVersions.java

+            processGroups.add(pgDTO);
+
+            final Integer previousVersion = pgDTO.getVersionControlInformation().getVersion();
+            if(previousVersion == newVersion) {


Let's add a space after the if for style consistency.

gresockj · 2024-04-17T13:49:40Z

...-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeAllVersions.java

+                changeVersionResults.put(pgDTO.getId(), new ChangeVersionResult(previousVersion, newVersion, "SUCCESS"));
+            } catch (Exception e) {
+                changeVersionResults.put(pgDTO.getId(), new ChangeVersionResult(previousVersion, null, e.getMessage()));
+                if(forceOperation) {


Space after the if

...lkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeVersion.java

gresockj · 2024-04-17T13:51:24Z

...lkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGChangeVersion.java

@@ -121,7 +132,7 @@ public VoidResult doExecute(final NiFiClient client, final Properties properties
        return VoidResult.getInstance();
    }

-    private int getLatestVersion(final NiFiClient client, final VersionControlInformationDTO existingVersionControlDTO)
+    public int getLatestVersion(final NiFiClient client, final VersionControlInformationDTO existingVersionControlDTO)


If I'm not mistaken, this could be package scope (just remove the public)

gresockj · 2024-04-17T13:51:47Z

.../nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGList.java

+        return getList(client, properties, parentPgId);
+    }
+
+    public ProcessGroupsResult getList(final NiFiClient client, final Properties properties, String pgID)


final pgID

gresockj · 2024-04-17T13:52:05Z

...lkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/result/nifi/ChangeVersionResult.java

+    Integer newVersion;
+    String message;
+
+    public ChangeVersionResult(Integer previousVersion, Integer newVersion, String message) {


Arguments could be final

gresockj · 2024-04-17T13:52:45Z

...main/java/org/apache/nifi/toolkit/cli/impl/result/nifi/ProcessGroupsVersionChangeResult.java

+
+    @Override
+    public ReferenceResolver createReferenceResolver(final Context context) {
+        final Map<Integer,ProcessGroupDTO> backRefs = new HashMap<>();


Space after the comma

gresockj

Thanks for the updates, @pvillard31, this works well.

Signed-off-by: Joe Gresock <[email protected]> This closes apache#8611.

gresockj requested changes Apr 17, 2024

View reviewed changes

gresockj approved these changes May 3, 2024

View reviewed changes

pvillard31 added 2 commits May 3, 2024 13:27

NIFI-13008 - CLI command to upgrade all instances of a versioned flow

03eb39c

review

4b8372a

pvillard31 force-pushed the NIFI-13008 branch from 65209ab to 4b8372a Compare May 3, 2024 11:27

asfgit closed this in ba1109f May 3, 2024

gresockj mentioned this pull request May 3, 2024

NIFI-13008 - CLI command to upgrade all instances of a versioned flow (support/nifi-1.x) #8735

Closed

11 tasks

shubhluck pushed a commit to shubhluck/nifi that referenced this pull request Jun 1, 2024

NIFI-13008 - CLI command to upgrade all instances of a versioned flow

562e9df

Signed-off-by: Joe Gresock <[email protected]> This closes apache#8611.

shubhluck pushed a commit to shubhluck/nifi that referenced this pull request Jun 1, 2024

NIFI-13008 - CLI command to upgrade all instances of a versioned flow

a23e2c6

Signed-off-by: Joe Gresock <[email protected]> This closes apache#8611.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-13008 - CLI command to upgrade all instances of a versioned flow #8611

NIFI-13008 - CLI command to upgrade all instances of a versioned flow #8611

pvillard31 commented Apr 8, 2024 •

edited

Loading

pvillard31 commented Apr 8, 2024

gresockj left a comment

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj Apr 17, 2024

gresockj left a comment

NIFI-13008 - CLI command to upgrade all instances of a versioned flow #8611

NIFI-13008 - CLI command to upgrade all instances of a versioned flow #8611

Conversation

pvillard31 commented Apr 8, 2024 • edited Loading

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

pvillard31 commented Apr 8, 2024

gresockj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gresockj left a comment

Choose a reason for hiding this comment

pvillard31 commented Apr 8, 2024 •

edited

Loading