[Feature Request]: Support writing to a Solace message broker #31905
Comments
iht
added a commit
to iht/beam
that referenced
this issue
Jul 16, 2024
This adds a base class and two providers for the new SolaceIO write connector. The basic authentication provider just uses a username and password. Users can write their own providers to set authentication mechanisms and other session properties, by extending from the base class. As an example, we provide a GoogleCloudSecretProvider, to show how to create your own. This provider can also be used "as is", but it is mainly meant as an example to implement your own. In upcoming PRs, I will be submitting the rest of the write connector. It is thousands of lines of code, so I am splitting in smaller PRs, to facilitate code reviews. This PR contributes to apache#31905
This was referenced Jul 17, 2024
Abacn
pushed a commit
that referenced
this issue
Jul 22, 2024
* JCSMP properties providers for new SolaceIO write connector This adds a base class and two providers for the new SolaceIO write connector. The basic authentication provider just uses a username and password. Users can write their own providers to set authentication mechanisms and other session properties, by extending from the base class. As an example, we provide a GoogleCloudSecretProvider, to show how to create your own. This provider can also be used "as is", but it is mainly meant as an example to implement your own. In upcoming PRs, I will be submitting the rest of the write connector. It is thousands of lines of code, so I am splitting in smaller PRs, to facilitate code reviews. This PR contributes to #31905 * Fix CheckStyle and Spotbugs errors * Fix class dependencies declaration * Unify session and auth providers/factories for the read and write connectors * Fix format violations in comment
iht
added a commit
to iht/beam
that referenced
this issue
Jul 23, 2024
…1905 This adds the interface of the Write connector and a few classes (data classes, POutput) that are used by the connector.
iht
added a commit
to iht/beam
that referenced
this issue
Jul 23, 2024
…1905 This adds the interface of the Write connector and a few classes (data classes, POutput) that are used by the connector.
16 tasks
Abacn
pushed a commit
that referenced
this issue
Jul 29, 2024
This adds the interface of the Write connector and a few classes (data classes, POutput) that are used by the connector.
|
.take-issue |
iht
added a commit
to iht/beam
that referenced
this issue
Aug 27, 2024
…31905. This PR adds the actual writer functionality, and some additional testing, including integration testing. This should be final PR for the SolaceIO write connector to be complete.
bzablocki
pushed a commit
to iht/beam
that referenced
this issue
Nov 12, 2024
…31905. This PR adds the actual writer functionality, and some additional testing, including integration testing. This should be final PR for the SolaceIO write connector to be complete.
Abacn
pushed a commit
that referenced
this issue
Nov 13, 2024
* This is a follow-up PR to #31953, and part of the issue #31905. This PR adds the actual writer functionality, and some additional testing, including integration testing. This should be final PR for the SolaceIO write connector to be complete. * Use static imports for Preconditions * Remove unused method * Logging has builtin formatting support * Use TypeDescriptors to check the type used as input * Fix parameter name * Use interface + utils class for MessageProducer * Use null instead of optional * Avoid using ByteString just to create an empty byte array. * Fix documentation, we are not using ByteString now. * Not needed anymore, we are not using ByteString * Defer transforming latency from nanos to millis. The transform into millis is done at the presentation moment, when the metric is reported to Beam. * Avoid using top level classes with a single inner class. A couple of DoFns are moved to their own files too, as the abstract class forthe UnboundedSolaceWriter was in practice a "package". This commits addresses a few comments about the structure of UnboundedSolaceWriter and some base classes of that abstract class. * Remove using a state variable, there is already a timer. This DoFn is a stateful DoFn to force a shuffling with a given input key set cardinality. * Properties must always be set. The warnings are only shown if the user decided to set the properties that are overriden by the connector. This was changed in one of the previous commits but it is actually a bug. I am reverting that change and changing this to a switch block, to make it more clear that the properties need to be set always by the connector. * Add a new custom mode so no JCSMP property is overridden. This lets the user to fully control all the properties used by the connector, instead of making sensible choices on its behalf. This also adds some logging to be more explicit about what the connector is doing. This does not add too much logging pressure, this only adds logging at the producer creation moment. * Add some more documentation about the new custom submission mode. * Fix bug introduced with the refactoring of code for this PR. I forgot to pass the submission mode when the write session is created, and I called the wrong method in the base class because it was defined as public. This makes sure that the submission mode is passed to the session when the session is created for writing messages. * Remove unnecessary Serializable annotation. * Make the PublishResult class for handling callbacks non-static to handle pipelines with multiple write transforms. * Rename maxNumOfUsedWorkers to numShards * Use RoundRobin assignment of producers to process bundles. * Output results in a GlobalWindow * Add ErrorHandler * Fix docs * Remove PublishResultHandler class that was just a wrapper around a Queue * small refactors * Revert CsvIO docs fix * Add withErrorHandler docs * fix var scope --------- Co-authored-by: Bartosz Zablocki <[email protected]>
reeba212
pushed a commit
to reeba212/beam
that referenced
this issue
Dec 4, 2024
…1906) * JCSMP properties providers for new SolaceIO write connector This adds a base class and two providers for the new SolaceIO write connector. The basic authentication provider just uses a username and password. Users can write their own providers to set authentication mechanisms and other session properties, by extending from the base class. As an example, we provide a GoogleCloudSecretProvider, to show how to create your own. This provider can also be used "as is", but it is mainly meant as an example to implement your own. In upcoming PRs, I will be submitting the rest of the write connector. It is thousands of lines of code, so I am splitting in smaller PRs, to facilitate code reviews. This PR contributes to apache#31905 * Fix CheckStyle and Spotbugs errors * Fix class dependencies declaration * Unify session and auth providers/factories for the read and write connectors * Fix format violations in comment
reeba212
pushed a commit
to reeba212/beam
that referenced
this issue
Dec 4, 2024
…1905 (apache#31953) This adds the interface of the Write connector and a few classes (data classes, POutput) that are used by the connector.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What would you like to happen?
I'd like to add a native Java connector to write messages to Solace message broker. See the design doc for details: https://docs.google.com/document/d/1mPQFT4OCEUjFIoEwkL3W4mQVj0JuCC0TXmYt-8CBV-E/edit?usp=sharing
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
The text was updated successfully, but these errors were encountered: