Is it possible to deploy & scale connectors on the same K8s cluster hosting Fluvio ?

[kikokikok]

Hi everyone,
I was giving Fluvio a ride and wanted to get my head around all the concepts of SmartModules and Connectors.
I have managed to setup a small PoC to give it a ride and test how it behaves under load.

The dev loop experience is great with cdk and smdk !
I managed to deploy a SmartModule on my K8s cluster in cloud and then I wanted to do the same with a custom (sink) connector responsible for pushing the data out for external processing.
Unfortunately, it seems likes there is no way to deploy connectors in the Fluvio Cluster or at least on the same K8s cluster fluvio is running in.

It makes it less than convenient to test the solution end to end with real performance metrics.
I might be missing something which is why I am asking the question here, if not, any plans to support such a scenario ?

FYIW, it is a great stack you've been building ! Continue the good work :)

Thanks

Christian

[sehz]

Thanks for feedback.

Regarding connector, you can deploy connector in the same cluster. It's just that deployment steps are not automated in the open source. You can also try to use https://infinyon.cloud where connectors are fully managed.

[ajhunyady]

Hi @kikokikok, thanks for the feedback.

We are still working out the details on how to best support custom connectors in InfinyOn Cloud without compromising the security of the environment. The current workaround is to point us to the repository, where we can review the code and publish the connector in the cloud for you to use.

[kikokikok]

Hi @ajhunyady and @sehz
Thanks for your answers and suggestions, highly appreciated !
I will definitely git InfinyOn cloud a shot but the connector is far from being Prod qaulity ready...

@sehz: would you mind giving me a hint as to how I could deploy it on a private cluster ? (as painful as it would be). Would definitely appreciate it.

BTW, are you considering the ability to have a dedicated / self-hoster cloud / hub ?

Thanks

[sehz]

To deploy connector, just need to create docker image and then deploy using simple Kubernetes deployment. This is similar approach we do in the cloud (minus secret support and other management support). Most of complexity would be make sure secrets are properly mapped in your deployment environment. We could add sample Dockerfile as an example.

Yes, we are considering to add support for alternative modes of deployments including self hosted cloud. For hub, it already supports private Smartmodule so you can share SmartModule within your org.