feat azure-cosmosdb: Add support for azure cosmos DB NoSQL

[Miuler]

Add suppor for read data from Azure Cosmos DB with Core (SQL) API

Fixes: #4675
Refs: apache/beam#23604

[Miuler]

Please @RustedBones / @clairemcginty, what's next step?

[eddumelendez]

Hi 👋🏽, Just a FYI regarding azure cosmosdb image. It doesn't run with ubuntu-latest in GHA. So far, it only works with 18.04. See Azure/azure-cosmos-db-emulator-docker#45 and Azure/azure-cosmos-db-emulator-docker#56

[RustedBones]

Thanks for the PR. Here are the main points to address:

Let's no depend on scribe for logging, even for tests.
The BoundedReader should be implemented respecting beam's API.

[RustedBones]

The project is not using scribe. The new module should also stick with the logging conventions with slf4j.
also, try to respect alphabetical orderings in the imports.

[Miuler]

The production code is implement with Slf4j (import org.slf4j.LoggerFactory)

But the dependency is implicit, I'm add the dependency explicit.

And respect with scribe, I'm using only in test, because is gives me explicit number line without overload.

[RustedBones]

Yes. It's more a coherence issue. As maintainer, it is preferable to have all module look as similar as possible.
It's also the same with the beam implementations. We've taken the habit to write them in java so upstream contribution to beam can be easy.
Since you've also opened a PR in beam, would you like some support on that ?

[Miuler]

Is there no possibility to support scribe only in tests? It's very scala style, plus it looks beautiful :)

[RustedBones]

To fix the checks in CI, you can run sbt headerCreateAll

[Miuler]

To fix the checks in CI, you can run sbt headerCreateAll

Ready!

[RustedBones]

I've checked a bit Azure cosmos DB documentation and it is very similar to Cassandra or BigTable.
The connector should be written following the same concepts of the CassadraIO/BigtableIO from beam and support splitting reads by partition key ranges (taking care of size and locality).
Right now, the proposed implementation can't be parallelized to run on multiple nodes. People using such connector will hit poor performance when consuming large amount of data

[Miuler]

I've checked a bit Azure cosmos DB documentation and it is very similar to Cassandra or BigTable. The connector should be written following the same concepts of the CassadraIO/BigtableIO from beam and support splitting reads by partition key ranges (taking care of size and locality). Right now, the proposed implementation can't be parallelized to run on multiple nodes. People using such connector will hit poor performance when consuming large amount of data

@RustedBones, This is why it is an experimental one, I already managed to migrate 72k rows to json (1.2Gb) in an azure storage in 20min, this is much better than other alternatives. That is why I have implemented the basic methods.

...I still don't know what would be the equivalent of split for cosmos:

But it is still functional, I am already using it, as soon as I can, I could give the metrics running in PRD.

And of course it can be improved, looking for how I could do that partitioning, but first I have this first version.

PD. The same basic idea is the one I have for the implementation of azure table storage

[Miuler]

The current implementation moves 68729 documents from a cosmosdb to an azure blob storage formatted to json (986M), in 10 minutes accounting for all the time it takes to start the machine in a kubernetes.

[Miuler]

@RustedBones, any news on this pull request? it's been 2 weeks

[RustedBones]

Will base the implementation on beam as explained in #4675