Source MongoDB: Failed to fetch Schema

[shubham307307]

I created source (MongoDB) and destination (BigQuery), checked connection - on this step all ok. After, i setup new connection, and started fetch data schema from mongo, this operation lasted about 30 min, and failed. Can you pleae tell is this due to large number of collection or data format issue on mongo side

[harshithmullapudi]

Could be possible that this is because of the large number of collections? Could you help us more information on

1. How many collections are there?
2. Also if you can open network tab and check the error and share the logs here?

[JCWahoo]

This has been happening to me for quite some time, roughly 40 collections. Old Ruby connector doesnt have this issue

[JCWahoo]

Actually I can no longer get schema on either version. Both are still funning loads jut fine, schema discovery fails

[VitaliiMaltsev]

@JCWahoo I can't reproduce this issue. I created a dataset of 40 collections and 100,000 documents in each, and the schema discovery is successful
Do you have any other tips on how to reproduce this?

[JCWahoo]

@VitaliiMaltsev We have more than 40 collections, some with millions of documents. No log messages are returned when using the v2 connector so I've got nothing to go on. The old Ruby connector returns an error around using mapReduce on a view when validating schema... Not sure if that is helpful or not.

When I watch Mongo during schema discovery in v2, it appears the connector is retrieving more than 10k documents for schema evaluation

[VitaliiMaltsev]

@JCWahoo just tested a dataset of 100 collections and 1.5 miliions documents in each, and the schema discovery is still successful
I'm not sure how to reproduce the same behaviour as yours

[JCWahoo]

@VitaliiMaltsev I know, its frustrating not having any error in the logs. It's an Atlas cluster that I'm connecting to the replica shard in standalone mode. The collections have several layers of nesting within documents. Happy to provide any more detail I can, I've been blocked on making any updates to the Mongo connection because the schema discovery continues to fail. The old connector and new connector are still pulling data hourly without issue, I just cant change them.

[VitaliiMaltsev]

@JCWahoo in that case could you please provide me with more information about your source so i can try to reproduce it again

1. Total number of databases in your mongo db and their names
2. Total number of collections in each of the databases
3. The name of the database you are using for the sync
4. Approximate number of documents in each collection
5. The data structure of documents that you use in collections with examples

[JCWahoo]

@VitaliiMaltsev Sure thing. For simplicity/privacy we'll call the database "Production". There are 80 collections, largest one is approx ~50 million documents. I'm using production as the database name for sync and admin as the authentication source. My user has readAnyDatabase @ admin and read @ local permissions in Mongo. Largest collection has around 20 fields, some with several layers of nesting such as

• collection

• array1
  

•     object1
  

•         nestedobject1
  

•         nestedobject2
  

•         nestedobject3
  

•            yetanotherlevelofnesting
  

•     object2
  

[VitaliiMaltsev]

Please provide an example of the document as json with all levels of nesting with the same field names as your largest collection

[JCWahoo]

Can I email it to you rather than here?

[VitaliiMaltsev]

Sure, my email is [email protected]

[VitaliiMaltsev]

@JCWahoo I tried to reproduce this problem with all your recommendations (80 collections, the biggest one with 50 million documents and the same data structure as you sent me) with no success. In my env schema discover works well
On the other hand, I found 2 potential bottlenecks that could lead to this problem.

1. Potential issue with Mongo Atlas latency
   Try changing the region of your cluster as described here
   https://www.mongodb.com/docs/atlas/tutorial/move-cluster/
2. For each collection, the formation of the final result that will be displayed in UI is done using a normal loop, which can be quite slow if you have many collections / documents
   I created a PR to parallelize this process, which will significantly increase the performance
   Hope this helps :)

[JCWahoo]

Great news, thanks!

…

On Wed, Oct 5, 2022, 12:23 PM VitaliiMaltsev ***@***.***> wrote: @JCWahoo <https://github.com/JCWahoo> I tried to reproduce this problem with all your recommendations (80 collections, the biggest one with 50 million documents and the same data structure as you sent me) with no success. In my env schema discover works well On the other hand, I found 2 potential bottlenecks that could lead to this problem. 1. Potential issue with Mongo Atlas latency Try changing the region of your cluster as described here https://www.mongodb.com/docs/atlas/tutorial/move-cluster/ 2. For each collection, the formation of the final result that will be displayed in UI is done using a normal loop, which can be quite slow if you have many collections / documents I created a PR <#17614> to parallelize this process, which will significantly increase the performance Hope this helps :) — Reply to this email directly, view it on GitHub <#14246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7TFL6E5ENX7WFHZM7G3WBWTPRANCNFSM52FNYLWQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[VitaliiMaltsev]

@JCWahoo i just released mongodb-source version 0.1.19. Please check it out and try to refresh schema once more

[JCWahoo]

Still no luck, will share over logs and what it looks like on the Mongo side.... appears to be getting stuck on nested array/object

…

On Fri, Oct 7, 2022 at 7:23 AM VitaliiMaltsev ***@***.***> wrote: @JCWahoo <https://github.com/JCWahoo> i just released mongodb-source version 0.1.19. Please check it out and try to refresh schema once more — Reply to this email directly, view it on GitHub <#14246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7TBAMEM6FLSMEKKQ4FTWCAB3XANCNFSM52FNYLWQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[VitaliiMaltsev]

@JCWahoo I have one more assumption that your collections contain documents that have a different structure within the same collection. This may be causing your issue.
I can advise you to try to reduce the DISCOVER_LIMIT parameter in the MongoUtils class by a multiple.
At the moment it is hardcoded and equals 10000 documents
Try to set it to 100 documents, build the source-mongodb-v2 connector with the dev version and maybe that will solve your problem

[JCWahoo]

Thanks - Any docs/guidance on that last bit? Not sure I've done that before, or its been so long I dont recall

[VitaliiMaltsev]

i believe our teem need to implement this task so that you can choose this option yourself on UI