Spring Reactive DB Drivers - MongoDB Support

[tspring]

Add out-of-the-box Java agent support for MongoDB under the Spring Reactive framework

See MMF doc

Add support for Reactive calls for MongoDB so that database interactions are instrumented by the Java Agent.

MongoDB, Redis, and Cassandra all have native reactive support in Spring Data. Many relational databases (Postgres, Microsoft SQL Server, MySQL, H2, and Google Spanner) have reactive support via R2DBC.

Confirm support for async.client.MongoClient

Particularly Mongo will report incorrect metrics for Async which has been an issue for a while.

Review https://github.com/topics/nrlabs instrumentation for Mongo async, Lettuce, etc.

Aha! Link: https://newrelic.aha.io/features/JAVA-295

Write instrumentation and instrumentation tests for MongoDB reactive streams.

[jasonjkeller]

The existing mongo instrumentation seems to be partially applying when mongo reactive is being used. This is causing a number of issues (skewed response times, memory pressure) as those instrumentation modules aren't intended to support mongo reactive. See this internal slack thread for some additional context.

To prevent such issues when using mongo reactive the existing instrumentation should be disabled.

  class_transformer:
    com.newrelic.instrumentation.mongodb-3.7:
      enabled: false
    com.newrelic.instrumentation.mongodb-3.1:
      enabled: false
    com.newrelic.instrumentation.mongodb-3.0:
      enabled: false

As part of the work to officially support mongo reactive we need to determine whether or not the old instrument should apply and be supplemented by an additional instrumentation specific to mongo reactive. Or we need to ensure that the old instrumentation modules don't apply at all and only the new mongo reactive instrumentation does.

[Stephan202]

@jasonjkeller the instrumentation artifact IDs use mongodb rather than mongo. Updating your suggestion accordingly, I can confirm that the following works:

  class_transformer:
    com.newrelic.instrumentation.mongodb-3.7:
      enabled: false
    com.newrelic.instrumentation.mongodb-3.1:
      enabled: false
    com.newrelic.instrumentation.mongodb-3.0:
      enabled: false

Any update on reactive driver support? The vast majority of our MongoDB integrations us the reactive driver, and we're pretty blind at the moment. To be honest, it's a pretty big issue :(

[Stephan202]

We have a number of applications that use both the sync and async driver. When using a version 3.x MongoDB driver we could see the synchronous queries, while the asynchronous queries did not show up in New Relic. Now that we have upgraded to driver version 4.x we hit the transaction timeout issue discussed above. This forces us to disable instrumentation of the synchronous driver as suggested, thus further reducing our visibility.

I see that version 4.0.0 of the MongoDB driver landed in Maven Central on 2020-03-04, i.e. 16 months ago. Given MongoDB's popularity, can you please prioritize support for the 4.x line? As a customer, is there anything we can do to help?

[Stephan202]

Upon further testing we concluded that disabling just the MongoDB 3.7 instrumentation is enough:

  class_transformer:
    com.newrelic.instrumentation.mongodb-3.7:
      enabled: false

This way new Relic still reports queries by the synchronous driver, thereby maintaining the status quo. (Apparently the fallback to the 3.1 or 3.0 instrumentation suffices.)

@nathankooij filed a PR that is expected to restore this behavior out of the box: #341.

[jasonjkeller]

Current Mongo DB drivers

• Core: org.mongodb:mongodb-driver-core
  • The Java operations layer for the MongoDB Java Driver. Third parties can wrap this layer to provide custom higher-level APIs. It contains low-level methods to do all the operations common to the sync and async drivers. Unless you are making a new API / Driver for MongoDB, you shouldn't directly use the core driver.
• Sync: org.mongodb:mongodb-driver-sync
  • The MongoDB Synchronous Driver. Synchronous driver that you can use to search, create, read, update and delete documents. Blocks until result it returned.
• Sync: org.mongodb:mongodb-driver
  • The MongoDB Driver uber-artifact that combines mongodb-driver-sync and the legacy driver
• Sync: org.mongodb:mongo-java-driver
  • The MongoDB Java Driver uber-artifact, containing the legacy driver, the mongodb-driver, mongodb-driver-core, and bson.
• Async: org.mongodb:mongodb-driver-reactivestreams
  • Asynchronous driver that you can use to search, create, read, update and delete documents. Calling thread won't block as the result is returned in a callback.

---

SpringBoot 2.0.x introduced two different async drivers for mongdb.

• org.mongodb:mongodb-driver-async
• org.mongodb:mongodb-driver-reactivestreams

SpringBoot 2.0.x

org.mongodb:mongodb-driver-async:3.6.4
org.mongodb:mongodb-driver-reactivestreams:1.7.1

SpringBoot 2.1.x

org.mongodb:mongodb-driver-async:3.8.2
org.mongodb:mongodb-driver-reactivestreams:1.9.2

SpringBoot 2.2.x

org.mongodb:mongodb-driver-async:3.11.2
org.mongodb:mongodb-driver-reactivestreams:1.12.0

---

SpringBoot 2.3.x saw the removal of org.mongodb:mongodb-driver-async.

SpringBoot 2.3.x

org.mongodb:mongodb-driver-reactivestreams:4.0.6

SpringBoot 2.4.x

org.mongodb:mongodb-driver-reactivestreams:4.1.2

SpringBoot 2.5.x

org.mongodb:mongodb-driver-reactivestreams:4.2.3

[jasonjkeller]

The problem with the existing mongo sync client instrumentation applying to the mongo reactive client is that an unexpected thread switch happens between the mongo command starting and completing which causes Segment timeouts.

A mongo command is started and the instrumentation's CommandListener creates a Segment and saves it to the ThreadLocal on the executing thread. This happens on a thread like Thread[reactor-http-nio-5,5,main] when using the reactive client.

newrelic-java-agent/instrumentation/mongodb-3.7/src/main/java/com/nr/agent/mongodb/NewRelicCommandListener.java

Lines 37 to 43 in 95edf01

	@Override
	public void commandStarted(CommandStartedEvent event) {
	try {
	Segment tracer = NewRelic.getAgent().getTransaction().startSegment(null);
	if (tracer != null) {
	holder.set(tracer);

When the command completes the CommandListener ends the Segment timing and removes it from the ThreadLocal. This happens on a different thread like Thread[nioEventLoopGroup-3-4,10,main] when using the reactive client.

newrelic-java-agent/instrumentation/mongodb-3.7/src/main/java/com/nr/agent/mongodb/NewRelicCommandListener.java

Lines 73 to 79 in 95edf01

	private void tryFinishEvent() {
	try {
	Segment tracerEvent = holder.get();
	if (tracerEvent != null) {
	tracerEvent.end();
	holder.remove();
	}

This means that the Segment is never actually ended or removed. This results in the Segment timeout limit of 10 minutes being hit for every mongo query and the buildup of Segment objects that can lead to memory pressure.