com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Direct self-reference leading to cycle

[mintuhouse]

broken on 0.70.0, 0.69.0, 0.68.0
works with 0.67.0

getting a reproducer is challenging as it goes through many internal libraries.

Using DD_INTEGRATION_RXJAVA_ENABLED=false as temporary fix

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Direct self-reference leading to cycle (through reference chain: io.reactivex.internal.operators.observable.ObservableFlatMap["__datadogContext$io$reactivex$Observable"]->datadog.trace.agent.core.DDSpan["localRootSpan"]->datadog.trace.agent.core.DDSpan["localRootSpan"])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
	at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter._handleSelfReference(BeanPropertyWriter.java:944)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:721)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3905)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3219)
	at com.dream11.api.route.AbstractRoute.getString(AbstractRoute.java:157)
	at com.dream11.api.route.AbstractRoute.prepareResponse(AbstractRoute.java:75)
	at com.dream11.api.route.AbstractRoute.lambda$handle$0(AbstractRoute.java:67)
	at io.reactivex.internal.observers.LambdaObserver.onNext(LambdaObserver.java:63)
	at datadog.trace.instrumentation.rxjava2.TracingObserver.onNext(TracingObserver.java:28)
	at io.reactivex.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:200)
	at io.reactivex.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:252)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	```

[mcculls]

The issue here is that Jackson is attempting to serialize a transient field which is injected by the tracer to help track context. The tracer uses a field for this association to avoid the memory overhead of using a separate (weak) map.

The injected field is deliberately marked as transient so it won't get serialized but it appears that Jackson is ignoring this flag because it also has an associated getter. (This behaviour is different to JDK serialization which just looks at the transient flag.)

According to FasterXML/jackson-databind#296 you can tell Jackson to handle the transient flag just like @JsonIgnore (ie. overriding the fact that it has a getter) by enabling MapperFeature.PROPAGATE_TRANSIENT_MARKER in your mapper.

You could also turn off field-injection, but this is not really recommended because of the increase in memory needed for tracing (especially when asynchronous flows are involved) so if possible I would recommend using that mapper feature.

There are also some things we could look at doing in the tracer to help avoid this situation, but those would require code changes.

Note that 0.67.0 is not affected because it didn't have rxJava 2 instrumentation and therefore didn't use any field-injection for these specific types.

[mcculls]

I've made a change in the tracer to help avoid this in the future: #2229 - it will be available in the next tracer release (0.71.0)

[github-actions]

🤖 This issue has been addressed in the latest release. See full details in the Release Notes.