Test failing -- TestElasticSearchSuite/TestSparkDependenciesES

[jpkrohling]

The test TestElasticSearchSuite/TestSparkDependenciesES as part of es-otel is currently failing, affecting #1111, #1110 and #1105. The master branch is currently also failing after #1102 was merged, but I don't think that PR is the cause, as the failures on the mentioned PRs were happening before that got merged.

cc @kevinearls

[moolen]

I'm looking into this. intel so far:

It boils down to a combination of TestSparkDependenciesES and TestSimpleProd. If i comment out TestSimpleProd everything works just fine.
The TestSimpleProd seems to leave some ES indices behind which confuse TestSparkDependenciesES. If we remove the indices everything works just fine.
This assumes that the TestSimpleProd runs before TestSparkDependenciesES. Unclear: why does this issue arise only when using OTEL? 🤔

Thesis: if we delete the indices created in TestSimpleProd at the end of the test run all following tests should work.

[kevinearls]

I've been able to reproduce this, and you can see repeated stack traces in the log for the test-spark-deps-spark-dependencies cron job here:

https://gist.github.com/kevinearls/400f6d2920fbd16629078a8fcd7c0a7d

@jpkrohling unfortunately I probably won't have much time to look at this today. It might also be better to have @pavolloffay take a look if he's free. @moolen Thanks for the tip, although I'm afraid if we just change test order or clean up it may be covering up another problem. This was passing until recently.

[moolen]

I dug a little and found this in elasticsearch:

{
    "_index": "jaeger-span-2020-07-04",
    "_type": "span",
    "_id": "28USGnMBbz6bVeBZk-Ui",
    "_score": 1.0,
    "_source": {
        "traceID": "41488369ec34f3c8",
        "spanID": "41488369ec34f3c8",
        "operationName": "SmokeTest",
        "references": null, //   <------- should be []
    }
}                

This seems to be caused by returning nil here (introduced ~3d ago in jaegertracing/jaeger#2295):
https://github.com/jaegertracing/jaeger/blob/master/cmd/opentelemetry/app/exporter/elasticsearchexporter/esmodeltranslator/modeltranslator.go#L137

[yurishkuro]

what about tags and logs, can they be null?