Expose failed processor using exception metadata

[DaveCTurner]

... instead of headers.

#18342 introduced some extra information on exceptions thrown by a compound processor, using headers, which is one of the many metadata mechanisms we currently have. As part of #27672 we'd like to reduce the variety of metadata mechanisms for the benefit of clients. Could this information be passed through using addMetadata() instead?

@talevy do you have any opinions on this?

[DaveCTurner]

Also @simonw said this: #16276 (comment) - do you (Simon) have opinions about preferring metadata over headers?

[DaveCTurner]

The code in question is

elasticsearch/core/src/main/java/org/elasticsearch/ingest/CompoundProcessor.java

Lines 158 to 163 in 24d7fa6

	if (processorType != null) {
	exception.addHeader("processor_type", processorType);
	}
	if (processorTag != null) {
	exception.addHeader("processor_tag", processorTag);
	}

[talevy]

Thanks for bringing this up @DaveCTurner. At the time, headers were the only way to inject custom context without creating a new custom Exception (something that is frowned upon). I was not aware that metadata was added in 5.3 to clean up the semantics of these additional fields (#22703). It seems like metadata is the more appropriate place for this information given the direction of the code.

I will take a look at this and see if it is as I thought. We can update this code for 7.0.

[javanna]

Wasn't also not having to parse response body a reason for using headers over metadata? Do we know if any of our product reads this stuff returned as part of response headers?

[dakrone]

Closing this as we are working on something better with the introduction of the failure store for data streams.