FEATURE_SECURE_PROCESSING threshold too low in XmlDecoder

[dr0i]

Got:

<=<=Exception in thread "main" org.metafacture.framework.MetafactureException: org.xml.sax.SAXParseException; lineNumber: 8496675; columnNumber: 3876; JAXP00010004: Die akkumulierte Größe von Entitys ist "50.000.001" und überschreitet den Grenzwert "50.000.000", der von "FEATURE_SECURE_PROCESSING" festgelegt wurde.
at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:79)
at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:44)
at org.metafacture.io.FileOpener.process(FileOpener.java:158)
at org.metafacture.io.FileOpener.process(FileOpener.java:41)
at org.metafacture.flux.parser.StringSender.process(StringSender.java:43)
at org.metafacture.flux.parser.Flow.start(Flow.java:118)
at org.metafacture.flux.parser.FluxProgramm.start(FluxProgramm.java:168)
at org.metafacture.runner.Flux.main(Flux.java:87)
Caused by: org.xml.sax.SAXParseException; lineNumber: 8496675; columnNumber: 3876; JAXP00010004: Die akkumulierte Größe von Entitys ist "50.000.001" und überschreitet den Grenzwert "50.000.000", der von "FEATURE_SECURE_PROCESSING" festgelegt wurde.
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1243)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:73)

when analyzing Alma Basedump in a single process.

[blackwinter]

How is this a bug in Metafacture? You can adjust the limit if necessary:

https://github.com/hbz/limetrans/blob/c9bde02b7e42680f923232ab24f3134344e0df1a/src/main/java/hbz/limetrans/Limetrans.java#L284-L287

[dr0i]

Thx @blackwinter - I have hoped there is a setting :)
So, using the runner we change its build.gradle to:
applicationDefaultJvmArgs = ["-agentlib:hprof=heap=sites,cpu=samples,depth=${depth},cutoff=${cutoff},file=${file}.hprof.txt -Dinvokejdk.xml.totalEntitySizeLimit=0"] (EDIT dr0i: this is not working. Try JAVA_TOOL_OPTIONS=-Djdk.xml.totalEntitySizeLimit=0 ./gradlew :metafix-runner:run --args="$pathToFlux" )?

May we think about:
a) making this default
b) introducing a parameter for XmlDecoder

[blackwinter]

Changing the JVM args in the Metafix runner is effectively your option a), right? That's a 👎 from me. Since this is a security-related setting, it should be the user's decision to relax any limits.

We might want to introduce a setter for a more targeted approach - your option b) - instead of requiring to set the limit globally. But that's just a convenience feature, isn't it?

[dr0i]

But that's just a convenience feature, isn't it?

It's definitely convenient, but not only that: it also make the restrictions of XmlDecoder more overt to users so that they may circumvent this restriction even before they ran one such big ETL erroring.

[blackwinter]

Okay, no objection to making this limit more discoverable.