Large responses written slowly due to allocating too small buffers

[fmarchioni]

Note by @ppalaga 2023-11-23: This issue is only observable when sending a chunked response large enough to surpass the default buffer size used in VertxServletOutputStream to a HTTP/1.1 client. (Note that there is no Transfer-Encoding: chunked in HTTP/2.)

I have found evidence of slow performance of Quarkus CXF extension.
In a nutshell, a single invocation of a SOAP endpoint takes about 10 seconds to return a response. The performance further degrades when concurrent calls are made.

From my initial analysis, I can see that the main Thread that replies to your request interacts with Saxon parsing library for most of the time, until the stream of data is returned to the customer:

"vert.x-worker-thread-0" #114 prio=5 os_prio=0 cpu=8272.53ms elapsed=222.58s tid=0x00007f438c0daa70 nid=0x2d1a runnable  [0x00007f44fb2fa000]
   java.lang.Thread.State: RUNNABLE
at com.sun.org.apache.xerces.internal.dom.ParentNode.nodeListItem([email protected]/ParentNode.java:766)
at com.sun.org.apache.xerces.internal.dom.ParentNode.item([email protected]/ParentNode.java:797)
at net.sf.saxon.dom.DOMNodeWrapper$ChildEnumeration.skipFollowingTextNodes(DOMNodeWrapper.java:1122)
at net.sf.saxon.dom.DOMNodeWrapper$ChildEnumeration.next(DOMNodeWrapper.java:1159)
- locked <0x000000045182aaa8> (a com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl)
at net.sf.saxon.tree.util.Navigator$EmptyTextFilter.next(Navigator.java:1292)
at net.sf.saxon.tree.util.Navigator$EmptyTextFilter.next(Navigator.java:1274)
at net.sf.saxon.tree.iter.LookaheadIteratorImpl.next(LookaheadIteratorImpl.java:53)
at net.sf.saxon.tree.jiter.WrappingJavaIterator.next(WrappingJavaIterator.java:48)
at net.sf.saxon.tree.jiter.WrappingJavaIterator.next(WrappingJavaIterator.java:24)
at net.sf.saxon.tree.util.Navigator.copy(Navigator.java:695)
. . . .
"vert.x-worker-thread-0" #114 prio=5 os_prio=0 cpu=15308.80ms elapsed=230.93s tid=0x00007f438c0daa70 nid=0x2d1a runnable  [0x00007f44fb2fc000]
   java.lang.Thread.State: RUNNABLE
at com.sun.org.apache.xerces.internal.dom.ParentNode.internalInsertBefore([email protected]/ParentNode.java:371)
at com.sun.org.apache.xerces.internal.dom.ParentNode.insertBefore([email protected]/ParentNode.java:286)
at com.sun.org.apache.xerces.internal.dom.NodeImpl.appendChild([email protected]/NodeImpl.java:230)
at org.apache.cxf.staxutils.W3CDOMStreamWriter.writeCharacters(W3CDOMStreamWriter.java:311)
at org.apache.cxf.staxutils.StaxUtils.copy(StaxUtils.java:759)
at org.apache.cxf.staxutils.StaxUtils.copy(StaxUtils.java:707)
at org.apache.cxf.staxutils.StaxUtils.copy(StaxUtils.java:633)
at org.apache.cxf.databinding.source.XMLStreamDataWriter.write(XMLStreamDataWriter.java:127)
at org.apache.cxf.databinding.source.XMLStreamDataWriter.write(XMLStreamDataWriter.java:68)
at org.apache.cxf.databinding.source.XMLStreamDataWriter.write(XMLStreamDataWriter.java:56)

I'm attaching a reproducer for the endpoint available at: http://localhost:8080/foo/ReferenceData?wsdl
Here is a sample SOAP request you can use for a test:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ref="http://fuse.ml.cbs.alpha.gr/ReferenceData/">
   <soapenv:Header/>
   <soapenv:Body>
<ref:getListReferenceDataItem xmlns:ref="http://fuse.ml.cbs.alpha.gr/ReferenceData/">
	<ref:getListReferenceDataItemPayload>
		<ref:RequestDataList>
			<ref:RequestData>
				<ref:Name>ErrorMessages</ref:Name>
				<ref:Version>1</ref:Version>
				<ref:LanguageId>1</ref:LanguageId>
				<ref:SystemsId>1</ref:SystemsId>
				<ref:Value/>
			</ref:RequestData>
		</ref:RequestDataList>
		<ref:LoggingInfo>
			<ref:RequestId>fe9a0075-6035-4399-83e5-acd18efb0999</ref:RequestId>
			<ref:SessionId>c2dbd0fb-914e-4a</ref:SessionId>
			<ref:BusinessCaseId>08ced443-98f2-437c-b8e4-8b92cda8d9dc</ref:BusinessCaseId>
			<ref:SequenceId>1</ref:SequenceId>
			<ref:UserId>StressTest</ref:UserId>
			<ref:CBSUnId/>
		</ref:LoggingInfo>
		<ref:EnvParams>
			<ref:WorkingDate>2020-04-21</ref:WorkingDate>
			<ref:PreviousWorkingDate>2020-04-16</ref:PreviousWorkingDate>
			<ref:NextWorkingDate>2020-04-22</ref:NextWorkingDate>
			<ref:ResourceID>673</ref:ResourceID>
			<ref:UnitCodeLevel1>014</ref:UnitCodeLevel1>
			<ref:UnitTypeCodeLevel1>1</ref:UnitTypeCodeLevel1>
			<ref:UnitCodeLevel2>0096</ref:UnitCodeLevel2>
			<ref:UnitTypeCodeLevel2>03</ref:UnitTypeCodeLevel2>
			<ref:ChannelTypeCode>03</ref:ChannelTypeCode>
			<ref:ResourceTypeCode>05</ref:ResourceTypeCode>
			<ref:BusinessFlowName>SNP00001</ref:BusinessFlowName>
		</ref:EnvParams>
	</ref:getListReferenceDataItemPayload>
</ref:getListReferenceDataItem>
</soapenv:Body>
</soapenv:Envelope>