Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The other day I saw the memory usage of my mysql server implementation (built with go-mysql) surge to 150GB and on further inspection I learned this was caused by a SELECT-query with a rather large resultset. This spiked my interest in the recently implemented
ExecuteSelectStreaming
function but I had a hard time implementing anything useful with it.My server implementation acts as a proxy between clients and multiple mysql servers. Using the
ExecuteSelectStreaming
function instead ofExecute
in for my handler'sHandleQuery
function lacked any way of properly streaming the resultset from the backend mysql server back to my client. This PR solves this issue for me without being too obtrusive.Biggest change is the additional callback
ExecuteSelectStreaming
typeSelectPerResultCallback
. Once the SELECT query was successful the preliminary Result, without the actual data, is passed to this callback. This allows me to write this result directly back to the client, which at that point means only the number of fields and field information is fed back to the client followed by an EOF. Then, when rowdata comes in and therefor calls to theSelectPerRowCallback
callback, I directly write these rows to the client with the newwriteFieldValues
function. For this to work I had to exportformatTextValue
andwriteValue
.This is a relevant snippet of my
HandleQuery
whereExecuteSelectStreaming
is used:Using
ExecuteSelectStreaming
this way shows no spike or memory usage increase at all when I run the exact same query that consumed 150G of memory before.I couldn't find any relevant examples for the use of
ExecuteSelectStreaming
and I know this breaks the API forExecuteSelectStreaming
but if this PR would be merged and released in a version like 1.4.0 that doesn't necessarily has to matter. I can imagine this added flexibility could makeExecuteSelectStreaming
much more suitable to solve more people's problems.Besides that, I added this functionality for prepared SELECT statements as well.