fix: compress pipeline graphs before sending to mermaid #8767
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
When pipelines get too big, the resulting base64 output is too long when we send a request to the mermaid server leading to a HTTP 400 error. Mermaid supports compressed URLS using Pako. With
zlib
, we are able to compress our input in the expected format and send it off to Mermaid.How did you test it?
I wanted to do as close to a 1:1 change so that this has 0 impact to the user. The same tests were used which hopefully means the changes are 1:1 as they have all passed.
I am also asking for tests from Haystack users with big pipelines on Discord.
Notes for the reviewer
I will make another PR that will use this change as a foundation for:
Aaaaand.... offline support! #7896
Checklist
fix:
,feat:
,build:
,chore:
,ci:
,docs:
,style:
,refactor:
,perf:
,test:
and added!
in case the PR includes breaking changes.