In-memory streaming conversions and other performance improvements #62
Assignees
Labels
Comments
pgaskin
added a commit
that referenced
this issue
Jun 11, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Converted files should be identical to before this change, except for: * Whitespace changes in content.opf. * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 11, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Whitespace changes in content.opf. * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 11, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Whitespace changes in content.opf. * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
changed the title
pgaskin
added a commit
that referenced
this issue
Jun 15, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 15, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 15, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 15, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 16, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
pgaskin
added a commit
that referenced
this issue
Jun 16, 2021
* Use the OPF package document to find HTML files (fixes #55). * Refactor content/files/opf transformation code. * Rewrite conversion code to allow converting directly from a EPUB zip.Reader or fs.FS into an output zipped KEPUB io.Writer (closes #62). * This simplifies the code for kepub conversion. * Input and transformation code is unified. * Useless function to convert a directory to a KEPUB in-place has been eliminated. * Arbitrary virtual file-systems can now be used as input. * This makes the code easier to test. * This resolves security concerns with extracting untrusted EPUBs directly into a temp folder with limited path sanitization, which is important when embedding kepubify into server-side software. * This allows kepubify to easily be compiled and used as a WebAssembly library. * This allows us to greatly reduce the amount of IO required when converting books with a large amount of media or fonts by directly piping it into the output file. And, on Go 1.17+, it will also significantly reduce the CPU time, while also increasing the amount of time spent doing content transformation in parallel rather than waiting for unchanged files to compress by directly copying the untransformed compressed files as-is. * The slightly increased memory cost (~8-12%) is negligible compared to the performance gains mentioned in the previous point and the reduced time waiting for disk IO (especially on HDDs). * Make use of Go 1.16's io/fs for more flexible code and tests. * Remove cascadia dependency * We don't need full selector parsing or specificity. * Depending on cascadia complicates replacing x/net/html. * Doing things manually is slightly more efficient, and almost as concise. * Reduce exposed functions for kepub library. * They weren't really used. * Removing them increases flexibility for future improvements. * Use a less obtrusive hack for giving kobotest access to the un-exported kepub functions. * Make documentation for transformations more detailed. * Converted files should be identical to before this change, except for: * Improved MathML/SVG tag filtering in content files (some instances which would have previously been incorrectly modified are now left as-is). * Content files not listed in the package document are now left as-is. * Content files with nonstandard extensions, but listed in the package document, should now be converted correctly. * Performance should be equal or better than before this change on Go 1.16, and significantly faster for books with many non-content files on Go 1.17. On slow storage, kepubify should also be much faster.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Benefits of in-memory streaming conversions:
Other potential performance improvements:
The text was updated successfully, but these errors were encountered: