Skip to content

Commit

Permalink
kepub: Rewrite conversion code
Browse files Browse the repository at this point in the history
* Use the OPF package document to find HTML files (fixes #55).
* Refactor content/files/opf transformation code.
* Rewrite conversion code to allow converting directly from a EPUB zip.Reader
  or fs.FS into an output zipped KEPUB io.Writer (closes #62).
  * This simplifies the code for kepub conversion.
    * Input and transformation code is unified.
    * Useless function to convert a directory to a KEPUB in-place has been
      eliminated.
    * Arbitrary virtual file-systems can now be used as input.
  * This makes the code easier to test.
  * This resolves security concerns with extracting untrusted EPUBs directly
    into a temp folder with limited path sanitization, which is important when
    embedding kepubify into server-side software.
  * This allows kepubify to easily be compiled and used as a WebAssembly
    library.
  * This allows us to greatly reduce the amount of IO required when
    converting books with a large amount of media or fonts by directly piping it
    into the output file. And, on Go 1.17+, it will also significantly reduce
    the CPU time, while also increasing the amount of time spent doing content
    transformation in parallel rather than waiting for unchanged files to
    compress by directly copying the untransformed compressed files as-is.
  * The slightly increased memory cost (~8-12%) is negligible compared to the
    performance gains mentioned in the previous point and the reduced time
    waiting for disk IO (especially on HDDs).
* Make use of Go 1.16's io/fs for more flexible code and tests.
* Remove cascadia dependency
  * We don't need full selector parsing or specificity.
  * Depending on cascadia complicates replacing x/net/html.
  * Doing things manually is slightly more efficient, and almost as concise.
* Reduce exposed functions for kepub library.
  * They weren't really used.
  * Removing them increases flexibility for future improvements.
* Use a less obtrusive hack for giving kobotest access to the un-exported kepub
  functions.
* Make documentation for transformations more detailed.
* Converted files should be identical to before this change, except for:
  * Improved MathML/SVG tag filtering in content files (some instances which
    would have previously been incorrectly modified are now left as-is).
  * Content files not listed in the package document are now left as-is.
  * Content files with nonstandard extensions, but listed in the package
    document, should now be converted correctly.
* Performance should be equal or better than before this change on Go 1.16, and
  significantly faster for books with many non-content files on Go 1.17. On slow
  storage, kepubify should also be much faster.
  • Loading branch information
pgaskin committed Jul 2, 2021
1 parent e721d07 commit 948788e
Show file tree
Hide file tree
Showing 15 changed files with 940 additions and 1,174 deletions.
38 changes: 33 additions & 5 deletions cmd/kepubify/kepubify.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
package main

import (
"archive/zip"
"context"
"fmt"
"io"
"os"
Expand Down Expand Up @@ -81,8 +83,6 @@ func main() {
}
}

kepub.Verbose = *verbose

// --- Make converter --- //

var opts []kepub.ConverterOption
Expand Down Expand Up @@ -256,7 +256,37 @@ func main() {
log(true, " Error (%d): %v\n", i, err)
continue
}
if err := converter.ConvertEPUB(input, output); err != nil {
if err := func() error {
fi, err := zip.OpenReader(input)
if err != nil {
return err
}
defer fi.Close()

fo, err := os.CreateTemp(filepath.Dir(output), ".kepubify."+filepath.Base(output)+".*")
if err != nil {
return err
}
defer os.Remove(fo.Name())

if err := converter.Convert(context.Background(), fo, fi); err != nil {
return err
}

if err := fo.Sync(); err != nil {
return err
}

if err := fo.Close(); err != nil {
return err
}

if err := os.Rename(fo.Name(), output); err != nil {
return err
}

return nil
}(); err != nil {
errs.Store(input, err)
atomic.AddInt64(&errored, 1)
log(true, " Error (%d): %v\n", i, err)
Expand Down Expand Up @@ -313,8 +343,6 @@ func helpExit() {
fmt.Fprintf(os.Stderr, "\n%s:\n%s", strings.Split(category, ".")[1], categories[category].FlagUsagesWrapped(160))
}

// TODO: examples?

fmt.Fprintf(os.Stderr, "\nLinks:\n")
for _, v := range [][]string{
{"Website", "https://pgaskin.net/kepubify"},
Expand Down
2 changes: 0 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@ require (
)

require (
// kepub
github.com/andybalholm/cascadia v1.2.0
github.com/beevik/etree v1.1.0
github.com/kr/smartypants v0.1.0
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553
Expand Down
2 changes: 0 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
github.com/andybalholm/cascadia v1.2.0 h1:vuRCkM5Ozh/BfmsaTm26kbjm0mIOM3yS5Ek/F5h18aE=
github.com/andybalholm/cascadia v1.2.0/go.mod h1:YCyR8vOZT9aZ1CHEd8ap0gMVm2aFgxBp0T0eFw1RUQY=
github.com/bamiaux/rez v0.0.0-20170731184118-29f4463c688b h1:5Ci5wpOL75rYF6RQGRoqhEAU6xLJ6n/D4SckXX1yB74=
github.com/bamiaux/rez v0.0.0-20170731184118-29f4463c688b/go.mod h1:obBQGGIFbbv9KWg92Qu9UHeD94JXmHD1jovY/z6I3O8=
github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs=
Expand Down
Loading

0 comments on commit 948788e

Please sign in to comment.