Skip to content

Commit

Permalink
kepub: Fixed FD leak in content transformation
Browse files Browse the repository at this point in the history
The FDs didn't get closed until after all content files were
processed due to the deferred close being in a for loop. This
wasn't noticed due to it not being an issue until the concurrency
in cbb9af0 causing it to exceed
the default FD limit on macOS for a large (500+ content file)
book.

I've ensured there weren't any other leaks left behind by testing
this with a batch conversion of large books and a ulimit of the
theoretical maximum amount kepubify should have:

    5 + max(1, max(cpu_cores, cpu_cores*max(3, cpu_cores + 1))
    |       |      |          |             |  ^ kepub (transformAllContentParallel): parallel content file transformation (+1 for scanning)
    |       |      |          |             ^ kepub (ConvertEPUB): packing/unpacking (zip+input_output+temp)
    |       |      |          ^ cmd/kepubify (convert): concurrent conversion
    |       |      ^ cmd/kepubify (convert): concurrent scanning/copying
    |       ^ cmd/kepubify (transform): path transformation
    ^ cmd/kepubify: kepubify itself (binary/stdout/stderr/etc...)

This issue was reported by email on 2020-01-01.
  • Loading branch information
pgaskin committed Apr 1, 2020
1 parent c6b1733 commit 2196def
Showing 1 changed file with 5 additions and 6 deletions.
11 changes: 5 additions & 6 deletions kepub/converter.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,14 +136,13 @@ func (c *Converter) transformAllContentParallel(dir string) error {
for i := 0; i < runtime.NumCPU(); i++ {
g.Go(func() error {
for fn := range contentFiles {
f, err := os.OpenFile(fn, os.O_RDWR, 0)
if err != nil {
if f, err := os.OpenFile(fn, os.O_RDWR, 0); err != nil {
return fmt.Errorf("open content file %#v: %w", fn, err)
}
defer f.Close()

if err := c.TransformContentDocFile(f); err != nil {
} else if err := c.TransformContentDocFile(f); err != nil {
f.Close()
return fmt.Errorf("transform content file %#v: %w", fn, err)
} else {
f.Close()
}

select {
Expand Down

0 comments on commit 2196def

Please sign in to comment.