Compaction fixes

[jwilder]

Required for all non-trivial PRs

• Rebased/mergable
• Tests pass
• CHANGELOG.md updated

This PR re-introduces the fix for #6556 that was reverted in #6637. The original fix could drop points during compactions as well as cause an interface conversion panic (#6652) in some cases. What happened was that merged values from one series could get mixed with the values for the next series. If the series were different types, a panic could occur. If it was the last block to be merged, the values would be lost. Both of these issues should be fixed now as well as the original issue of overwriting a point in a large series triggering memory spikes.

This also fixes #6406 that can occur if enough points are written into a single series to cause the TSM index to exceed the max index entries allotment. It will now handle that error and rollover to a new file.

[mention-bot]

By analyzing the blame information on this pull request, we identified @joelegasse to be a potential reviewer

[jwilder]

@aderumier @daviesalex @easyrasta if you have a dataset that can easily reproduce #6652, would you be able to test this PR on it?

[jwilder]

@e-dard @benbjohnson

[daviesalex]

So this fixed the panic, but not the missing data.

The startup was very fast, but much of our data (most really) only exists from midnight on the 16th (not all , though). See example data:

FWIW, during the startup, most of the time (by a long way) is spent in sync/atomic.AddUint32. This compares to runtime.mapiternext and runtime.indexbytebody in 0.13 stable. I submitted #6658 to track this:

Samples: 4M of event 'cycles', Event count (approx.): 430757131359
  28.53%  influxd                       [.] sync/atomic.AddUint32
  11.64%  influxd                       [.] strings.Index
   5.80%  influxd                       [.] scanblock
   4.72%  [kernel]                      [k] _raw_spin_lock
   4.60%  influxd                       [.] runtime.MSpan_Sweep
   2.33%  influxd                       [.] github.com/influxdata/influxdb/tsdb.(*DatabaseIndex).Series
   1.82%  influxd                       [.] runtime.memmove
   1.76%  influxd                       [.] runtime.aeshashbody
   1.54%  influxd                       [.] runtime.memeqbody
   1.52%  [kernel]                      [k] down_read_trylock
   1.39%  influxd                       [.] runtime.mallocgc
   1.34%  influxd                       [.] runtime.mapaccess1_faststr
   1.22%  influxd                       [.] runtime.deferreturn
   1.18%  influxd                       [.] runtime.newdefer
   1.14%  influxd                       [.] runtime.writebarrierptr
   1.10%  influxd                       [.] runtime.cas64
   1.02%  influxd                       [.] runtime.xchg
   0.95%  influxd                       [.] runtime.readvarint
   0.89%  influxd                       [.] github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).WalkKeys
   0.82%  [kernel]                      [k] page_fault
   0.80%  influxd                       [.] github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).KeyAt
   0.78%  [kernel]                      [k] up_read
   0.76%  influxd                       [.] runtime.findfunc
   0.74%  influxd                       [.] runtime.gentraceback
   0.73%  influxd                       [.] runtime.releasem
   0.69%  influxd                       [.] getfull
   0.66%  influxd                       [.] runtime.freedefer
   0.55%  influxd                       [.] runtime.acquirem
   0.53%  influxd                       [.] runtime.atomicload64
   0.53%  influxd                       [.] runtime.memclr

[daviesalex]

Log showing the start with this PR (branch unknown in log, but it was built from branch jw-compact-fix - 0b2554) that works but is missing most data, and then a start with the 0.13 stable release at https://gist.github.com/daviesalex/f9e70754fb2c0ac85454b6db341b34a4 which works and has the data.

[e-dard]

This seems redundant since there is an err != nil check below.

[benbjohnson]

I don't mind the redundant return but I think specifying ErrMaxBlocksExceeded twice is redundant. You can just do a return err instead.

[benbjohnson]

Also, stylistically I prefer grouping the entire thing together so the err is confined to a specific block:

if err := w.WriteBlock(key, minTime, maxTime, block); err == ErrMaxBlocksExceeded {
    if err := w.WriteIndex(); err != nil {
        return err
    }
    return ErrMaxBlocksExceeded
} else if err != nil {
    return err
}

[e-dard]

s/return ErrMaxBlocksExceeded/return err though 😉

[e-dard]

Just a nit, but LGTM 👍

[aderumier]

@jwilder It's starting fine with this PR. (still around 10min of startup time)

[benbjohnson]

Is this meant to say, "Record"?

[benbjohnson]

Minor nits but lgtm 👍

[jwilder]

@daviesalex With the same dataset/host, does 0.13 load all the data? I'm not sure why you would be missing data using this branch as it does not have any changes related to startup.

[jwilder]

Also, were there any errors in the logs?

[daviesalex]

@jwilder yes. 0.13 loads all the data. This branch does not (nor does the nightly).

The full log (a start under this version, which fails to load all the data, and 0.13, which does) is in the gist above - https://gist.github.com/daviesalex/f9e70754fb2c0ac85454b6db341b34a4

[jwilder]

@daviesalex If the current nightly has missing data, then it's not caused by this PR. Any chance you could git bisect 0.13 and master to see if you can find the commit that is introduced the issue? I suspect 0dbd489 might have introduced the bug.

Would you be able to test that commit and just before it?

[jwilder]

I was able to reproduce the missing data on startup. It is due to 0dbd489#diff-82e4e65bd72f00208a49c4797d466ecdR383.

[jwilder]

@daviesalex I pushed a fix for the data not reloading on this branch. Let me know if that resolves it for you.

[daviesalex]

@jwilder good news. Confirmed that data is there.

[run] 2016/05/18 13:12:08 InfluxDB starting, version unknown, branch unknown, commit unknown
[run] 2016/05/18 13:28:48 Listening for signals

Startup down from about 55 minutes to about 15. I tested on two machines and saw the same just over 3x speedup.

Great success! Thank you.