refactor I/O to do more buffering

[Keno]

Noting down ideas for the I/O refactor that I discussed with @JeffBezanson (do add anything I forgot).

• Instead of having the basic I/O operate on single bytes, have a new interface that can read/write arrays of bytes (probably also nb_available)
• All writes are buffered, with the buffer flushed when we enter the event loop next (or explicitly once we reach the system network buffer limit)
• This new BufferedIO type would be the one to be extended for things like endianness, text-encodings, etc. rather than the underlying stream.

[kmsquire]

As I mentioned in #3820, consider Apache's bucket brigades when implementing this. In particular, the referenced implementation

• uses multiple, dynamically allocatable, possibly variable sized buffers
  • with one buffer, the current IOBuffer moves data to the front of the buffer when compacting, which is potentially costly, and avoidable with multiple buffers
• offers copy-free access to data in the buffer, when possible
  • takes into account data alignment and the amount of data requested
  • byte-order issues could be potentially be handled in place
  • care would need to be taken with mark/reset (RFM: mark/reset for IOStream, IOBuffer, & AsyncStream (addresses #2638) #3656) if direct data access is available, although it's possible direct data access could be implemented using a mark/reset-like mechanism.

I'm not sure how feasible in-place data access is in julia (it would likely have the same pointer/gc issues mentioned in #3224), but I would love to see this.

[vtjnash]

I'm curious how costly the copy strategy that I employed actually is, in practice, not because of how the compact is written, but because of the heuristics in ensureroom that I intended to limit the amount of data that gets copied. In particular, it only occurs just before a write (so read access doesn't move data). If the buffer was empty prior to the write, the pointers are immediately reset. Otherwise, they are reset if we will need to resize the buffer anyways, or the amount of space wasted at the beginning is greater than the amount of data in the buffer or there is a substantial amount of wasted space at the beginning of the buffer.

I imagine a strategy that was able to map/move/free entire pages (via mmap/mremap(MREMAP_FIXED)) could be significantly more efficient, but annoying to write.

[vtjnash]

moving this out of v0.4.x: there doesn't seem to be a pressing need to implement this right now, with all the possible issues that might be discovered later.

[vtjnash]

We've slowly started to add this to more streams. It's still a good work item, but needs someone to spend a fair amount of focus on it.

[StefanKarpinski]

Is this a breaking change or a performance optimization?

[ViralBShah]

Close?

[JeffBezanson]

I don't think we've done this yet.