fix: support retry when io write incompletely

[foreverneverer]

apache/incubator-pegasus#659 has reported the io write operation bug and only retry can decrease the probability of occurrence. This pr refer stackovflow: handling-incomplete-write-calls , rocksdb and add the retry operation.

Note：

• This pr add FAIL_POINT_INJECT_VOID_F macro to mock pwrite to reproduce the bug

[zhangyifan27]

I think you could look at the implementation of Kudu: https://github.com/apache/kudu/blob/6983d25c35df8569c1851bf27e2c75e98036c10d/src/kudu/util/env_posix.cc#L490

When errno == EINTR, Kudu would keeps retrying util other errors occur or write successfully, and also, keeps retrying until the number of remaining bytes (unwritten bytes) is 0.

In addition, it's better to test it on a real cluster to see if this issue could be resolved.

[foreverneverer]

I think you could look at the implementation of Kudu: https://github.com/apache/kudu/blob/6983d25c35df8569c1851bf27e2c75e98036c10d/src/kudu/util/env_posix.cc#L490

When errno == EINTR, Kudu would keeps retrying util other errors occur or write successfully, and also, keeps retrying until the number of remaining bytes (unwritten bytes) is 0.

In addition, it's better to test it on a real cluster to see if this issue could be resolved.

Like this ? @zhangyifan27

ssize_t
write_with_retry (int fd, const void* buf, size_t size)
{
    ssize_t ret;
    while (size > 0) {
        do
        {
             ret = write(fd, buf, size);
        } while ((ret < 0) && (errno == EINTR));
        if (ret < 0)
            return ret;
        size -= ret;
        buf += ret;
    }
    return 0;
}

[foreverneverer]

I have test write incomplete case that the log is too large(>=2147479552 bytes) in test cluster, the result show the retry will resolve the problem:

// this request log only show once, is also say the retry is sucessful
write incomplete, request_size=2816868364, total_write_size=2147479552, this_write_size=2147479552, and will retry it.

however maybe since the log is too large so that the tcmalloc report problem when loop to write such a large amount of data:

asio write faitcmalloc: large alloc 3501096960 bytes == 0x7570e8000 @  0x7faab64b8e9b 0x7faab64dc8cb 0x4c42c2 0x7faab92ca929 0x7faab92cec18 0x7faab92cee04 0x7faab92ccf39 0x7faab8f25aeb 0x7faab8f3acf9 0x7faab8f436b1 0x7faab8f43edc 0x7faab92cac3b 0x7faab9232581 0x7faab924c4fa 0x7faab924c6fd 0x7faab4fbffd0
led: Connection reset by peer

But this is another problem and not the pr target.