Error handling2

[kspangsege]

There are two main things introduced:

1. New File and File::Map classes that serve as abstraction layers over systems particulars

2. Scoped transaction. All units tests have been changed to use them.

Also, SharedGroup::is_valid() has been removed.

[astigsen]

Would it be an idea to use fallocate directly when supported (on linux). In samba they are quite explicit about which fallocate they use. See http://lists.samba.org/archive/rsync/2008-February/020103.html

[rrrlasse]

SetFileValidData requires SE_MANAGE_VOLUME_NAME privilege to be enabled in Windows. It's disabled by default in all windows distributions and can only be enabled in the Server editions of Windows and Ultimate. The cheaper editions doesn't allow you to enable it.

[astigsen]

If might be an idea to add a flag parameter as well. When mapping files for transient data, we would like the ability to tell it that it that syncing is not needed if the os supports it (MAP_NOSYNC on freebsd).

[astigsen]

Why do you need to unlock before downgrading? Is it to keep windows semantics?

[astigsen]

Looks fine to me (and passes all test in xcode).

[rrrlasse]

Issuing more than ~64 MB to WriteFile()/write() will make them do only a partial write both on Win/Linux, so the test with (n == static_cast(size) will fail.

Also, issuing too much data at a time will block the file system (at least I've seen this on Windows 7) and gives bad latency for other sevices. Around 32 KB still gives nearly max throughput.

Missing test of close() return value so double-close could go undetected.

If the customer wants unicode characters in file names - but also paths! - we would need to use the Ex versions of file API which use unicode w_char and wstring (*nix kernels use UTF8 so no problem here).

Just a memo: we need tests for hidden shares on Windows and UNC paths (common on servers).

Will review more later...

[kspangsege]

Lasse - According to Microsofts documentation for FileWrite(), partial writes are only possible in asynchronous mode or when writing to a pipe. Also, I have not been able to find any other people claiming that partial write are possible in our case. If you are sure that partial writes are possible, let me know.

The WriteFile function returns when one of the following conditions occur:

• The number of bytes requested is written.
• A read operation releases buffer space on the read end of the pipe (if the write was blocked). For more information, see the Pipes section.
• An asynchronous handle is being used and the write is occurring asynchronously.
• An error occurs.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365747%28v=vs.85%29.aspx

On Linux and indeed on any POSIX compliant system, a write() that reports success can never be partial unless non-blocking I/O is explicitly enabled.

With respect to your second point, that large writes should be submitted from the application to the kernel in smaller chunks, I think that should be up to the user of the File API.

[rrrlasse]

I think partial writes are probably covered by the quoted.

But another issue is failed writes that could have been avoided. I once tried to tune eXdupe by issuing 256 MB blocks and got heaps of bug reports. Like timeouts when writing to slow network drives mapped by Terminal Server (worked fine on SMB), and tape devices that would make write() return an error code, etc, etc, etc. Also weird behaviour caused by being so long in kernel space.

I'm of course thinking of the case where you would dump 10+ GB somewhere.