Add PR=quorum in HEADs at GC and PW=1 at writing blocks [RCS-282]

[kuenishi]

This commit tries to address following potential issues:

• False-notfound in case of faulty cluster that has several nodes
  down, block server misunderstands that block was already deleted
  which may lead to block leak (PR default is 0)
• Make sure at least single replica of a block is written in one
  of primary nodes (PW default is 0)
• Prevent block server from crash in case heading a block returning
  other errors than notfound - this makes GC correctly skip unreachable
  blocks and go to try to collect next blocks and manifests as much
  as possible

The price of this commit: with PR and PW values increased at blocks,
the availability of PUTs and throughtput of garbage collection may
decrease than previous releases of Riak CS. A few Riak nodes being
unreachable may prevent PUT requests from successful return, or
garbage collection from collecting all blocks until unreachable nodes
come back.

We could use PW=2 for putting blocks but we didn't this is because
it decreases availability more - and we expect such blocks be repaired
during leeway period.

[shino]

No much time for now, sorry for no digging into details. GC seems to be hang up when executing batch for 4-node cluster which has two stopped node.

[shino]

Quick diff:

--- a/src/riak_cs_block_server.erl
+++ b/src/riak_cs_block_server.erl
@@ -219,7 +219,8 @@ handle_cast({delete_block, ReplyPid, Bucket, Key, UUID, BlockNumber}, State=#sta
                 %% Report errors in HEADs to prevent crashing block
                 %% servers, as crash logs forces lager to sync log
                 %% files at each line.
-                Result = format_delete_result(Error, BlockNumber),
+                Result = format_delete_result(Error, {UUID, BlockNumber}),

[shino]

Much safer (= less possibility of false negativity) than before 🔒

[kuenishi]

@borshop merge

[kuenishi]

For release note: Github Issue has full description about this fix.

[Basho-JIRA]

For release note: Github Issue has full description about this fix.

_[posted via JIRA by Kota Uenishi]_