Extend eager-lock to Metadata transactions

[pranithk]

1. Eager-lock implementation in AFR at the moment only supports data transactions. When replication is used along with shard, and shard does fxattrops for doing size/blocks updates high contention is observed.
2. What we also observed during some usecases is that eager-lock implementation in AFR at the moment doesn't handle conflicting writes from same fd in parallel.

Contrast to this, in EC, eager-lock is enabled for both data and metadata operations. It also handles 2) well, in addition to that, When there are conflicting writes, it doesn't lead to unlocks/post-ops, it tries to do most of the work using same lock as much as possible. This implementation should be borrowed in AFR as well.

EC uses same inode lock for both data and metadata domains, but AFR can't do that to keep backward compatibility.

Lifecycle of lock:

First transaction is added to inode->owners list and an inodelk will be sent on
the wire. All the next transactions will be put in inode->waiters list until
the first transaction completes inodelk and [f]xattrop completely. Once
[f]xattrop also completes, all the requests in the inode->waiters list are
checked if it conflict with any of the existing locks which are in
inode->owners list and if not are added to inode->owners list and resumed with
doing transaction. When these transactions complete fop phase they will be
moved to inode->post_op list and resume the transactions that were paused
because of conflicts. Post-op and unlock will not be issued on the wire until
that is the last transaction on that inode. Last transaction when it has to
perform post-op can choose to sleep for deyed-post-op-secs value. During that
time if any other transaction comes, it will wake up the sleeping transaction
and takes over the ownership of the lock and the cycle continues. If the
dealyed-post-op-secs expire, then the timer thread will wakeup the sleeping
transaction and it will set lock->release to true and starts doing post-op and
then unlock. During this time if any other transactions come, they will be put
in inode->frozen list. Once the previous unlock comes it will move the frozen
list to waiters list and moves the first element from this waiters-list to
owners-list and attempts the lock and the cycle continues. This is the general
idea. There is logic at the time of dealying and at the time of new
transaction or in flush fop to wakeup existing sleeping transactions or
choosing whether to delay a transaction etc, which is subjected to change based
on future enhancements etc.
At the moment, when more than 1 fd is open it falls back to non-eager-lock mode.

[pranithk]

https://review.gluster.org/19711 - has the spec. Documentation needs to be added at the time of doing release notes.

@nigelbabu I did add "Updates #418" to the patch but update is not happening on the github issue. Am I doing something wrong?

[gluster-ant]

A patch https://review.gluster.org/19503 has been posted that references this issue.
Commit message: cluster/afr: Make AFR eager-locking similar to EC