Features/584 data parallel -- DASO

[coquelin77]

Description

Implementation of the Distributed Asynchronous and Selective Optimization (DASO) method.
Other changes include a second Image-Net example using DALI on TFRecords with DASO, MPI sum functions for torch.half and torch.bfloat16, as well as a class to determine if a metric is plateauing.

Issue/s resolved: N/A

Changes proposed:

• DASO implementation
• MPI sum functions for torch.half and torch.bfloat16 dtypes
• nn.DataParallelMultiGPU: utilize torch.distributed on a node-local level for the forward and backward steps of a NN
• optim.DetectMetricPlateau can be used to detect when a metric has plateaued. this is often used during training of the optimizers

Type of change

• New feature (non-breaking change which adds functionality)

Due Diligence

• All split configurations tested
• Multiple dtypes tested in relevant functions
• Documentation updated (if needed)
• Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

[coquelin77]

rerun tests