All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog.
- Added support for batched memory copies in GPUAllgather. (#3590)
- Added support for batched memory copies in GPUReducescatter. (#3621)
- Added
hvd.grouped_allgather()
andhvd.grouped_reducescatter()
operations. (#3594) - TensorFlow: Added doc string for
hvd.grouped_allreduce()
. (#3594) - Added warning messages if output tensor memory allocations fail. (#3594)
- Added
register_local_source
anduse_generic_names
funtionality to DistributedGradientTape. (#3628)
- Updated Eigen submodule to fix build on macOS with aarch64. (#3619)
- Fix FuseResponses() on BATCHED_D2D_PADDING edge cases for Reducescatter and/or ROCm. (#3621)
- PyTorch: Fixed Reducescatter functions to raise
HorovodInternalError
rather thanRuntimeError
. (#3594) - PyTorch on GPUs without GPU operations: Fixed grouped allreduce to set CPU device in tensor table. (#3594)
- Added
hvd.reducescatter()
operation with implementations in NCCL, MPI, and Gloo. (#3299, #3574) - Added AMD GPU XLA Op Implementation. (#3486)
- Added Horovod job to spin up distributed TensorFlow Data Service. (#3525)
- Spark: Expose random seed as an optional parameter. (#3517)
- Add Helm Chart. (#3546)
- Elastic: Add elastic run API. (#3503)
- Spark Estimator: Expose random seed for model training reproducibility. (#3517)
- Spark Estimator: Add option whether to use GPUs at all. (#3526)
- Spark Estimator: Expose parameter to set start method for
multiprocessing
. (#3580)
- MXNet: Updated allreduce functions to newer
op
API. (#3299) - TensorFlow: Make TensorFlow output allocations asynchronous when using NCCL backend. (#3464)
- TensorFlow: Clear locally accumulated gradient by assigning with
zeros_like
to avoid infinite gradient not correctly cleared up. (#3505) - Make
HorovodVersionMismatchError
subclassImportError
instead of just a standardException
. (#3549) - Elastic: Catch any exception to prevent the discovery thread from silently dying. (#3436)
- Horovodrun: Exit check_build (
--check-build
) viasys.exit
to flush stdout. (#3272) - Spark: Use
env
to set environment vars in remote shell. (#3489) - Build: Avoid redundant ptx generation for maximum specified compute capability. (#3509)
- MXNet: Deprecated
average
argument of allreduce functions. (#3299) - Public and internal APIs: deprecate use of np, min_np, max_np. Use num_proc, min_num_proc, and max_num_proc, respectively, instead. (#3409)
- Horovodrun: Providing multiple NICS as comma-separated string via
--network-interface
is deprecated, use--network-interface
multiple times or--network-interfaces
instead. (#3506) - horovod.run: Argument
network_interface
with comma-separated string is deprecated, usenetwork_interfaces
withIterable[str]
instead. (#3506)
- Fallback to NCCL shared lib if static one is not found. (#3500
- Spark/Lightning: Added missing
tranform_spec
for Petastorm datamodule. (#3543) - Spark/Lightning: Fixed PTL Spark example with checkpoint usage by calling
save_hyperparameters()
. (#3527) - Elastic: Fixed empty hostname returned from
HostDiscoveryScript
. (#3490) - TensorFlow 2.9: Fixed build for API change related to
tensorflow_accelerator_device_info
. (#3513) - TensorFlow 2.10: Bumped build partially to C++17. (#3558)
- TensorFlow: Fixed gradient update timing in TF
AggregationHelperEager
. (#3496) - TensorFlow: Fixed resource
NotFoundError
in TFAggregationHelper
. (#3499)
- Make DBFSLocalStore support "file:/dbfs/...", implement get_localized_path. (#3510)
- Setup: Require fsspec >= 2010.07.0 (#3451)
- Fix ignored cuda arch flags (#3462
- Extended CMake build script to often find CUDA even if
nvcc
is not in$PATH
. (#3444)
- Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. (#3190)
- TensorFlow: Added in-place broadcasting of variables. (#3128)
- Elastic: Added support for resurrecting blacklisted hosts. (#3319)
- MXNet: Added support for MXNet async dependency engine. (#3242, #2963)
- Spark/Lightning: Added history to lightning estimator. (#3214)
- Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. (#3261, #3371)
- Moved released Docker image
horovod
andhorovod-cpu
to Ubuntu 20.04 and Python 3.8. (#3393) - Spark Estimator: Don't shuffle row groups if training data requires non-shuffle (#3369)
- Spark/Lightning: Reduced memory footprint of async dataloader. (#3239)
- Elastic: Improved handling NCCL errors under elastic scenario. (#3112)
- Spark/Lightning: Do not overwrite model with checkpoint by default. (#3201)
- Make checkpoint name optional so that user can save to h5 format. (#3411)
- Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. (#3190)
- Spark: Removed
h5py<3
constraint as this is not needed anymore for Tensorflow >2.5.0. (#3301)
- Elastic Spark: Fixed indices in initial task-to-task registration. (#3410)
- PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. (#3352)
- PyTorch: Fixed finalization of ProcessSetTable. (#3351)
- Fixed remote trainers to point to the correct shared lib path. (#3258)
- Fixed imports from
tensorflow.python.keras
with tensorflow 2.6.0+. (#3403) - Fixed Adasum communicator init logic. (#3379)
- Lightning: Fixed resume logger. (#3375)
- Fixed the checkpoint directory structure for pytorch and pytorch lightning. (#3362)
- Fixed possible integer overflow in multiplication. (#3368)
- Fixed the
pytorch_lightning_mnist.py
example. (#3245, #3290) - Fixed barrier segmentation fault. (#3313)
- Fixed
hvd.barrier()
tensor queue management. (#3300) - Fixed PyArrow "list index out of range" IndexError. (#3274)
- Elastic: Fixed all workers sometimes failing on elastic Horovod failure. (#3264)
- Spark/Lightning: Fixed setting
limit_train_batches
andlimit_val_batches
. (#3237) - Elastic: Fixed ElasticSampler and
hvd.elastic.state
losing some indices of processed samples when nodes dropped. (#3143) - Spark/Lightning: Fixed history metrics for estimator serialization. (#3216)
- Ray: Fixed RayExecutor to fail when
num_workers=0
andnum_hosts=None
. (#3210) - Spark/Lightning: Fixed checkpoint callback
dirpath
typo. (#3204)
- Added process sets to concurrently run collective operations on subsets of Horovod processes in TensorFlow, PyTorch, and MXNet. (#2839, #3042, #3043, #3054, #3083, #3090)
- Added XLA support for Allreduce via
tf.function(jit_compile=True)
. (#3053) - Added fused buffer scaling and unpack/pack kernels on GPU. (#2973)
- Added support for NCCL on CUDA 11.4. (#3182)
- Added fp16 compression for MXNet. (#2987)
- Added terminate_on_nan flag to Spark Lightning estimator. (#3088)
- Added barrier() API to torch module to support simple synchronization among ranks and to achieve parity with PyTorch DDP and similar frameworks. #3139
- Added params for customizing Tensorboard callback. (#3153)
- Added
hvd.cross_rank()
for keras. (#3008) - Added barrier() API to torch module to support simple synchronization among ranks and to achieve parity with PyTorch DDP and similar frameworks. #3139
- Implemented more asynchronous dependency handling on GPU. (#2963)
- Ray: RayExecutor will now use the current placement group instead of always creating a new one. (#3134)
- Lightning: turned off shuffling for validation dataset. (#2974)
- Ray: RayExecutor will use the current placement group if one exists. (#3134)
- Extended
hvd.join()
to return the last rank that joined. (#3097
- Spark/Keras: remove bare Keras support. (#3191)
- Fix Horovod develop/editable install mode and incremental builds. (#3074)
- Estimator/Lightning: use lightning datamodule. (#3084)
- Fix Horovod Spark StringType and numpy type mapping issue. (#3146)
- Fixed error in Keras LearningRateScheduler. (#3135)
- Fixed bug in Lightning Profiler on Ray. (#3122)
- Fixed torch op lazy release to prevent OOM in elastic training. (#3110)
- Lightning: Fixed usage of the checkpoint callback. (#3186)
- Fixed MPICH support to use Intel MPI's implementation. (#3148)
- Fixed race condition in PyTorch async dataloader. (#3120)
- Keras: Fixed learning rate scheduler. (#3142, #3135)
- Estimator: added support for loading data from S3, GCS, ADLS, and other remote filesystems. (#2927)
- Estimator: added custom Spark data loader interface. (#2938)
- LightningEstimator: added support to supply a logger and associated parameter to control the frequency of logging. (#2926)
- Estimator: added check to ensure all ranks have the same device type. (#2942)
- Changed behavior from using TensorBoardLogger to now using it as a fallback if a logger is not supplied. (#2926)
- Ray: disabled capturing child tasks in placement group. (#2920)
- Fixed
hvd.tensorflow.keras.Compression
, accidentally removed in v0.22.0. (#2945) - TorchEstimator: fixed usage of
validation_steps
in place ofvalidation_steps_per_epoch
. (#2918) - TensorFlow: fixed C++ API for TF v2.6.0. (#2932)
- PyTorch: fixed
sparse_allreduce_async
for PyTorch v0.10.0. (#2965)
- Added pytorch_lightning spark estimator which enables training pytorch_lightning models. (#2713)
- Added NVTX tracing hooks for profiling with Nsight Systems. (#2723)
- Added a generic
num_workers
API forRayExecutor
(#2870) - Supports Ray Client without code changes. (#2882)
- Supports inmemory cache option for Keras Estimator. (#2896)
- Added FP16 support for GPU tensor in mxnet. (#2915)
- Added response caching for allgather operations. (#2872)
- Estimator: add petastorm reader_pool_type into constructor (#2903)
- Changed
alltoall
to return the received splits as a second return value if non-uniform splits are sent. (#2631) - Changed
RayExecutor
to use Ray Placement Groups for worker colocation. (#2824) - Changed
Inmemory dataloader
usage for Torch Estimator with petastorm v0.11.0 release. (#2896)
- Changed RayExecutor to use Ray node ID to enable multi-container:single-host setups. (#2883)
- Support sparse gradients aggregation in TF1 Keras. (#2879)
- Respect
global_step
parameter for LegacyOptimizers when aggregating gradients. (#2879) - Fixed compatibility with PyTorch 1.9.0. (#2829)
- Add
groups
parameter inDistributedOptimizer
for custom allreduce groups. (#2523)
- Removed
num_groups
parameter inDistributedOptimizer
, replaced withgroups
. (#2523)
- Fixed worker desynchronization deadlock issue in TensorFlow 2.4. (#2647)
- Deduped Keras
LearningRateWarmupCallback
log after gradual learning rate warmup. (#2661)
- Added support for Intel(R) MPI in horovodrun. (#2374)
- Add support for callbacks in Ray Elastic Executor. (#2639)
- Added forwarding of stdout/stderr captured to driver over Gloo. (#2646)
- Fixed broadcast_optimizer_state to handle NoneType params for PyTorch 1.8. (#2624)
- Fixed
local_rank
support for Ray. (#2596) - Fixed DL estimators to obtain the output df schema without sampling the input. (#2611)
- Fixed wrong default for horovod.tensorflow.keras.allreduce average (#2627)
- Added in-memory dataset caching param to
TorchEstimator
. (#2434) - Added
val_batch_size
param to the Estimator API. (#2505) - Added support for TorchScript modules when using
TorchEstimator
. (#2494)
- Migrated to oneCCL aligned with oneAPI specification v1.0. (#2513)
- Added knob to set cache hint for oneCCL allreduce. (#2560)
- Renamed
horovodrun
arg--ccl-bgt-affinity
to--thread-affinity
. (#2562) - Changed default build parallelism from
-j8
to-j1
to address potential race condition. (#2572)
- Fixed building Horovod for ROCm PyTorch with newer hipify script. (#2360)
- Fixed "Executable class" support for Ray. (#2510)
- Fixed TorchEstimator returning model without switching to eval mode. (#2517)
- Remove ssh reliance for Ray elastic training. (#2528)
- Fixed error handling for changing framework without reinstalling horovod. (#2529)
- Fixed "Intermediate path does not exist" error with DBFSLocalStore. (#2526)
- Avoid synchronization if workers are only shrinked in elastic mode. (#2514)
- Fixed Ray resource test. (#2575)
- Fixed usage of env variable
HOROVOD_GLOO_TIMEOUT_SECONDS
withhorovodrun
. (#2571)
- Added support for backward_passes_per_step > 1 for TF Keras graph mode. (#2346)
- Added support for backward_passes_per_step > 1 for TF Keras eager execution. (#2371)
- Added support for backward_passes_per_step > 1 for TF LegacyOptimizer in graph mode. (#2401)
- Added grouped allreduce to enable more efficient tensor fusion and deterministic training. (#2453)
- Add support for specifying
op
andcompression
inhorovod.tensorflow.keras.allreduce()
. (#2423) - Adding support for batched D2D memcopy kernel on GPU. (#2435)
- Added schema inference in Spark Estimator without sampling. (#2373)
- Added
Store.create("dbfs:/")
mapping toDBFSLocalStore("/dbfs/...")
. (#2376)
- Changed Keras callbacks to require parameter
initial_lr
ofLearningRateScheduleCallback
andLearningRateWarmupCallback
. (#2459) - Changed default cycle time from 5ms to 1ms and fusion threshold from 64MB to 128MB. (#2468)
- Fixed support for TensorFlow v2.4.0. (#2381)
- Fixed averaging using CUDA half2 implementation one element half buffers. (#2375)
- Fixed
HOROVOD_THREAD_AFFINITY
when using oneCCL. (#2350) - Added timeout to SSH check in horovodrun to prevent hanging. (#2448)
- Added
HOROVOD_GLOO_TIMEOUT_SECONDS
value to error messages. (#2436) - Fixed race condition in dynamic timeline API. (#2341)
- Fixed --log-hide-timestamp to apply to driver logs with Gloo. (#2388)
- Fixed the search order of Eigen and Flatbuffers paths. (#2473)
- Fixed type checks in
TorchEstimator
to correctly useisinstance()
. (#2480)
- Added Elastic Ray integration. (#2291)
- Removed dependency on SSH access for Ray. (#2275)
- Fixed building Horovod without HOROVOD_WITHOUT_MXNET when MXNet is not installed. (#2334)
- Added Databricks storage
DBFSLocalStore
and support for GPU-aware scheduling to horovod.spark Estimator. (#2234) - Added ElasticSampler and PyTorch Elastic ImageNet example. (#2297)
- Added ability to dynamically start and stop timeline programmatically. (#2215)
- Added support for Gloo on macOS. (#2254)
- Exposed name argument to TensorFlow allreduce operation. (#2325)
- Added option to strip outer name scope from Horovod ops in TensorFlow. (#2328)
- Fixed usage of VERBOSE=1 when setting custom MAKEFLAGS. (#2239)
- Fixed bugs in Keras Elastic Callback classes. (#2289)
- Fixed RelWithDebInfo build and made it the default with -03 optimizations. (#2305)
- Fixed usage of tf.cond in TensorFlow alltoall gradient. (#2327)
- Fixed allreduce averaging for TF IndexedSlices in ROCm path. (#2279)
- Include stdexcept to handle certain compiler / frameworks that don't include it already. (#2238)
- Fixed Debug builds by setting compiler options based on CMake build type. (#2263)
- Skipped launching zero-sized send/recvs for NCCLAlltoall. (#2273)
- Fixed missing run in tf keras elastic mode. (#2272)
- Fixed loss function in TensorFlow2 elastic synthetic benchmark. (#2265)
- Fixed usage of HOROVOD_MIXED_INSTALL env var in alltoall tests. (#2266)
- Removed keras requirement from Ray example. (#2262)
- Added bare-metal elastic mode implementation to enable auto-scaling and fault tolerance. (#1849)
- Added Elastic Horovod support for Spark auto-scaling. (#1956)
- Added All-to-All operation for TensorFlow, PyTorch, and MXNet. (#2143)
- Added support for
gradient_predivide_factor
and averaging in Horovod backend. (#1949) - Added NCCL implementation of the allgather operation. (#1952)
- Added
HOROVOD_GPU_OPERATIONS
installation variable to simplify enabling NCCL support for all GPU operations. (#1960) - Added TensorFlow implementation of
SyncBatchNormalization
layer. (#2075) - Added
hvd.is_initialized()
method. (#2020) - Added
hvd.allgather_object
function for TensorFlow, PyTorch, and MXNet. (#2166) - Added
hvd.broadcast_object
function for MXNet. (#2122) - Added
label_shapes
parameter to KerasEstimator and TorchEstimator. (#2140) - Added optional
modelCheckPoint
callback to KerasEstimator params. (#2124) - Added
ssh_identity_file
argument tohorovodrun
. (#2201) - Added support for
horovodrun
onkubeflow/mpi-job
. (#2199) - Added Ray integration. (#2218)
- Moved
horovod.run.runner.run
tohorovod.run
. (#2099) - HOROVOD_THREAD_AFFINITY accepts multiple values, one for every Horovod rank (#2131)
- Migrated build system for native libraries to CMake (#2009)
- HOROVOD_CCL_BGT_AFFINITY is deprected. Use HOROVOD_THREAD_AFFINITY instead (#2131)
- Dropped support for Python 2. (#1954)
- Dropped support for TensorFlow < 1.15. (#2169)
- Dropped support for PyTorch < 1.2. (#2086)
- Fixed MXNet allgather implementation to correctly handle resizing the output buffer. (#2092)
- Fixed Keras Spark Estimator incompatibility with TensorFlow 1.15 due to
tf.autograph
. (#2069) - Fixed API compatibility with PyTorch 1.6. (#2051)
- Fixed Keras API compatibility with TensorFlow 2.4.0. (#2178)
- Fixed allgather gradient for TensorFlow 2 in cases where the tensor shape is not known during graph construction. (#2121)
- Fixed running using Gloo with an imbalanced number of workers per host. (#2212)