Code

zero3 performance optimizations#3622

Code

Merged

tjruwase merged 13 commits intodeepspeedai:masterdeepspeedai/DeepSpeed:masterfrom BacharL:perf1BacharL/DeepSpeed-hablb:perf1Copy head branch name to clipboardJun 8, 2023

+91-55

Commits

Commits on May 22, 2023

• Remove dead code

  Show description for e1e4b75
  BacharL

  committed
  e1e4b75

  Copy full SHA for e1e4b75

• Prevent evaluation of debug strings

  Show description for e3dbb7a
  BacharL

  committed
  e3dbb7a

  Copy full SHA for e3dbb7a

Commits on May 29, 2023

• Use contiguous gradients tensor reduce scatter between ranks

  Show description for bd4d724
  BacharL

  committed
  bd4d724

  Copy full SHA for bd4d724

Commits on May 30, 2023

• move overflow tracker to optimizer.step

  Show description for 9cf826d
  BacharL

  committed
  9cf826d

  Copy full SHA for 9cf826d

• Merge branch 'master' into perf1

  tjruwase

  authored
  6847b30

  Copy full SHA for 6847b30

Commits on Jun 1, 2023

• Merge branch 'master' into perf1

  tjruwase

  authored
  9b80d60

  Copy full SHA for 9b80d60

Commits on Jun 2, 2023

• Merge branch 'master' into perf1

  tjruwase

  authored
  a813c77

  Copy full SHA for a813c77

• Merge branch 'master' into perf1

  tjruwase

  authored
  b0bfdc1

  Copy full SHA for b0bfdc1

Commits on Jun 5, 2023

• Merge branch 'master' into perf1

  tjruwase

  authored
  7b83224

  Copy full SHA for 7b83224

Commits on Jun 6, 2023

• Merge branch 'master' into perf1

  tjruwase

  authored
  89add18

  Copy full SHA for 89add18

• Merge branch 'master' into perf1

  tjruwase

  authored
  16e6f1c

  Copy full SHA for 16e6f1c

Commits on Jun 7, 2023

• Merge branch 'master' into perf1

  tjruwase

  authored
  290bdbd

  Copy full SHA for 290bdbd

Commits on Jun 8, 2023

• Merge branch 'master' into perf1

  BacharL

  authored
  df88b94

  Copy full SHA for df88b94