Code

[trainer] deepspeed integration#9211

Code

Merged

stas00 merged 69 commits intohuggingface:masterhuggingface/transformers:masterfrom stas00:dsstas00/transformers:dsCopy head branch name to clipboardJan 13, 2021

+741-57

Commits

Commits on Dec 19, 2020

• deepspeed integration

  stas00

  committed
  b99b665

  Copy full SHA for b99b665

• style

  stas00

  committed
  cf2f0d2

  Copy full SHA for cf2f0d2

• add test

  stas00

  committed
  d417f55

  Copy full SHA for d417f55

Commits on Dec 22, 2020

• ds wants to do its own backward

  stas00

  committed
  112be60

  Copy full SHA for 112be60

• fp16 assert

  stas00

  committed
  4c2809d

  Copy full SHA for 4c2809d

• Update src/transformers/training_args.py

  Show description for f4de6ff
  

  stas00

  and

  sgugger

  authored
  f4de6ff

  Copy full SHA for f4de6ff

• Merge branch 'ds' of github.com:stas00/transformers into ds

  stas00

  committed
  bd350f6

  Copy full SHA for bd350f6

• style

  stas00

  committed
  8565c8a

  Copy full SHA for 8565c8a

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  9653f8e

  Copy full SHA for 9653f8e

Commits on Dec 23, 2020

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  e980e21

  Copy full SHA for e980e21

• for clarity extract what args are being passed to deepspeed

  stas00

  committed
  9cc3b63

  Copy full SHA for 9cc3b63

• introduce the concept of self.wrapped_model

  stas00

  committed
  1510444

  Copy full SHA for 1510444

Commits on Dec 25, 2020

• s/self.wrapped_model/self.model_wrapped/

  stas00

  committed
  f28566e

  Copy full SHA for f28566e

• complete transition to self.wrapped_model / self.model

  stas00

  committed
  caa32dc

  Copy full SHA for caa32dc

• fix

  stas00

  committed
  9f199e7

  Copy full SHA for 9f199e7

• doc

  stas00

  committed
  fb0c13e

  Copy full SHA for fb0c13e

• give ds its own init

  stas00

  committed
  6af645f

  Copy full SHA for 6af645f

• add custom overrides, handle bs correctly

  stas00

  committed
  765594d

  Copy full SHA for 765594d

• fix test

  stas00

  committed
  8ebe5c7

  Copy full SHA for 8ebe5c7

• clean up model_init logic, fix small bug

  stas00

  committed
  e8c1080

  Copy full SHA for e8c1080

• complete fix

  stas00

  committed
  3b7c581

  Copy full SHA for 3b7c581

• collapse --deepspeed_config into --deepspeed

  stas00

  committed
  aa8f9a1

  Copy full SHA for aa8f9a1

• style

  stas00

  committed
  a83c46a

  Copy full SHA for a83c46a

• start adding doc notes

  stas00

  committed
  aaf97e1

  Copy full SHA for aaf97e1

• style

  stas00

  committed
  3dedd7a

  Copy full SHA for 3dedd7a

Commits on Jan 2, 2021

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  af22ec1

  Copy full SHA for af22ec1

Commits on Jan 3, 2021

• implement hf2ds optimizer and scheduler configuration remapping

  stas00

  committed
  869173f

  Copy full SHA for 869173f

• oops

  stas00

  committed
  2f73bfe

  Copy full SHA for 2f73bfe

Commits on Jan 4, 2021

• call get_num_training_steps absolutely when needed

  stas00

  committed
  9c77823

  Copy full SHA for 9c77823

• workaround broken auto-formatter

  stas00

  committed
  e6610da

  Copy full SHA for e6610da

Commits on Jan 5, 2021

• deepspeed_config arg is no longer needed - fixed in deepspeed master

  stas00

  committed
  a1ed838

  Copy full SHA for a1ed838

• use hf's fp16 args in config

  stas00

  committed
  25a93d9

  Copy full SHA for 25a93d9

Commits on Jan 6, 2021

• clean

  stas00

  committed
  39f6446

  Copy full SHA for 39f6446

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  b3667b0

  Copy full SHA for b3667b0

• start on the docs

  stas00

  committed
  df1154c

  Copy full SHA for df1154c

Commits on Jan 7, 2021

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  c5afaec

  Copy full SHA for c5afaec

• rebase cleanup

  stas00

  committed
  c9a6266

  Copy full SHA for c9a6266

• finish up --fp16

  stas00

  committed
  8bb2462

  Copy full SHA for 8bb2462

• clarify the supported stages

  stas00

  committed
  c9e7de7

  Copy full SHA for c9e7de7

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  33087ca

  Copy full SHA for 33087ca

Commits on Jan 8, 2021

• big refactor thanks to discovering deepspeed.init_distributed

  stas00

  committed
  dc00de7

  Copy full SHA for dc00de7

• cleanup

  stas00

  committed
  ad967b4

  Copy full SHA for ad967b4

• revert fp16 part

  stas00

  committed
  52d8000

  Copy full SHA for 52d8000

• add checkpoint-support

  stas00

  committed
  31dba17

  Copy full SHA for 31dba17

• more init ds into integrations

  stas00

  committed
  6c5432a

  Copy full SHA for 6c5432a

• extend docs

  stas00

  committed
  8364b27

  Copy full SHA for 8364b27

• cleanup

  stas00

  committed
  f462b5c

  Copy full SHA for f462b5c

• unfix docs

  stas00

  committed
  152c239

  Copy full SHA for 152c239

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  38ce263

  Copy full SHA for 38ce263

• clean up old code

  stas00

  committed
  2d4de17

  Copy full SHA for 2d4de17

• imports

  stas00

  committed
  f90822f

  Copy full SHA for f90822f

• move docs

  stas00

  committed
  42260fe

  Copy full SHA for 42260fe

• fix logic

  stas00

  committed
  dda4a44

  Copy full SHA for dda4a44

• make it clear which file it's referring to

  stas00

  committed
  f9d6028

  Copy full SHA for f9d6028

• document nodes/gpus

  stas00

  committed
  57f5888

  Copy full SHA for 57f5888

• style

  stas00

  committed
  c8ef31e

  Copy full SHA for c8ef31e

• wrong format

  stas00

  committed
  c0f5e1b

  Copy full SHA for c0f5e1b

• style

  stas00

  committed
  20e3ab6

  Copy full SHA for 20e3ab6

Commits on Jan 10, 2021

• deepspeed handles gradient clipping

  stas00

  committed
  777ae49

  Copy full SHA for 777ae49

• easier to read

  stas00

  committed
  0bbc65e

  Copy full SHA for 0bbc65e

• major doc rewrite

  stas00

  committed
  c65a680

  Copy full SHA for c65a680

Commits on Jan 11, 2021

• Apply suggestions from code review

  Show description for 4b9dd76
  

  stas00

  and

  sgugger

  authored
  4b9dd76

  Copy full SHA for 4b9dd76

• docs

  stas00

  committed
  36c6f57

  Copy full SHA for 36c6f57

• switch to AdamW optimizer

  stas00

  committed
  7f5e579

  Copy full SHA for 7f5e579

• style

  stas00

  committed
  b210ae3

  Copy full SHA for b210ae3

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  96b7d3a

  Copy full SHA for 96b7d3a

Commits on Jan 12, 2021

• Apply suggestions from code review

  Show description for d8da0c7
  

  stas00

  and

  LysandreJik

  authored
  d8da0c7

  Copy full SHA for d8da0c7

• clarify doc

  stas00

  committed
  19ad552

  Copy full SHA for 19ad552

• Merge remote-tracking branch 'origin/master' into ds

  stas00

  committed
  19e4972

  Copy full SHA for 19e4972