Code

[WIP] Voltron v0 #1

Code

Open

winglian wants to merge 42 commits intomainOpenAccess-AI-Collective/voltronformers:mainfrom voltron-v0OpenAccess-AI-Collective/voltronformers:voltron-v0Copy head branch name to clipboard

+2,038

Commits

Commits on Apr 13, 2024

• wip for v0 for testing

  winglian

  committed
  f43824f

• mini fixes

  winglian

  committed
  b7cd6ea

• fix install and train

  winglian

  committed
  9b56525

• fix config and dataset

  winglian

  committed
  a357c7f

• fix lr

  winglian

  committed
  1bfbdb5

• fix text field of dataset

  winglian

  committed
  072b74a

• improve data handling

  winglian

  committed
  38a872d

• flesh out the model w/ attn

  winglian

  committed
  6c0e92b

• fix args/kwargs ordering

  winglian

  committed
  91c4fa2

• use LlamaBitMGQA

  winglian

  committed
  c298db1

• fix order of init for module

  winglian

  committed
  b5487fd

• make tinier and fix dataset map

  winglian

  committed
  a7854a2

• fix back to use dataset.data.columns

  winglian

  committed
  bc4ce8a

• make sure to remove extra columns

  winglian

  committed
  122316e

• fix data loop and make tinier

  winglian

  committed
  a1c7b2a

• use generic collator to pad equally

  winglian

  committed
  0d76c4e

• accont for position_ids in mod block

  winglian

  committed
  91d4a11

• flesh out rotary embeddigs

  winglian

  committed
  81e18b9

• misc fixes

  winglian

  committed
  bc5ac07

• fix tokenizer and activation checkpointing

  winglian

  committed
  c5a4c71

• more fixes

  winglian

  committed
  c493b65

• remove hard dependencies from axolotl

  winglian

  committed
  8aa8d81

• remove more hard deps

  winglian

  committed
  e6edf93

• re-enable DWA again

  winglian

  committed
  270150f

• actually check for dwa

  winglian

  committed
  49cc04a

• wandb on main rank only

  winglian

  committed
  4437672

• fix modulo for log steps

  winglian

  committed
  c2f804c

• attempt to use accelerator loop

  winglian

  committed
  88f25a9

• update configuration

  winglian

  committed
  a10c31a

Commits on Apr 14, 2024

• wip rms norm

  winglian

  committed
  8239c6e

• use apex rms norm optim

  winglian

  committed
  2b2f332

• queued dataloader and gradient norm

  winglian

  committed
  cba6e66

• fixes for loss calc, grad accum, dataloader for dispatch_batches

  winglian

  committed
  ffafd7a

Commits on Apr 15, 2024

• tweak size names

  winglian

  committed
  0255ffe

• upcast/downcast

  winglian

  committed
  d289d98

• integrate infini-attention

  winglian

  committed
  84d755a

• handle position_id if passed, throw it on the floor

  winglian

  committed
  665530b

• match infini-attention segment len to mixture of depth

  winglian

  committed
  2439688

• misc fixes for integrations

  winglian

  committed
  a77e6ae

• make infini-attention work

  winglian

  committed
  6a49def

• fix dimensions passed to infini-attention

  winglian

  committed
  555087a

Commits on Apr 16, 2024

• fix perplexity calculation and add quick instructions

  winglian

  committed
  b117faa