Code

Adds mixing loader for FSL datasets#70

Code

Merged

undfined merged 61 commits intomainallenai/OLMo-core:mainfrom undfined/mixing-loaderallenai/OLMo-core:undfined/mixing-loaderCopy head branch name to clipboardNov 1, 2024

+1,231-93

Commits

Commits on Oct 18, 2024

• WIP: Generate a mixture dataset

  undfined

  committed
  a8b5ec6

• WIP: Adds dry run

  undfined

  committed
  637fee9

• Test cleanup

  undfined

  committed
  346135c

Commits on Oct 19, 2024

• WIP: Make it fast

  undfined

  committed
  53def38

• WIP: Simple benchmark

  undfined

  committed
  8649cc8

Commits on Oct 23, 2024

• WIP: Refactor

  undfined

  committed
  e3d7011

• Launch script

  undfined

  committed
  efe766b

• temp changes to test

  undfined

  committed
  5dff40c

• deps for now

  undfined

  committed
  2703538

• Try with session

  undfined

  committed
  3ee3278

• Try internal client

  undfined

  committed
  cd1c6d2

• Try boto3

  undfined

  committed
  9895b23

Commits on Oct 24, 2024

• Fixes

  undfined

  committed
  3c15f52

• ?

  undfined

  committed
  0c9355b

• Cleanup + session stuff

  undfined

  committed
  abb362a

• Use environ

  undfined

  committed
  82a1af9

• JUST use env vars please boto

  undfined

  committed
  d0a80ba

• No unions of containers

  undfined

  committed
  e621f8e

• prepare first

  undfined

  committed
  0689c42

• Loader handles prepare

  undfined

  committed
  8ab2e99

• Try recording torch exceptions

  undfined

  committed
  dcfda67

• Don't need overrides

  undfined

  committed
  23a0806

• Figure out why config/creds are missing

  undfined

  committed
  8cfa282

• fmt

  undfined

  committed
  fd1a508

• Env not ready yet

  undfined

  committed
  01a40ea

• print beaker user

  undfined

  committed
  8bde2b3

• uncomment eval file

  undfined

  committed
  ae208f6

• replicate CommonComponents setup

  undfined

  committed
  ce9d06f

• Some class init stuff

  undfined

  committed
  d1eb4df

• Some more config logging

  undfined

  committed
  dbce279

• Conflict in CHANGELOG

  undfined

  committed
  5daa274

• checks cleanup

  undfined

  committed
  980e05a

Commits on Oct 25, 2024

• Fixes for duplicate paths in mixture

  undfined

  committed
  f27bd73

• In case there a ton of files

  undfined

  committed
  c69b228

• Maybe fix trainer launch

  undfined

  committed
  18efafd

• Match other example

  undfined

  committed
  5ceed46

• More tests

  undfined

  committed
  4c7513e

• Try diff gpus

  undfined

  committed
  8401580

• keep fsdp

  undfined

  committed
  0d77422

• checks

  undfined

  committed
  d22ed10

• Less tokens

  undfined

  committed
  68a4d28

• Exclude ai2/allennlp-elanding-a100-40g temp

  undfined

  committed
  c35514c

Commits on Oct 28, 2024

• Merge branch 'main' of github.com:allenai/OLMo-core into undfined/mixing-loader

  undfined

  committed
  02cb49b

• Feedback

  undfined

  committed
  c453e65

• Drop examples

  undfined

  committed
  a288d9e

• A bit more cleanup

  undfined

  committed
  9c49f25

• Outdated changelog

  undfined

  committed
  89504bc

• Unused deps

  undfined

  committed
  3aa5c35

• One more dep

  undfined

  committed
  8f729dd

• uncomment test assertions

  undfined

  committed
  a848195

• Drop todo

  undfined

  committed
  5322bf1

• 0 is an invalid token

  undfined

  committed
  5c22665

Commits on Oct 29, 2024

• More feedback

  undfined

  committed
  87e9168

• Randomly sample instances when segmenting

  undfined

  committed
  fe50a32

Commits on Oct 30, 2024

• Memray + limit marker

  undfined

  committed
  cffcba3

• Add dep

  undfined

  committed
  293be02

• Lint

  undfined

  committed
  e45d2c3

• Bigger array is more informative

  undfined

  committed
  ffe7660

• Merge branch 'main' into undfined/mixing-loader

  undfined

  authored
  19db2a9

• Feedback

  undfined

  committed
  1fdb995

Commits on Nov 1, 2024

• Merge branch 'main' into undfined/mixing-loader

  undfined

  authored
  74d4c5a