Code

Validation#866

Code

Merged

XiaohanZhangCMU merged 72 commits intomosaicml:byod/data_validationmosaicml/llm-foundry:byod/data_validationfrom XiaohanZhangCMU:validationXiaohanZhangCMU/llm-foundryX:validationCopy head branch name to clipboardJan 12, 2024

+94-1,271

Commits

Commits on Dec 23, 2023

• add validation script

  xiaohanzhan-db

  committed
  8cb6522

Commits on Jan 3, 2024

• update

  xiaohanzhan-db

  committed
  c59c11f

• change token count function

  

  xiaohanzhan-db

  committed
  66f34eb

Commits on Jan 5, 2024

• reorganize cells

  

  xiaohanzhan-db

  committed
  2cd387b

• Add unit tests

  xiaohanzhan-db

  committed
  3eac3bf

Commits on Jan 6, 2024

• Add a printout for CPT

  xiaohanzhan-db

  committed
  d2d9767

• update question

  xiaohanzhan-db

  committed
  be25591

Commits on Jan 8, 2024

• Add questions

  

  xiaohanzhan-db

  committed
  4651be7

• Fix lints

  xiaohanzhan-db

  committed
  5cd6a94

• Merge branch 'main' into validation

  XiaohanZhangCMU

  authored
  8e2c1f4

• update format

  xiaohanzhan-db

  committed
  e6e4a81

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  34c5690

• update

  xiaohanzhan-db

  committed
  1668b9a

• nb source

  xiaohanzhan-db

  committed
  2219135

• add validation script

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  86c6e87

• update

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  678b376

• change token count function

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  297e057

• reorganize cells

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  09d0ebb

• Add unit tests

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  460df65

• Add a printout for CPT

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  3ffd200

• update question

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  9362886

• Add questions

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  898e5ac

• Fix lints

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  a4bef71

• update format

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  4ca9cc6

• update

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  d636a0f

• nb source

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  827d155

• Remove license insert for validation notebook

  xiaohanzhan-db

  committed
  6bbf3fc

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  4f6a4fb

Commits on Jan 11, 2024

• Add validation utils

  xiaohanzhan-db

  committed
  5966b68

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  da17813

• Minor cleanups (mosaicml#858)

  Show description for a7c36bc
  mvpatel2000

  authored
  a7c36bc

• update utils/__init__.py to include extra validation functions

  xiaohanzhan-db

  committed
  55e4626

• update notebook

  

  xiaohanzhan-db

  committed
  45544a1

• update

  xiaohanzhan-db

  committed
  d2797b3

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  019da77

• update

  xiaohanzhan-db

  committed
  756fdae

• Read UC delta table (mosaicml#773)

  Show description for 6de8c37
  

  XiaohanZhangCMU

  and

  xiaohanzhan-db

  authored
  6de8c37

• Add download remote function to util

  xiaohanzhan-db

  committed
  93b5a9f

• update

  xiaohanzhan-db

  committed
  b47c878

• remove fused layernorm (mosaicml#859)

  mvpatel2000

  authored
  fa8f3d9

• update

  xiaohanzhan-db

  committed
  13fd34c

• update

  xiaohanzhan-db

  committed
  610f669

• update

  xiaohanzhan-db

  committed
  9f2e51b

• update

  xiaohanzhan-db

  committed
  ec68f10

• update

  xiaohanzhan-db

  committed
  1e76068

• update

  xiaohanzhan-db

  committed
  7a5c164

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  e76038f

• update

  xiaohanzhan-db

  committed
  5b413f5

• update

  xiaohanzhan-db

  committed
  a1aa31f

• update

  xiaohanzhan-db

  committed
  d24fd5c

Commits on Jan 12, 2024

• Remove hardcoded combined.jsonl with a flag (mosaicml#861)

  Show description for da3bea1
  

  XiaohanZhangCMU

  and

  xiaohanzhan-db

  authored
  da3bea1

• bump (mosaicml#828)

  mvpatel2000

  authored
  936e3a1

• Add dask and dataframe_to_mds

  xiaohanzhan-db

  committed
  55fce37

• update

  xiaohanzhan-db

  committed
  86e2412

• update

  xiaohanzhan-db

  committed
  bbfec65

• update

  xiaohanzhan-db

  committed
  b2e880d

• update

  xiaohanzhan-db

  committed
  596443a

• Add notebook

  xiaohanzhan-db

  committed
  ea65187

• update

  xiaohanzhan-db

  committed
  378a4e0

• update

  

  xiaohanzhan-db

  committed
  af6e9aa

• remove script and tests, keep notebook

  xiaohanzhan-db

  committed
  4e286ec

• update

  xiaohanzhan-db

  committed
  09c4892

• update

  xiaohanzhan-db

  committed
  c82da6c

• update

  xiaohanzhan-db

  committed
  e5f83cc

• update

  xiaohanzhan-db

  committed
  17d2b9f

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  6579d55

• Merge branch 'byod/data_validation' into validation

  XiaohanZhangCMU

  authored
  56308ff

• Always initialize dist (mosaicml#864)

  Show description for 6517a30
  mvpatel2000

  authored
  6517a30

• updated notebook

  

  xiaohanzhan-db

  committed
  4daa324

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  b809691

• remove scripts keep notebook

  xiaohanzhan-db

  committed
  8b75f94

• merge with byod/data_validation

  xiaohanzhan-db

  committed
  99bf2cd