Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation #898

Merged
merged 84 commits into from
Jan 23, 2024
Merged

Validation #898

merged 84 commits into from
Jan 23, 2024

Commits on Dec 23, 2023

  1. add validation script

    @xiaohanzhan-db
    xiaohanzhan-db committed Dec 23, 2023
    Configuration menu
    Copy the full SHA
    8cb6522 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    c59c11f View commit details
    Browse the repository at this point in the history

  2. change token count function

    xiaohanzhan-db committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    66f34eb View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. reorganize cells

    xiaohanzhan-db committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    2cd387b View commit details
    Browse the repository at this point in the history

  2. Add unit tests

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    3eac3bf View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2024

  1. Add a printout for CPT

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    d2d9767 View commit details
    Browse the repository at this point in the history

  2. update question

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    be25591 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. Add questions

    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    4651be7 View commit details
    Browse the repository at this point in the history

  2. Fix lints

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    5cd6a94 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into validation

    @XiaohanZhangCMU
    XiaohanZhangCMU authored Jan 8, 2024
    Configuration menu
    Copy the full SHA
    8e2c1f4 View commit details
    Browse the repository at this point in the history

  4. update format

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    e6e4a81 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX … 

    …into validation
    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    34c5690 View commit details
    Browse the repository at this point in the history

  6. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    1668b9a View commit details
    Browse the repository at this point in the history

  7. nb source

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    2219135 View commit details
    Browse the repository at this point in the history

  8. add validation script

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    86c6e87 View commit details
    Browse the repository at this point in the history

  9. update

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    678b376 View commit details
    Browse the repository at this point in the history

  10. change token count function

    @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    297e057 View commit details
    Browse the repository at this point in the history

  11. reorganize cells

    @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    09d0ebb View commit details
    Browse the repository at this point in the history

  12. Add unit tests

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    460df65 View commit details
    Browse the repository at this point in the history

  13. Add a printout for CPT

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    3ffd200 View commit details
    Browse the repository at this point in the history

  14. update question

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    9362886 View commit details
    Browse the repository at this point in the history

  15. Add questions

    @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    898e5ac View commit details
    Browse the repository at this point in the history

  16. Fix lints

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    a4bef71 View commit details
    Browse the repository at this point in the history

  17. update format

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    4ca9cc6 View commit details
    Browse the repository at this point in the history

  18. update

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    d636a0f View commit details
    Browse the repository at this point in the history

  19. nb source

    @xiaohanzhan-db @XiaohanZhangCMU
    xiaohanzhan-db authored and XiaohanZhangCMU committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    827d155 View commit details
    Browse the repository at this point in the history

  20. Remove license insert for validation notebook

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    6bbf3fc View commit details
    Browse the repository at this point in the history

  21. Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX … 

    …into validation
    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    4f6a4fb View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. Add validation utils

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    5966b68 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into validation

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    da17813 View commit details
    Browse the repository at this point in the history

  3. Minor cleanups (mosaicml#858) 

    * nits

* logger

* add log

* lint
    @mvpatel2000
    mvpatel2000 authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    a7c36bc View commit details
    Browse the repository at this point in the history

  4. update utils/__init__.py to include extra validation functions

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    55e4626 View commit details
    Browse the repository at this point in the history

  5. update notebook

    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    45544a1 View commit details
    Browse the repository at this point in the history

  6. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    d2797b3 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX … 

    …into validation
    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    019da77 View commit details
    Browse the repository at this point in the history

  8. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    756fdae View commit details
    Browse the repository at this point in the history

  9. Read UC delta table (mosaicml#773) 

    * initial commit

* use databricks-sql to read delta table and convert to json

* update

* update

* update

* add mocked unittest

* Fix lints

* update

* update

* restructure code

* Add timer for optimizing

* Add db-connect

* add wrapper

* update

* add install dbconnect

* update

* update

* patch dbconnect to allow multiple return formats

* update

* add arrow

* use compression

* clean up

* Add cluster rt check

* Fix lints

* remove patch.py for CI

* update

* update

* updat

* update

* fix tests

* fix lint

* update

* update

* Add more tests

* update

* update

* update

* change to download_json

* update

* fix lints

* Add decompressed option for arrow

* format json to jsonl

* Add comments

* Make cf_collect_type global option

* fix comments

* fix lints

* fix comments

* Fix lints

* change to use workspaceclient

* Add CPT support

* Rewire method assignment logic

* Fix bug in stripping https

* Add tests for rewired method assignment logic

* Fix lints

* Fix lints

* Removed logger set_level

* Remove pyspark. It conflicts with databricks-connect

* Update the comment

* skip cluster version check when cluster_id is serverless

* Add use_serverless flag

* update tests with use_serverless flag

* Fix lints

---------

Co-authored-by: Xiaohan Zhang <[email protected]>
    @XiaohanZhangCMU @xiaohanzhan-db
    XiaohanZhangCMU and xiaohanzhan-db authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    6de8c37 View commit details
    Browse the repository at this point in the history

  10. Add download remote function to util

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    93b5a9f View commit details
    Browse the repository at this point in the history

  11. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    b47c878 View commit details
    Browse the repository at this point in the history

  12. remove fused layernorm (mosaicml#859)

    @mvpatel2000
    mvpatel2000 authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    fa8f3d9 View commit details
    Browse the repository at this point in the history

  13. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    13fd34c View commit details
    Browse the repository at this point in the history

  14. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    610f669 View commit details
    Browse the repository at this point in the history

  15. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    9f2e51b View commit details
    Browse the repository at this point in the history

  16. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    ec68f10 View commit details
    Browse the repository at this point in the history

  17. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    1e76068 View commit details
    Browse the repository at this point in the history

  18. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    7a5c164 View commit details
    Browse the repository at this point in the history

  19. Merge branch 'main' into validation

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e76038f View commit details
    Browse the repository at this point in the history

  20. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    5b413f5 View commit details
    Browse the repository at this point in the history

  21. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    a1aa31f View commit details
    Browse the repository at this point in the history

  22. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    d24fd5c View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. Remove hardcoded combined.jsonl with a flag (mosaicml#861) 

    * Remove hardcoded combined.jsonl with a flag

* update

* change output_json_path output_json_folder

---------

Co-authored-by: Xiaohan Zhang <[email protected]>
    @XiaohanZhangCMU @xiaohanzhan-db
    XiaohanZhangCMU and xiaohanzhan-db authored Jan 12, 2024
    Configuration menu
    Copy the full SHA
    da3bea1 View commit details
    Browse the repository at this point in the history

  2. bump (mosaicml#828)

    @mvpatel2000
    mvpatel2000 authored Jan 12, 2024
    Configuration menu
    Copy the full SHA
    936e3a1 View commit details
    Browse the repository at this point in the history

  3. Add dask and dataframe_to_mds

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    55fce37 View commit details
    Browse the repository at this point in the history

  4. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    86e2412 View commit details
    Browse the repository at this point in the history

  5. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    bbfec65 View commit details
    Browse the repository at this point in the history

  6. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    b2e880d View commit details
    Browse the repository at this point in the history

  7. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    596443a View commit details
    Browse the repository at this point in the history

  8. Add notebook

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    ea65187 View commit details
    Browse the repository at this point in the history

  9. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    378a4e0 View commit details
    Browse the repository at this point in the history

  10. update

    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    af6e9aa View commit details
    Browse the repository at this point in the history

  11. remove script and tests, keep notebook

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    4e286ec View commit details
    Browse the repository at this point in the history

  12. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    09c4892 View commit details
    Browse the repository at this point in the history

  13. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    c82da6c View commit details
    Browse the repository at this point in the history

  14. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    e5f83cc View commit details
    Browse the repository at this point in the history

  15. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    17d2b9f View commit details
    Browse the repository at this point in the history

  16. Merge branch 'main' into validation

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    6579d55 View commit details
    Browse the repository at this point in the history

  17. Merge branch 'byod/data_validation' into validation

    @XiaohanZhangCMU
    XiaohanZhangCMU authored Jan 12, 2024
    Configuration menu
    Copy the full SHA
    56308ff View commit details
    Browse the repository at this point in the history

  18. Always initialize dist (mosaicml#864) 

    * fix dev

* lint

* remove gpu
    @mvpatel2000
    mvpatel2000 authored Jan 12, 2024
    Configuration menu
    Copy the full SHA
    6517a30 View commit details
    Browse the repository at this point in the history

  19. updated notebook

    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    4daa324 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'main' into validation

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    b809691 View commit details
    Browse the repository at this point in the history

  21. remove scripts keep notebook

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    8b75f94 View commit details
    Browse the repository at this point in the history

  22. merge with byod/data_validation

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    99bf2cd View commit details
    Browse the repository at this point in the history

  23. update notebook. rephrase.

    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    22014d6 View commit details
    Browse the repository at this point in the history

  24. merged

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    d9f28aa View commit details
    Browse the repository at this point in the history

  25. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    43c8ac9 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. Add response tokens

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    b8ac771 View commit details
    Browse the repository at this point in the history

  2. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    1b9681c View commit details
    Browse the repository at this point in the history

  3. merge

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    16883c2 View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2024

  1. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 20, 2024
    Configuration menu
    Copy the full SHA
    c7567f1 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. Disable MDSWrite, return token counts

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    1764b72 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Change plot settings

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    808ced5 View commit details
    Browse the repository at this point in the history

  2. Fix conflict

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    26ae516 View commit details
    Browse the repository at this point in the history

  3. update notebook

    xiaohanzhan-db committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    a212ee8 View commit details
    Browse the repository at this point in the history

  4. update

    @xiaohanzhan-db
    xiaohanzhan-db committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    d279817 View commit details
    Browse the repository at this point in the history