Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow DataLoaderAdapter subclasses to be pickled by implementing __reduce__ #3074

Merged
merged 20 commits into from
Sep 5, 2024
Merged

Allow DataLoaderAdapter subclasses to be pickled by implementing __reduce__ #3074

merged 20 commits into from
Sep 5, 2024

Commits on Sep 3, 2024

  1. initial fix for breaking accelerator pickling

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    42a253b View commit details
    Browse the repository at this point in the history

  2. cleanup

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    364a7fd View commit details
    Browse the repository at this point in the history

  3. skip_first_batches should be used on raw dls

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    8c4f15a View commit details
    Browse the repository at this point in the history

  4. multigpu sanity test

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    7ce63ef View commit details
    Browse the repository at this point in the history

  5. bugs

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    99f2afe View commit details
    Browse the repository at this point in the history

  6. does this work with iterable dsets?

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    3235e99 View commit details
    Browse the repository at this point in the history

  7. fix typo

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    dff2666 View commit details
    Browse the repository at this point in the history

  8. ignore these commits, i'm just syncing the origin so i can test on my… 

    … cloud workstation
    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    7092981 View commit details
    Browse the repository at this point in the history

  9. comment out failing tests, unsure if those are existing bugs or a rec… 

    …ent regression
    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    3b0702b View commit details
    Browse the repository at this point in the history

  10. torch 2.4.0?

    @byi8220
    byi8220 committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    b43c85d View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. pickling generator issues

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    40ec962 View commit details
    Browse the repository at this point in the history

  2. test_pickle_accelerator

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    1964011 View commit details
    Browse the repository at this point in the history

  3. test_pickle_accelerator should work now)

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    c6a68c3 View commit details
    Browse the repository at this point in the history

  4. base.__len__() -> len(base)

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    0f0a1f1 View commit details
    Browse the repository at this point in the history

  5. undo reduce

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    17713e7 View commit details
    Browse the repository at this point in the history

  6. undo super().__reduce__() again

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    073d7b3 View commit details
    Browse the repository at this point in the history

  7. pass args through superclass

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    ee681ed View commit details
    Browse the repository at this point in the history

  8. remove prints

    @byi8220
    byi8220 committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    47557b7 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Merge branch 'huggingface:main' into fix-pickle

    @byi8220
    byi8220 authored Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2056150 View commit details
    Browse the repository at this point in the history

  2. doc changes + make style && make quality

    @byi8220
    byi8220 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    9bba8f2 View commit details
    Browse the repository at this point in the history