Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using ijson to avoid loading full json in memory. #356

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Using ijson to avoid loading full json in memory. #356

wants to merge 2 commits into from

Commits on Sep 20, 2023

  1. Using ijson to avoid loading full json in memory. 

    Using ijson to load item by item, so it is possible to load dataset
using dataset/convert_cc_sbu.py and dataset/convert_laion.py on machines
with low RAM.
    @SamimAB
    SamimAB committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a8eb69e View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2023

  1. Added "ijson" to the list of dependencies 

    Added ijson==3.2.3 in environment.yml
    @SamimAB
    SamimAB committed Sep 23, 2023
    Configuration menu
    Copy the full SHA
    64451a1 View commit details
    Browse the repository at this point in the history