Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory consumption in bulk loader #3724

Merged
merged 13 commits into from
Jul 31, 2019
Merged

Reduce memory consumption in bulk loader #3724

merged 13 commits into from
Jul 31, 2019

Commits on Jul 23, 2019

  1. Avoid encoding and decoding before sorting

    Lucas Wang committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    f94f067 View commit details
    Browse the repository at this point in the history

  2. Clearing the entries and encodedSize for the next batch

    Lucas Wang committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    7191ff7 View commit details
    Browse the repository at this point in the history

  3. Maintain the buffer correctly

    Lucas Wang committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    09b7f5b View commit details
    Browse the repository at this point in the history

  4. make the buf to have a big begin size to allow direct access

    Lucas Wang committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    2f9953a View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2019

  1. Adding wait in the test script to wait for alpha to be truly ready

    Lucas Wang committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    561ae0d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into gitlw/improve_bulk_loader

    Lucas Wang committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    ddb5fa6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into gitlw/improve_bulk_loader

    Lucas Wang committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    d2d499d View commit details
    Browse the repository at this point in the history

  4. Fix the gosimple comment

    Lucas Wang committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    572b55b View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2019

  1. Use gzip format for mapper output

    Lucas Wang committed Jul 30, 2019
    Configuration menu
    Copy the full SHA
    946189b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into gitlw/improve_bulk_loader

    Lucas Wang committed Jul 30, 2019
    Configuration menu
    Copy the full SHA
    bdb5672 View commit details
    Browse the repository at this point in the history

  3. Fixed the reading of map entries from files, 

    Properly flushing the intermediate files
    Lucas Wang committed Jul 30, 2019
    Configuration menu
    Copy the full SHA
    0ae5cd4 View commit details
    Browse the repository at this point in the history

  4. Improving comments

    Lucas Wang committed Jul 30, 2019
    Configuration menu
    Copy the full SHA
    fced99d View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2019

  1. Addressed Manish's comments

    Lucas Wang committed Jul 31, 2019
    Configuration menu
    Copy the full SHA
    19a029c View commit details
    Browse the repository at this point in the history