New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

finish code to read file in chunks. #1

Open

sk-g opened this issue Apr 5, 2018 · 0 comments

Assignees

Labels

Owner

sk-g commented Apr 5, 2018

storageResearch/Data/Microsoft/EDA.py

Line 151 in 30ff498

# creating an iterator to read in chunks, not functional atm

Although the traces are big (>10gb in some cases) there are only few DiskXXXX operations (~10k out of ~10M). It is pointless to read the entire file and then use groupby methods.

to do:
read in chunks, create dataframes based on first column of each row. (.starswith..?)
fix iterator block to handle rows<chunksize for the last chunk.

sk-g self-assigned this

sk-g added the help wanted label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment