Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage of pole_of_inaccessibility.py #250

Open
justinanderson opened this issue Apr 4, 2022 · 0 comments
Open

Reduce memory usage of pole_of_inaccessibility.py #250

justinanderson opened this issue Apr 4, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@justinanderson
Copy link
Member

pole_of_inaccessibility.py, run by the postprocessor service on new shapefile tables, peaks at around 500 MB of active memory when run against Australia's SA2 shapefile. Depending on someone's Docker setup, the postprocessor container might be killed early for exceeding system limits even for that small shapefile. The Atlas should be able to handle much larger datasets than that.

The script currently uses geopandas to read from and write to PostGIS, and there are issues with geopandas which prevent using Pandas' chunksize to work with tables chunk by chunk. One alternative is to rewrite the script to use psycopg2 directly. One could then limit the read to only the primary key and geometry columns, and the write to only setting the pole of inaccessibility for each row, rather than reading and writing the entire table.

@justinanderson justinanderson added the enhancement New feature or request label Apr 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant