Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large graph causes sql query issues #18

Open
sheridana opened this issue Dec 11, 2024 · 1 comment
Open

Large graph causes sql query issues #18

sheridana opened this issue Dec 11, 2024 · 1 comment

Comments

@sheridana
Copy link
Contributor

When attempting to process a large graph I run into sql query issues:

psycopg2.errors.InternalError_: invalid memory alloc request size 1073741824

which is triggered by this line

should this be updated to process in batches? probably here

@sheridana
Copy link
Contributor Author

@pattonw as discussed, adding:

if roi is None:
     node_condition = "true"

to read_edges() is a quick fix. The problem then remains for the case when we need to process a large roi if the query then doesn't fit in memory. Adding a join is likely necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant