Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefetch descriptor functions at graph-handling time, not resource-handling time #10453

Closed
jacobtylerwalls opened this issue Dec 18, 2023 · 1 comment · Fixed by #10743
Closed
Assignees

Comments

@jacobtylerwalls
Copy link
Member

#10439 applied in two more places the trick from #10312 to skip queries for descriptor functions on the assumption that resources are generally grouped by graph:

# Reuse the queryset for FunctionXGraph rows if the graph is the same.
if last_resource and (last_resource.graph_id == this_resource.graph_id):
this_resource.descriptor_function = last_resource.descriptor_function

But we could get more savings by removing this "trick" and just prefetching the descriptor functions before we start iterating the resources.

That would go something like this:

  • When querying for resources, use a Prefetch() object to do the filtering for the graph's descriptor function and save it to descriptor_function on each Resource.
  • Then remove the "trick" linked above in both files it appears

Then, with the hood popped on this (looking at callers of save_descriptors()), there are a few more wins to be had:

  • index_resources_by_type() refetches graph objects because the earlier call in index_resources() only gets a values array
  • Tile.delete() refetches the resource instance
  • Tile.delete() saves the resource descriptor once per node
  • import_business_data_without_mapping() fetches graphs one-by-one
@jacobtylerwalls
Copy link
Member Author

The last few bullet-points tacked on here don't provide enough value to work on.

index_resources_by_type() refetches graph objects because the earlier call in index_resources() only gets a values array
Tile.delete() refetches the resource instance
Tile.delete() saves the resource descriptor once per node
import_business_data_without_mapping() fetches graphs one-by-one

[1] -- not very many queries
[2] -- refetching is necessary, because we need the Proxy model
[3] -- already fixed in #10481
[4] -- too hard to refactor

@jacobtylerwalls jacobtylerwalls linked a pull request Apr 8, 2024 that will close this issue
6 tasks
@jacobtylerwalls jacobtylerwalls moved this from 🏗 In Progress to 👀 In Review in pipeline Apr 8, 2024
@github-project-automation github-project-automation bot moved this from 👀 In Review to ✅ Done in pipeline Apr 17, 2024
jacobtylerwalls added a commit that referenced this issue Apr 18, 2024
whatisgalen added a commit that referenced this issue Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant