Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add filter strategy for optimized entity name/property queries #6267

Merged
merged 26 commits into from
Jul 29, 2024

Conversation

seadowg
Copy link
Member

@seadowg seadowg commented Jul 15, 2024

Closes #5623

Blocked by #6261
Blocked by getodk/javarosa#780
Blocked by getodk/javarosa#782

The "closes" here is slightly cheating as we'll also need to a better implementation of EntitiesRepository (using SQLite most likely) to get the real world load time gains. Additionally, the total memory footprint won't have dropped - the current EntitiesRepository implementation requires a whole list in memory for both getById and getByProperty. That all said, we're assuming that our new implementation (which we'll build as part #6012) will give us faster lookups and memory use that's only increased by the number of results (as opposed to needing the whole list in memory ever). @lognaturel is putting together a project for testing all this, and we've decided that this implementation is a good enough structural change to serve as a foundation we can tweak to get the performance we want.

Why is this the best possible solution? Were any other approaches considered?

The basic idea here is to query the EntitiesRepository for name= and <property>= predicates instead of letting JavaRosa use its own built in FilterStrategy chain. Combined with "partial parsing" (provided by LocalEntitiesInstanceProvider) and the new DataInstance#replacePartialElement this lets us keep the majority of the entity list out of memory until its all needed (like when we show an unfiltered select).

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?

This risk here will be to forms that use filtered selects and calculations based on secondary instances. Entity forms are the most likely to have problems, but there is some risk that there are some unintended consequences for non-entity forms as well.

There's no need to test the actual performance increase/decrease here. As I said in the introduction, that will be hard to properly verify until other changes are made.

Before submitting this PR, please make sure you have:

  • added or modified tests for any new or changed behavior
  • run ./gradlew connectedAndroidTest (or ./gradlew testLab) and confirmed all checks still pass
  • added a comment above any new strings describing it for translators
  • added any new strings with date formatting to DateFormatsTest
  • verified that any code or assets from external sources are properly credited in comments and/or in the about file.
  • verified that any new UI elements use theme colors. UI Components Style guidelines

@seadowg seadowg force-pushed the entity-filter-strategy branch 4 times, most recently from e1e5fb1 to 4ef9c1c Compare July 17, 2024 18:56
@seadowg seadowg changed the title Add filter strategy for optimized entity name queries Add filter strategy for optimized entity name/property queries Jul 17, 2024
@seadowg seadowg force-pushed the entity-filter-strategy branch 2 times, most recently from 37b4d9b to 5e6fff8 Compare July 18, 2024 12:26
@seadowg seadowg removed the blocked label Jul 19, 2024
@seadowg seadowg marked this pull request as ready for review July 19, 2024 12:13
import org.odk.collect.entities.storage.EntitiesRepository
import org.odk.collect.entities.storage.Entity

class LocalEntitiesInstanceAdapter(private val entitiesRepository: EntitiesRepository) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we'll eventually pull out an interface (InstanceDataAdapter for example) from this to use for normal secondary instances as well. The entity InstanceProvider and FilterStrategy are both actually generic enough to support that with minimal changes and could be pulled out into another package, but that felt too far for the moment.

@seadowg
Copy link
Member Author

seadowg commented Jul 29, 2024

@grzesiek2010 OK that's the missed comment fixed!

@seadowg seadowg marked this pull request as draft July 29, 2024 15:15
@seadowg
Copy link
Member Author

seadowg commented Jul 29, 2024

Marking this as a draft while I fix problems with the local entities setting that are complicated by JavaRosa's static state.

@seadowg
Copy link
Member Author

seadowg commented Jul 29, 2024

@grzesiek2010 ok that's fixed!

@seadowg seadowg marked this pull request as ready for review July 29, 2024 15:48
@grzesiek2010 grzesiek2010 merged commit ef79db0 into getodk:master Jul 29, 2024
6 checks passed
@seadowg seadowg deleted the entity-filter-strategy branch July 30, 2024 07:23
@dbemke
Copy link

dbemke commented Aug 6, 2024

Tested with Success!

Verified on a device with Android 10

Verified Cases:

  • exploratory tests in forms with entities
  • forms with filtered selects and entities
  • regression checks in forms with selects and forms with secondary instances

@WKobus
Copy link

WKobus commented Aug 6, 2024

Tested with Success!

Verified on a device with Android 14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize load times and memory footprints of form entry with entities
4 participants