Batch searching when list might be large or when count is not defined

[LZRS]

IMPORTANT: Where possible all PRs must be linked to a Github issue

Fixes #3440

Engineer Checklist

• I have written Unit tests for any new feature(s) and edge cases for bug fixes
• I have added any strings visible on UI components to the strings.xml file
• I have updated the CHANGELOG.md file for any notable changes to the codebase
• I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the project's style guide
• I have built and run the FHIRCore app to verify my change fixes the issue and/or does not break the app
• I have checked that this PR does NOT introduce breaking changes that require an update to Content and/or Configs? If it does add a sample here or a link to exactly what changes need to be made to the content.

Code Reviewer Checklist

• I have verified Unit tests have been written for any new feature(s) and edge cases
• I have verified any strings visible on UI components are in the strings.xml file
• I have verifed the CHANGELOG.md file has any notable changes to the codebase
• I have verified the solution has been implemented in a configurable and generic way for reuseable components
• I have built and run the FHIRCore app to verify the change fixes the issue and/or does not break the app

[codecov]

Codecov Report

Attention: Patch coverage is 66.66667% with 13 lines in your changes missing coverage. Please review.

Project coverage is 27.2%. Comparing base (115c4f4) to head (6ee690d).
Report is 6 commits behind head on main.

Files	Patch %	Lines
...rcore/engine/util/extension/FhirEngineExtension.kt	66.6%	4 Missing and 3 partials ⚠️
...er/fhircore/engine/data/local/DefaultRepository.kt	87.5%	1 Missing ⚠️
...ster/fhircore/engine/p2p/dao/BaseP2PTransferDao.kt	0.0%	1 Missing ⚠️
...fhircore/engine/task/FhirCompleteCarePlanWorker.kt	0.0%	1 Missing ⚠️
...hircore/engine/util/extension/MeasureExtensions.kt	0.0%	1 Missing ⚠️
...e/quest/ui/questionnaire/QuestionnaireViewModel.kt	0.0%	1 Missing ⚠️
...st/ui/report/measure/worker/MeasureReportWorker.kt	0.0%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##              main   #3456     +/-   ##
=========================================
- Coverage     27.2%   27.2%   -0.1%     
- Complexity     742     746      +4     
=========================================
  Files          276     276             
  Lines        13811   13903     +92     
  Branches      2504    2524     +20     
=========================================
+ Hits          3764    3788     +24     
- Misses        9540    9593     +53     
- Partials       507     522     +15     

Flag	Coverage Δ
engine	64.3% <70.2%> (-0.4%)	⬇️
geowidget	11.5% <ø> (+<0.1%)	⬆️
quest	5.5% <0.0%> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ster/fhircore/engine/task/FhirCarePlanGenerator.kt	78.1% <100.0%> (ø)
...tregister/fhircore/engine/task/FhirResourceUtil.kt	85.8% <100.0%> (ø)
...hircore/engine/util/extension/ResourceExtension.kt	67.5% <100.0%> (+0.3%)	⬆️
...er/fhircore/engine/data/local/DefaultRepository.kt	65.0% <87.5%> (ø)
...ster/fhircore/engine/p2p/dao/BaseP2PTransferDao.kt	72.4% <0.0%> (ø)
...fhircore/engine/task/FhirCompleteCarePlanWorker.kt	73.9% <0.0%> (ø)
...hircore/engine/util/extension/MeasureExtensions.kt	50.0% <0.0%> (ø)
...e/quest/ui/questionnaire/QuestionnaireViewModel.kt	2.9% <0.0%> (+0.1%)	⬆️
...st/ui/report/measure/worker/MeasureReportWorker.kt	0.0% <0.0%> (ø)
...rcore/engine/util/extension/FhirEngineExtension.kt	58.8% <66.6%> (+1.2%)	⬆️

... and 1 file with indirect coverage changes

[ellykits]

Thanks for this @LZRS.

I have a few questions and comments on this implementation.

The decision to load this data in batches sounds reasonable, however, based on this implementation I believe the gain would be negligible or there would be no gain at all. Technically, the data will still be cached in memory inside the loop. This differs from the approach where we would load and process the data in batches. Data processing should happen after fetching each batch to improve the app responsiveness, otherwise, there are some drawbacks with batching including:

1. Each batch fetch requires a new database query, introducing overhead. The overhead of executing smaller queries may outweigh the benefit of lower memory.

2. Using OFFSET requires the database to skip over records
   before returning the batch. A larger offset would require scanning and discarding many rows thus if indexing is not done properly, this will impact fetching data in batches.

@LZRS Would it be possible to measure the performance impact introduced by this implementation for comparison?

The paging3 library loads data in batches and allows for data processing with each fetch.

cc @ndegwamartin @pld

[LZRS]

Yeah, generally, I think the performance gains would probably be negligible (in one of our cases, a difference of 20ms) especially when assuming only our query of interest would be running a single time. I think the significance of these changes would come about when we have multiple queries from different coroutines/threads.

Assuming for example we have a background long running operation accessing the db, and in the UI someone goes to a page that to render also requires a db fetch. Instead of waiting for the long running background operation to finish, the db fetch for the UI page can get queued in between the batches for the long running background. Looking to also add integration tests on the same

The changes here would ensure that no single query(in batch) would take more than a couple of milliseconds, a long running query would still take long to be loaded (as it's split and cached in memory) but it would allow space for other queries to also get run.

I also agree that the best case would be data processing after fetching each batch although that would be quite tasking to implement, I feel.

In terms of MalawiCore, it improved the app's performance from here to here.
We'll working on similar tests for FhirCore.

Use of paging3 to load would be the recommended approach but it would also need approval from the Google's sdk team that would also take time, I think

Note:

Room will only perform at most one transaction at a time, additional transactions are queued and executed on a first come, first serve order.