Api optimization

[seankim658]

Switched from the indirect method using an intermediary results collection to passing the burden of computation to the database by using the MongoDB aggregation framework. This helped significantly with large disk retrievals.

Before a text search with a generic term such as "cancer" required significant batching and cold start disk retrievals could take 5-7+ seconds for retrieving from disk. Implemented a naive LRU caching solution that took repeat batch retrievals from averaging ~2 seconds to ~0.02 seconds. However, this still did not fix the root cause of a cache cold start on the initial text queries. Putting the burden of processing onto the database and removing the two step list ID process got large retrievals on cold starts down significantly, from averaging ~35 seconds before to ~5 seconds.

Further optimizing, the wildcard text index was not ideal as the data model has a significant amount of nesting and fields. By manually creating an internal field that concatenated the string values from each string field into one field called all_text, this allowed for text indexing the all_text field directly and dropping the wildcard text index. This dropped vague queries down to ~2.5 seconds.