modify schema, reindex script and dependencies to support dimensions and population types

[DavidSubiros]

What

Work done to prove that the new elasticsearch model works for dimensions and population types, corresponding to the spike defined by this trello card. The following trello cards have been implemented in order to prove the spike work:

• Reindex with models containing population_type and dimensions. Trello card

  • Modify reindex script to use and correctly map the new models for dimensions and population types
  • Improve reindex script flexibility, by adding flags to be able to debug issues without needing to reindex all the data from zebedee and dataset api
  • Add onSuccess and onFailure callbacks on the indexer, otherwise the async errors failed to be reported.
  • Add unit tests for new functionality in reindex script

• Update elasticsearch models to include population_type and dimensions. Trello card

  • Modify elasticsearch json schema (elasticsearch/search-index-=settings.json)

• Update search query templates to allow filtering by population type. Trello card

  • Modify query/templates/search/v710/contentFilters.tmpl to add the population type filter if present in the struct that is executed against the template.
  • Add query/templates/search/v710/populationTypeFilters.tmpl to allow filtering by population type name and/or label.

• Update search query templates to allow filtering by dimensions. Trello card

  • Modify query/templates/search/v710/contentFilters.tmpl to add the dimensions filter if present in the struct that is executed against the template.
  • Add query/templates/search/v710/dimensionsFilters.tmpl to allow filtering by dimension name, label and/or raw_label.

• Update search query templates to allow counting / aggregating by one category, while filtering by the other categories. Trello card

  • Add the search templates to allow counting content types while filtering by other categories:
    • query/templates/search/v710/countContentTypeQuery.tmpl
    • query/templates/search/v710/countContentTypeHeader.tmpl
    • query/templates/search/v710/countContentTypeFilters.tmpl
  • Add the search templates to allow counting dimensions while filtering by other categories:
    • query/templates/search/v710/countDimensionsQuery.tmpl
    • query/templates/search/v710/countDimensionsHeader.tmpl
    • query/templates/search/v710/countDimensionsFilters.tmpl
  • Add the search templates to allow counting population types while filtering by other categories:
    • query/templates/search/v710/countPopulationTypeQuery.tmpl
    • query/templates/search/v710/countPopulationTypeHeader.tmpl
    • query/templates/search/v710/countPopulationTypeFilters.tmpl
  • Add the search templates to allow counting topicx while filtering by other categories:
    • query/templates/search/v710/countTopicQuery.tmpl
    • query/templates/search/v710/countTopicHeader.tmpl
    • query/templates/search/v710/countTopicFilters.tmpl

• Other refactors with no functional changes:

  • Refactor the code for the search endpoint a bit to make it more consistent with the release endpoint, including a validator and the way the handler is registered to the api.
  • Refactor the code to create the search and count query structures from the http.Request query params (CreateRequests func), which is then passed to the query package to execute the templates
  • Add Dockerfile.local and reflex so that this repo can be used in dp-compose/v2 search stack

Depends on the following PRs:

• Elasticsearch client with callbacks
• Latest models and mapping logic in dp-search-data-extractor, required by the reindex script
• Latest models and transformer logic in dp-search-data-importer, required by the reindex script

How to review

• Make sure code changes make sense
• Make sure unit tests pass
• You may have a look to the spike document attached to the trello card an check it makes sense
• You may also run the reindex script against dp-dataset-api in sandbox (optionally, or ask me more information about this)

Who can review

Anyone