Data entry metadata API endpoint [DHIS2-12522]

[larshelge]

Context

The new data entry application needs to retrieve an extensive set of metadata on page load. Load time and network usage are of great importance as users are often on slow lines with prepaid data plans, and there could a huge amount of metadata. It is critical that the app only loads the necessary metadata and avoids loading metadata it does not need.

Endpoint

A new metadata endpoint dedicated for the data entry app should be created. The response should basically contain all the data sets for which the current user has data read sharing access to, and then metadata which are associated with the data sets. All metadata entities should be placed at the first level of the JSON response, and references between the objects should be done by ID (UID).

To summarize:

• Include only datasets with data sharing (read and read-write).
• All objects should be positioned at the root/metadata as lists.
• Include data set org unit associations in get metadata endpoint under data sets.
• The entities in the response should be fixed.
• We may or may not want to use field filtering to control the fields in the response.
• Option sets will have to be retrieved and cached separately due to the typical large payload.

Metadata:

• Data sets (with data set elements, sections, data set associations)
• Data elements
• Category combos
• Categories
• Category options
• Indicators

Format

The proposed format looks like this:

{
  "dataSets": [{
    "id": "lyLU2wR22tC",
    "name": "ART",
    "more": "fields here",
    "dataSetElements": [{
      "dataElement": "ofSrFzA676u"
    }],
    "sections": [{
      "id": "yzIGHd4A54j"
    }],
    "organisationUnits": [{
      "id": "rLeYbuFiIC0"
    }]
  }],
  "dataElements": [{
    "id": "ofSrFzA676u"
  }],
  "categoryCombos": [{
    "id": "SngDgP8uxDZ"
  }],
  "categories": [{
    "id": "X5hcppOmVi2"
  }],
  "categoryOptions": [{
    "id": "HAC6ak5LwpF"
  }],
  "indicators": [{
    "id": "U1QABkWGipy"
  }]
} 

Questions

• Do we want to hard code the fields in the response, or allow for field filtering?

[Birkbjo]

Thanks for putting this together! I argue that it might be cleaner for nested object-references to just be an array of the ids. Eg.

"dataSets": [{
    "id": "lyLU2wR22tC",
    "name": "ART",
    "more": "fields here",
    "dataSetElements": [{
      "dataElement": "ofSrFzA676u"
    }],
    "sections": ["yzIGHd4A54j"],
    "organisationUnits": ["rLeYbuFiIC0"]
}]

Note that embedded-objects would of course still need the objects. This way we save some data that is unnecessary since these references will just contain ID's anyway.

This is how we are currently getting the metadata, using the ~pluck-transformer. This do however result in embedded-objects and other object-references having different APIs, so the client would need to know what properties are embedded or not. But this is how we would store it in the client anyway.

[larshelge]

I agree. It would be more compact and reduce network usage.

[Mohammer5]

The data that's needed in the context selection is:

dataSets [
  id,
  displayName,
  formType,
  periodType,
  sections[
    id,
    displayName
  ],
  categoryCombo [
    isDefault,
    displayName,
    categories [
      id,
      displayName,
      categoryOptions [
        id,
        displayName,
        startDate,
        endDate
      ]
    ]
  ]
  organisationUnits [
    id
  ]
],
me [
  organisationUnits [
    id
  ]
]

[Mohammer5]

I'm not sure about the organisationUnit part as that probably has to come from the org unit tree itself