Skip to content

mango

Jump to bottom
Laurent MICHEL edited this page Apr 21, 2021 · 17 revisions

Mango proposal

Mark CD comments

Models

  • Mango:
    • I've sumbitted Issues to the Mango repository with comments/suggestions on the model.
    • Parameter.description: is this a modeled concept?
      • there is some question (in my mind anyway) on whether something like this is a modeled concept (ie: an attribute on an object), since it can apply to any object, is completely arbitrary, often not included, and in no way contributes to the definition of the object itself.
      • the alternative is to not included it in the Model element(s), but let the Annotation address it in some way.
        "Any object may have a description. If supported by the implementation, the value is pulled from the VOTable DESCRIPTION sub-node of the associated PARAM|FIELD|TABLE" (we don't annotate GROUPs I think).

Annotations

  • Annotating content within a VOTable element (PARAM, FIELD, etc)
    This came up in the context of mango:Param.description where, if this is a modeled element, we may want to populate it from the VOTable content. However, the VOTable DESCRIPTION element cannot be referenced, it is internal to other VOTable elements (PARAM, FIELD, GROUP, TABLE).
    • The annotation syntax Laurent uses (Vodml-instance-vot) has elements (SC_*) which are supposed to handle this sort of problem.
      • Section 3.12 describes these "shortcuts"
        • example case (Quantity) illustrates annotating an ivoa:Quantity using this syntax.
          • uses independent ATTRIBUTEs for the Quantity.value and Quantity.unit
        • The SC_*QUANTITY tags consolidate this grouping into a single annotation tag.
        • Question: is there no way to assign a Quantity.unit from the 'unit' arg of PARAM|FIELD?
      • given the description in Section 3.12, I do not see how it satisfies this case of annotating content within a VOTable element atom.

LM answers

MANGO

  • Although all use-cases are supported by Mango, some weakness have been pointed out. They have been issued on the MANGO pages and will be subject to a major PR after the workshop.

Annotations

  • unit element shouldn't be mandatory in SC*QUANTITY. This part of the spec must be enhanced.

GL - VizieR mango experiments (and proposal)

a beta-prototype has been implemented to provide rich VOTable using Mango:

http://viz-beta.u-strasbg.fr/viz-bin/Mango? [vizier-asu-parameters]

e.g.:

The beta-prototype is available for vizier queries having a unique table in output.

The capabilities implemented are:

  • measure annotation available (currently) for positions, photometry, parallax
  • associated data is used to link the VizieR catalogue Provenance
  • grouping column mechanism to gather columns related to a same measure

The beta version is a piped process which takes a VizieR VOTable in input, extracts metadata from the VOTable (COOSYS and FIELDS definition) and returns in output a VOTable enriched with a VODML(lite) section header. The process is completed with a VizieR database connection to extract some metadata not available in the VOTable (v1.4) like the photometry systems.

Measure annotation

Important: filters are not (always) in the original article but assigned by CDS.

Provenance

Provenance is important for end-users to be aware of the data origin. There are indeed no standard way today to annotate authors, articles, year of publications in a VOTable. Furthermore, the VizieR process can assign metadata (like photometry) which is not part of original data. This information is described in a beta implementation of ProvDM.

In the VizieR Mango prototype, the provenance is added as an associated data with URLs (e.g.: http://viz-beta.u-strasbg.fr/viz-bin/Mango?-out.max=10&-source=J/ApJ/789/115 - see collection with dmrole="mango:Source.associatedDataDock")

Grouping columns

Mango enables to gather measures with errors. But the "associatedParameters" anables also to group columns related to a same measure. For instance a quality flag which qualifies the measure given in an other column.

e.g.: catalogue J/A+A/584/A5 (Evans, 2015)

The table “table2” contains 2 radial velocity columns: RV1 (primary velocity) and RV2 (secondary velocity). Each of theses columns are qualified with an error and a number of observation used to compute the radial velocity measure. So we can gather columns: (RV1, e_RV1, o_RV1) and (RV2, e_RV2, o_RV2)

Examples:

LM: Mango parser

This section shows the API prototype for the MANGO parser.

Annotated data sample can be found in dm-usecases/*/mango_based_proposal

  • Prerequisites
    • VOTable are annotated with MANGO using theModelInstanceInVot
    • These 2 standards are applied as they was at workshop time. Requested enhancements are not taken into account
    • The code presented here is a work in progress. It could be made more robust once all components will be more stable (syntax, models)
    • semantic fields are not properly set so far.

Python API.

Parsing output is always returned as Python dictionaries where model objects are referenced by their roles (key=role, value = object) In some case, human readable keys are generated either to improve the readability or to avoid key duplication.

Reading the MANGO annotation block

from client.parser.mango_browser import MangoBrowser
mango_browser = MangoBrowser(votable_path)
  • This action has no output.
  • It builds an internal representation of the bridge bind the model with the data.

Getting the list of the mapped parameters

Python

from utils.dict_utils import DictUtils

mango_parameters = mango_browser.get_parameters()
DictUtils.print_pretty_json(mango_parameters)

Return a dictionary of all parameters identified with a key made of the parameter rank + the parameter UCD

Output

{
  "#0 meta.id;meta.main": {
    "mango:MangoObject.identifier": {
      "id": "namesaada",
      "index": 24
    },
    "measure_type": "mango:MangoObject.identifier"
  },
  "#1 pos": {
    "coord_type": "mango:stcextend.LonLatPoint",
    "coords:SpaceFrame": {
      "@ID": "SpaceFrame_ICRS",
      "@dmtype": "coords:SpaceFrame",
      "coords:SpaceFrame.equinox": {
        "@dmtype": "coords:Epoch",
        "@value": "NoSet"
      },
      "coords:SpaceFrame.refPosition": {
        "@dmtype": "coords:StdRefLocation",
        "coords:StdRefLocation.position": {
          "@dmtype": "ivoa:string",
          "@value": "NoSet"
        }
      },
      "coords:SpaceFrame.spaceRefFrame": {
        "@dmtype": "ivoa:string",
        "@value": "ICRS"
      }
    },
    "coosys_type": "coords:SpaceFrame",
    "description": "Corrected position",
    "error_type": "meas:Error",
    "mango:stcextend.LonLatSkyPosition": {
      "field:latitude": {
        "id": "_dec_147",
        "index": 1
      },
      "field:longitude": {
        "id": "_ra_146",
        "index": 0
      }
    },
    "meas:Error": {
      "field:meas:Symmetrical.radius": {
        "id": "_poserr_148",
        "index": 2
      },
      "unit": "NotSet"
    },
    "measure_type": "mango:stcextend.LonLatSkyPosition",
    "semantic": "#position.corrected",
    "ucd": "pos"
  },
...
}

(truncated output)

Getting the space coordinate system of a specific mapped parameters

The searched parameter is identified by its parameter list key. This identifier can be used for different purpose

  • retrieving the complete description of the parameter
  • retrieving the parameter associated with a particular data cell

Python

space_coosys = mango_browser.get_param_coordsys("#1 pos")
DictUtils.print_pretty_json(space_coosys)

Output

{
  "@ID": "SpaceFrame_ICRS",
  "@dmtype": "coords:SpaceFrame",
  "coords:SpaceFrame.equinox": {
    "@dmtype": "coords:Epoch",
    "@value": "NoSet"
  },
  "coords:SpaceFrame.refPosition": {
    "@dmtype": "coords:StdRefLocation",
    "coords:StdRefLocation.position": {
      "@dmtype": "ivoa:string",
      "@value": "NoSet"
    }
  },
  "coords:SpaceFrame.spaceRefFrame": {
    "@dmtype": "ivoa:string",
    "@value": "ICRS"
  }
}

Getting the Astropy counterpart of that coordinate system

print(mango_browser.get_astropy_space_frame("#1 pos"))
<ICRS Frame>

Getting the taime coordinate system of a specific mapped parameters

Python

time_coosys = mango_browser.get_param_coordsys("#12 time;obs.start")
DictUtils.print_pretty_json(time_coosys)

Output

{
  "@ID": "TimeFrame_BARYCENTER",
  "@dmtype": "coords:TimeFrame",
  "coords:TimeFrame.refPosition": {
    "@dmtype": "coords:StdRefLocation",
    "coords:StdRefLocation.position": {
      "@dmtype": "ivoa:string",
      "@value": "BARYCENTER"
    }
  },
  "coords:TimeFrame.timescale": {
    "@dmtype": "ivoa:string",
    "@value": "TCB"
  }
}

Getting the Astropy counterpart of that coordinate system

The time frame being not an Astropy object, it is returned as (scale, location, format) tuple.

print(mango_browser.get_astropy_time_frame("#12 time;obs.start"))
('tcb', None, 'mjd')

Getting the position data

The parameters to be fetched are selected by either measure_type (model class name) or by ucd. If no selector is set, all mapped data are returned.

Python

mango_data = mango_browser.get_data(measure_type="mango:stcextend.LonLatSkyPosition")
DictUtils.print_pretty_json(mango_data)

Output

Returns an object containing all requested data with their model references. This method can be used to read all data in once or to store in a easy way the column-to-model binding for further uses (e.g. streaming readout) Only the first row is read in order limit the output size.

  • data: one array per row. The value order matches the model mapping not the column number
  • head: Reference to the model element matching the corresponding columns. This field contains the parameter key in [] in order to easily retrieve the complete description
  • selected_index: Gives the actual column number of each listed quantity.
{
  "data": [
    [
      340.91055060369,
      -17.071667101891,
      1.50765
    ]
  ],
  "head": [
    "field:longitude [#1 pos]",
    "field:latitude [#1 pos]",
    "error: field:meas:Symmetrical.radius [#1 pos]"
  ],
  "selected_index": [
    0,
    1,
    2
  ]
}

Please put your name or initials in the front of your constributions (see the guide)

Clone this wiki locally