generated from ivoa-std/doc-template
-
Notifications
You must be signed in to change notification settings - Fork 7
/
Copy pathEPNTAP.tex
2713 lines (2155 loc) · 120 KB
/
EPNTAP.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\documentclass[11pt,a4paper]{ivoa}
\input tthdefs
\usepackage{wasysym}
\iftth
\def\ucd#1{\texttt{#1}}
\else
\makeatletter
\def\ucd{\st@rtucd\re@lucd}
\def\re@lucd#1{\sl#1\@nducd}
\begingroup
% let LaTeX break UCD componds and at dots and semicolons in UCDs
\gdef\bre@kabledot{.\hskip0pt}
\gdef\bre@kablesemicolon{;\hskip0pt}
\catcode`\.=\active\catcode`\;=\active
\gdef\st@rtucd{\begingroup
\catcode`\.=\active\let.=\bre@kabledot
\catcode`\;=\active\let;=\bre@kablesemicolon}
\gdef\@nducd{\endgroup}
\endgroup
\makeatother
\fi
\DeclareUnicodeCharacter{00B0}{$^\circ$}
\DeclareUnicodeCharacter{00B5}{$\mu$}
\DeclareUnicodeCharacter{00B1}{$\pm$}
% epn-tap has some wide displays; give it a bit more room (at
% the expense of readability).
\usepackage[lmargin=2cm,rmargin=3cm]{geometry}
\usepackage{pdflscape}
\usepackage{longtable}
\usepackage{rotating}
\usepackage{tabulary}
\usepackage{todonotes}
\lstloadlanguages{SQL,XML}
\lstset{flexiblecolumns=true,numberstyle=\small,showstringspaces=False,
identifierstyle=\texttt}
\title{EPN-TAP: Publishing Solar System Data to the Virtual Observatory}
% see ivoatexDoc for what group names to use here
\ivoagroup{Data Access Layer}
\author{St\'ephane Erard}
\author{Baptiste Cecconi}
\author{Pierre Le Sidaner}
\author{Markus Demleitner}
\author{Mark Taylor}
\editor{St\'ephane Erard}
\editor{Baptiste Cecconi}
% \previousversion[????URL????]{????Funny Label????}
\previousversion{This is the first public release}
\begin{document}
\begin{abstract}
This document defines the EPN-TAP framework, which is using TAP with
the EPNCore metadata dictionary. The EPNCore metadata dictionary defines
the core components that are necessary to perform data discovery in the
Solar System related science fields. It includes parameters to describe
data products coverage (temporal, spectral, spatial, photometric), origin
(instrument, facility), content (target, physical parameters), access,
references, etc. Its implementation with TAP (Table Access Protocol) is
presented, including service registration guidelines. Topical extension
metadata dictionaries are also presented.
\end{abstract}
\section*{Acknowledgments}
EPNCore has been developed by the VESPA (Virtual European Solar
and Planetary Access) team. VESPA has been designed in the frame of
Europlanet 2020 Research Infrastructure project and finalized
during the Europlanet 2024 Research Infrastructure project,
based on an assessment phase in the Europlanet Research Infrastructure
programme, JRA4 work package (IDIS activity).
The Europlanet-RI project was funded by the European Commission under the
7th Framework Program, grant 228319 ``Capacities Specific Programme''.
The Europlanet 2020 Research Infrastructure project has
received funding from the European Union's Horizon 2020 research and
innovation programme under grant agreement No 654208. The Europlanet-2024
Research Infrastructure project has received funding from
the European Union's Horizon 2020 research and innovation programme under
grant agreement No 871149. Additional funding was provided in France by
the CNRS / Action Sp\'ecifique Observatoire Virtuel and Programme National
de Plan\'etologie / INSU, as well as CNES, through the participation to
IPDA activities.
\section*{Conformance-related definitions}
The words ``MUST'', ``SHALL'', ``SHOULD'', ``MAY'', ``RECOMMENDED'', and
``OPTIONAL'' (in upper or lower case) used in this document are to be
interpreted as described in IETF standard RFC2119 \citep{std:RFC2119}.
The \emph{Virtual Observatory (VO)} is a
general term for a collection of federated resources that can be used
to conduct astronomical research, education, and outreach.
The \href{http://www.ivoa.net}{International
Virtual Observatory Alliance (IVOA)} is a global
collaboration of separately funded projects to develop standards and
infrastructure that enable VO applications.
\section*{List of Acronyms}
\begin{itemize}
\item{ADQL} Astronomical Data Query Language
\item{APIS} Auroral Planetary Imaging and Spectroscopy
\item{CNRS} Centre National de la Recherche Scientifique
\item{CNES} Centre National d'Etudes Spatiales
\item{EPN} Europlanet (series of EC-funded programmes)
\item{IDIS} Integrated and Distributed Information System
\item{INSU} Institut National des Sciences de l'Univers
\item{IPDA} International Planetary Data Alliance
\item{PDS} Planetary Data System (NASA data archive for planetary missions)
\item{PDS3/4} Successive data formatting standards for space agencies
\item{PSA} Planetary Science Archive (ESA data archive for planetary missions)
\item{SPASE} Space Physics Archive Search and Extract
\item{TAP} Table Access Protocol
\item{UCD} Unified Content Descriptors
\item{VESPA} Virtual European Solar and Planetary Access
\end{itemize}
\section{Introduction}
This document defines the EPN-TAP framework. EPN-TAP is a protocol
used to describe and access data related to the study of the Solar
System in general, including observational, modeled, and experimental
data. It consists of 1) the usual TAP mechanism; 2) the EPNCore metadata
dictionary; 3) a set of rules defining table structure and column
usage. EPN-TAP relies on the TAP (Table Access Protocol) standard with
no modification \citep{2019ivoa.spec.0927D}, so that only items 2 and 3
are described here.
All EPN-TAP services are accessible through standard TAP clients.
Its implementation with TAP is presented, including parameter usage
(sections \ref{sect:dictionary} and \ref{sect:epntable})
and service registration
guidelines (section~\ref{sect:registry}).
The first version of EPN-TAP was developed as an assessment study
during the Europlanet-RI programme (2011), and was limited to test purposes. It
was redesigned to EPN-TAP 2.0 before the start of Europlanet 2020 RI and
completed during the following programme Europlanet 2024 RI based on a large
set of data services and use cases. This later version is described here.
\subsection{EPN-TAP design}
The EPNCore metadata dictionary defines the core components that are
necessary to perform data discovery in science fields related to the
Solar System and related fields. The EPNCore metadata dictionary includes
parameters to describe data products coverage (temporal, spectral, spatial,
illumination conditions), origin (instrument, facility), content (target,
physical parameters), access, references, etc. The notion of \textbf{parameter} is
derived from the ``keywords'' of the PDS
standard used to archive planetary space missions. They are intended
either as search parameters or descriptive ones, although in TAP they
can all be searched by value. In this sense, EPNCore bears similarities
with the ObsCore standard \citep{2017ivoa.spec.0509L} from which it borrows several concepts.
EPN-TAP also uses the notion of \textbf{granule} (inherited from the SPASE standard)
to refer to the data service granularity, which is smallest data unit
provided by a service and described individually in the associated
table. A ``granule'' can correspond to a data file, a set of scalar
values, a call to a web service, a query to a data service in a different
protocol, etc.
Grouping of data into granules is at the discretion of the data
provider. This may be equivalent to ``dataset'' or ``data product''
in other contexts, but these words are avoided here because they have
a different meaning in many Planetary Science archives.
An EPN-TAP service therefore consists of a single table describing a
list of granules with the metadata dictionary: a granule corresponds
to a row of the table, while a parameter corresponds to a column.
Only one data product can be described and linked per row of the table.
Observations, simulations, or experimental data are supported.
Several independent services can reside on a single server.
The global philosophy of EPN-TAP is to describe all tables with a common
set of mandatory parameters which can be used to query all EPN-TAP
services together, which is the scope of the VESPA portal. A unique query
can then be sent to and answered by all data services without generating
errors. This also requires that all parameter values are provided in
a standard form, in particular with standard units. In other words,
mandatory parameters have to be present in an EPN-TAP service and must
provide values in standard form. Although most of them can be left
empty when non-relevant or unknown, care should be taken to fill as many
parameters as possible to provide a better service to the user.
In addition to these mandatory parameters, the EPN-TAP dictionary includes
optional parameters which are grouped in two levels: 1) common, general
purpose parameters provide complementary information; 2) parameters from
thematic extensions have been defined from a set of related data services
and must be used consistently in data services in these fields. The first
group includes file access information and additional description of data
and target. The second group defines additional axes and provides extra
parameters to describe similar observations consistently. The optional
parameters can be used independently of each other
(except the min/max pairs which need to appear together).
Finally, EPN-TAP allows data providers to use entirely new parameters in a
data service to provide specific information, when no existing parameter
applies. This may occur in particular when the data consist in a few
scalar values included in the table itself as individual parameters. Such
parameters are best defined in consistency with existing ones, and may
eventually be included in thematic extensions.
Parameter names are given in lower case, they must not include spaces
or special characters; underscore is used as separator.
Section 2 provides a description of parameters and their usage. All the
currently defined parameters are listed in Section \ref{sect:epntable}
(table) with their
type, unit, and UCD.
EPNCore usage is not limited to data distribution and access through
TAP. It also provides a standard way to describe Planetary Science data,
e.g., to handle local databases and private projects while benefiting
from interfaces with VO tools, in particular with TOPCAT, Aladin, and
CASSIS, which were closely associated to this development.
\subsection{EPN-TAP rules applying to table structure and parameter usage}
\begin{itemize}
\item \textbf{Rules related to data services and table structure include:}
\begin{itemize}
\item A service consists of one database schema containing only one table.
The table must be called <schema>.epn\_core.
\item Only one data product is linked per table row, and parameters
in that row describe that product. In case the same data product is
distributed with several formats, there can be either one row per format,
or a single row providing access to the preferred format,
while other formats are distributed through an associated datalink.
\item Related files providing documentation of the data product can
however be associated through thumbnail\_{url} (quick-look image),
datalink\_url (various documents and links), and external\_link (web
page providing more discussion of this granule).
\item Data services are declared in the IVOA registry of services
according to TAP guidelines (section~\ref{sect:registry}).
\item The service metadata must include the usual elements to allow
discovery in the registry. Subject elements should be taken from the IVOA
flavored Unified Astronomy Thesaurus (UAT) when available. At least one of
these top-level keywords of the UAT must be provided: exoplanet-astronomy,
solar-physics, solar-system-astronomy, planetary-science.
\item Data service output VOTables should include TIMESYS and COOSYS
elements when constant. This is not always true, see time\_scale and
space\_origin parameters for details.
\item Data service output must pass validation;
an EPN-TAP service validator is included in
STILTS/taplint\footnote{\url{http://www.starlink.ac.uk/stilts/sun256/taplint.html}}.
\end{itemize}
\item \textbf{Rules related to parameters include:}
\begin{itemize}
\item Parameters defining a range along an axis most often appear as a
pair with *\_min and *\_max variations. Both values must be provided
to support ranges correctly. If only one value is available, it must
appear in both parameters.
\item Parameters accept values in lower case, except when a global
standard or vocabulary applies. This is the case in particular with
Solar System object names, which follow the IAU naming convention.
\item Empty parameters can be void or can contain the NULL value.
\item Most floating-point parameters only require single precision,
with the exception of time\_min and time\_max which must be stored and
returned in double precision to preserve the required accuracy.
\item Some string parameters can be multivalued. Lists of values must be provided
as hash-separated lists (values separated by internal \# character).
\item Although each row contains at most one linked data product, special
conventions exist:
Thumbnails may be provided on the same granule row,
as far as they are intended for quick-look in the VESPA portal or
similar user interfaces; inclusion of thumbnails is recommended.
In contrast, large previews should be handled as separated data products.
Associated documents, or in some cases alternative formats, can be
attached via the datalink\_url parameter. Related web services or SODA
data services are also linked via this parameter.
\item Some parameters only accept values from a predefined list. Such
lists are provided or discussed in section \ref{sect:dictionary} and are maintained
on an external web page with a Permanent ID: \url{https://hdl.handle.net/21.15110/epn_tap_extensions}.
Whenever possible, this refers to IVOA vocabularies maintained independently.
\item Internal spaces are supported in string parameters values, but
not leading or trailing spaces.
\item Special characters, quotes, and hash are not allowed in string parameter values,
and may be changed to \_ (underscore, which is the single-character
wildcard in ADQL LIKE conditions).
\item Although additional, service-specific parameters can be used,
care must be taken to avoid duplications or variations on existing parameters,
including those from thematic extensions.
\item Such free parameters, when very specific to a service,
may use a prefix related to this service to prevent conflicts
(e.g., myservice\_myparameter).
Associated UCDs must be extracted from the
current version of the standard (UCD1+).
\item When free parameters contain a numerical value, associated errors
should use the syntax: parameter\_error, or parameter\_error\_min
and parameter\_error\_max when asymmetrical (all non-negative). The
associated UCDs must start with ``stat.error;'', ``stat.error;stat.min;'',
or ``stat.error;stat.max;'', respectively.
\end{itemize}
Upper and lower limits can be introduced with the same 3 parameters
used to provide asymmetrical errors, say: val, val\_error\_min and
val\_error\_max. The following scheme is suggested:
\begin{itemize}
\item An upper limit can be provided with val = 0, val\_error\_min = Inf,
val\_error\_max = upper\_limit.
\item A lower limit can be provided with val = lower\_limit,
val\_error\_min = 0, val\_error\_max = Inf.
\end{itemize}
This scheme minimizes interpretation errors, simplifies graphical
representations, and preserves transforms such as value $\to$ -value.
\end{itemize}
\subsection{Role within the VO Architecture}
\begin{figure}[thb]
\centering
\includegraphics[width=0.9\textwidth]{role_diagram.pdf}
\caption{Architecture diagram for this document}
\label{fig:archdiag}
\end{figure}
Figure~\ref{fig:archdiag} shows how EPN-TAP fits into Virtual
Observatory architecture. EPN-TAP's main connections to other VO
standards are:
\begin{bigdescription}
\item[ADQL \citep{2008ivoa.spec.1030O}] ADQL is the preferred language to
query EPN-TAP in. The common language is a prerequisite for global
discovery within all EPN-TAP services.
\item[TAP \citep{2019ivoa.spec.0927D}] TAP is used to transport queries,
table metadata, and the results of queries. The re-use of TAP gives
EPN-TAP a wide range of client and server implementations.
\item[VODataService \citep{2021ivoa.spec.1102D}] VODataService tableSets
are used to discover EPN-TAP services in VO Registries.
\item[UCDs \citep{2021ivoa.spec.0616C}] UCDs associated to EPNCore
parameters provide context when manipulating EPN-TAP tables and result
VOTables. They also describe the physical quantities distributed in
services via the measurement\_type parameter, making searches by physical
quantity possible.
\item[DALI \citep{2017ivoa.spec.0517D}] DALI lays out several basic
conventions, including the serialization of data types like timestamps
and geometries.
\end{bigdescription}
\section{The EPNCore metadata parameter dictionary}
\label{sect:dictionary}
% GENERATED: python3 parse_source.py columndescription
% To ignore the following section,
% add 'Mandatory parameters' to IGNORED_SECTIONS
In EPN-TAP services, EPNCore parameters appear as table columns and describe the granules.
Mandatory parameters must be present in all EPN-TAP services, while optional
parameters can be included as needed, with specified units and UCD.
\subsection{Mandatory parameters}
\subsubsection{Granule references}
These 3 parameters must always be provided and contain a value. If granule grouping is not
relevant, use a single value for granule\_gid (and obs\_id can be
identical to granule\_uid).
\paragraph{granule\_uid}
Identifier for this row. This parameter is a primary key in epn\_core
tables, i.e., no two different rows may share the same granule\_uid.
There can be only one data file associated to a granule (plus possibly a
thumbnail for quick-look purpose in a search interface).\\
Basic ascii characters are allowed, including internal spaces, except for the \# character.
\paragraph{obs\_id}
Associates granules derived from the same data (e.g., various
representations or processing levels). May be the ID of the original
observation.
\paragraph{granule\_gid}
Associates granules of same type (e.g., same map projection, or geometry
data products) --- think of it as a simple and convenient way to group or
differentiate types of data.
When several files relate to the same data, this parameter helps
distinguishing them: this will allow the user to select the type of data
of interest. For instance, a service may provide links to calibrated images,
plus raw data and ancillary information for every observation; these
will share the same obs\_id, but will have different granule\_gid.
\subsubsection{Data Description}
\paragraph{dataproduct\_type}
The dataproduct\_type parameter describes the high level scientific
organization of the data product linked by the access\_url parameter,
or directly included in the table (in which case the value is `ci'
for catalogue\_item). EPNCore currently defines several types listed
below. Data providers must select the type most adapted to their
data. In complex situations (e.g., when a file contains several data
products), several types can be used to describe the same granule by
using a hash-separated-list — although using several granules to
describe the file content may be a better solution.
In EPN-TAP these types are identified by a 2-characters ID, so that
multivalued queries are unambiguous. Possible IDs are listed below with
their meaning:
\begin{itemize}
\item \textbf{im }(= image): scalar field with two spatial axes, or
association of several such fields, e.g., images with multiple color
planes, from multichannel or filter cameras. Preview images (e.g., map
with axis and caption) also belong here. Conversely, vectorial 2D fields
are described as spatial\_vector (see below).
\item \textbf{ma }(= map): scalar field / rasters with two spatial axes
covering a large area and projected either on the sky or on a planetary
body, associated to spatial\_coordinate\_description and map\_projection
parameters (with a short enumerated list of possible values); each pixel
is associated to 2D coordinates (e.g., fits with WCS). This is mostly
intended to identify radiometrically calibrated and orthorectified
images with complete coverage that can be used as reference basemaps,
but this also includes HiPS.
\item \textbf{sp }(= spectrum): measurements organized primarily along a
spectral axis, e.g., radiance spectra. This includes spectral aggregates
(series of related spectral segments with non-connected spectral ranges,
e.g., from several channels of the same instrument, various orders from
an échelle spectrometer, composite spectra, SED, etc).
\item \textbf{ds }(= dynamic\_spectrum): consecutive spectral measurements
through time, organized primarily as a time series. This typically
implies successive spectra of the same target / field of view.
\item \textbf{sc }(= spectral\_cube): sets of consecutive
spectral measurements with 1 or 2D spatial coverage, e.g., imaging
spectroscopy. The choice between image and spectral\_cube is dictated by
the characteristics of the instrument (which dimension is most resolved
and which dimensions are acquired simultaneously). The choice between
dynamic\_spectrum and spectral\_cube is related to the uniformity of
the field of view and by practices in the science field.
\item \textbf{pr }(= profile): scalar or vectorial measurements along
1 spatial dimension, e.g., atmospheric profiles, atmospheric paths,
sub-surface profiles, traverses…
\item \textbf{pf }(= photometric\_function): scalar or vectorial
measurements along 1 angular dimension, e.g., phase or polarization
curves, phase functions, emission-phase function sequences… Does not
handle variations along several angular axes. This is typically associated
to variations in illumination angle parameters.
\item \textbf{vo }(= volume): measurements with 3 spatial dimensions,
e.g., internal or atmospheric structures, including shells/shape models
(3D surfaces).
\item \textbf{mo }(= movie): sets of chronological 2D spatial measurements
(consecutive images)
\item \textbf{cu }(= cube): multidimensional data with 3 or more axes,
e.g., all that is not described by other 3D data types such as spectral
cube, volume, or movie. This is intended to accommodate unusual data with multiple
dimensions. This can be used for 3D ancillary data associated to spectral
cubes, e.g., providing the coordinates or illumination angles for
each spectrum.
\item \textbf{ts }(= time\_series): measurements organized primarily
as a function of time (with exception of dynamical spectra and movies,
i.e., usually a scalar quantity). Typical examples of time series include
space-borne dust detector measurements, daily or seasonal curves measured
at a given location (e.g., a lander), and light curves.
\item \textbf{ca }(= catalogue): applies to a granule providing a
catalogue of object parameters, a list of features, a table of granules
in another TAP service, a list of events, a list of spectral lines\dots The
result metadata table of a service query can also be considered as a
catalogue. Catalogues can be provided as VOTable (possibly containing
multiple tables, although this is not supported by SAMP). It is good
practice to describe the type of data included in the catalogue using
a hash-separated-list (e.g., a table of spectra should be described by
ca\#sp, so that it will respond to a query for spectra).
\item \textbf{ci }(= catalogue\_item): applies when the service itself
provides a catalogue with entries described as individual granules, in
particular when there is no associated file (e.g., a list of asteroid
properties or spectral lines). Catalogue\_item can be limited to scalar
quantities (including strings), and possibly to a single element. This
organization allows the user to search inside the catalogue from
the TAP query interface. In practice, Spice kernels are identified as
catalogue\_items because they are usually associated to a set of scalar
parameters.
\item \textbf{sv} (= spatial\_vector): vector information associated
to localization, such as a spatial footprints, a GIS-related element,
etc — e.g., a kml or geojson file (STC-S strings are provided though
the s\_region parameter, though). This includes maps of vectors, e.g.,
wind maps.
\item \textbf{ev} (= event): introduces individual VOevents
formatted according to IVOA standard (or possibly events with other
formatting). Characteristics are provided via the event\_* parameters.
\end{itemize}
\textbf{\\}
\emph{Example TAP queries}:
\begin{verbatim}
SELECT TOP 1000 * FROM iks.epn_core
WHERE dataproduct_type LIKE '%sp%'
\end{verbatim}
or
\begin{verbatim}
SELECT TOP 1000 * FROM iks.epn_core
WHERE ivo_hashlist_has(dataproduct_type, 'sp') = 1
\end{verbatim}
will return only image data (both syntaxes handle lists of values).
\paragraph{measurement\_type}
The measurement\_type parameter defines the physical quantities
contained in the data, using UCDs \citep{2021ivoa.spec.0616C}. It relates to the reported quantity,
not to the type of experiment. Therefore, only UCDs related to physical
quantities can be used; e.g., phys.absorption;em.opt.I is eligible,
while pos.lunar.occult (occultation by lunar limb) is not. This is used in particular to provide
indications to visualization/processing tools.
Whenever several quantities are comprised in the data, the
measurement\_type parameter must describe all these quantities, using
multiple UCDs in a hash-separated-list.
\textbf{\\}
\emph{Example}:
\begin{itemize}
\item For images in general (i.e., actual measurements with a camera),
the relevant UCD is phot.radiance;obs.image (or phot.radiance;obs.image;stat.uncalib if not
calibrated).
\item For spectra: phot.radiance describes radiance, phys.reflectance
stands for reflectance factor (= I/F), and phot.flux.density for a
\emph{flux} vector (irradiance). The associated spectral vector is
described by UCDs em.wl, em.freq or em.energy, and the related error
has the form stat.error;phys.reflectance (for I/F).
\item Quantities derived from modeling or simulations are described by the
regular UCD with ``;meta.modelled'' appended.
\end{itemize}
\paragraph{processing\_level}
The processing\_level parameter is intended to provide the user with
a quick evaluation of data readiness level. EPN-TAP uses
a simplified scheme as described below.
Several classifications are in use in different contexts, as summarized
in Table~\ref{table:proclev}. EPNCore uses the CODMAC / PDS3 levels but removes
the intermediate calibration levels; this is equivalent to using the
simplified PDS4 levels and maintaining a separate level for ancillary
data. ``Partially calibrated'' data collections are in general considered as not
calibrated, but this evaluation is up to the data provider, depending on
context. ``Ancillary'' data include all extra information documenting
the measurements, in particular coordinates or geometry files. Several
processing levels can be included in the same service (in particular
calibrated and ancillary data, but also raw data). When mixed in the
same file, the choice is left to the data provider.
Note that the optional processing\_level\_desc parameter
is available to provide, e.g., a specific encoding of processing
levels related to a data collection, or more details about partial calibrations.
Most EPN-TAP data services are expected to include Calibrated or
Derived data.
\begingroup\small\begin{inlinetable}
\begin{tabular}{lllllp{2cm}lp{0.35\textwidth}}
\sptablerule
\vbox{\vskip 2pt\hbox{EPN-}\vskip 3pt\hbox{TAP2}}
&CODMAC&PSA&NASA&PDS3&PDS4&ObsTAP&Description\\
\sptablerule
1 &1 (raw)&(0?)&&UDR&Telemetry&0
&Unprocessed Data Record (low-level encoding,
e.g., telemetry from a spacecraft instrument.
Normally available only to the original team)\\
2 &2 (edited)&1&0&EDR&Raw&1
&Experiment Data Record (often referred to as ``raw data'':
decommutated, but still affected by instrumental effects)\\
2 or 3&\_&&&\_&Partially calibrated&
&Processed beyond the raw stage,
but have not yet reached calibrated status (PDS4)\\
3 &3 (calibrated)&2&1A&RDR&Calibrated&2
&Reduced Data Record (``calibrated'' in physical units, no resampling)\\
5 (yes, 5)&4 (resampled)&&1B&REFDR&Derived&3
&Reformatted Data Record
(mosaics or composite of several observing sessions,
involving some level of data fusion)\\
5 &5 (derived)&3&2-5&DDR&Derived&4
&Derived Data Record
(result of data analysis, directly usable by other communities
with no further processing)\\
6&6 (ancillary)&&&ANCDR&Derived&
&Ancillary Data Record
(extra data specifically supporting a data set,
such as coordinates, geometry… but also dark currents, flat fields…) \\
\end{tabular}
\label{table:proclev}
\end{inlinetable}\endgroup
Notes:
\begin{itemize}
\item This table is a compilation of information from
PSA, PDS4, and ObsCore documents
\item The PDS3 column corresponds to the PDS3/PSA PRODUCT\_TYPE keyword
\item Descriptions are extracted from PSA and PDS4 documentations,
with comments.
\end{itemize}
\subsubsection{Target description}
\paragraph{target\_name}
The target\_name parameter identifies a target by name or ID. The
target may be any Solar System body, exoplanet, planetary sample, or
meteorite, plus in some cases astronomical objects or spacecraft. Any
other feature (craters, regions, atmospheric layers\dots) must be named
using the optional feature\_name parameter (see \ref{par:feature_name}). This parameter
can be multivalued only to describe several targets related to a granule
(e.g., with events). Alternative names of the same target must not be
listed here, but may be provided through the optional alt\_target\_name
parameter. In some rare cases no target name can be defined, so this
parameter can remain empty.
The best practice is to use the official designation of the target as
defined by IAU. This parameter is case sensitive (mixing lower/upper
cases) and all values must use the standard spelling and case; unusual
characters (such as intermediate spaces) are allowed, except quotes
and hashes (preferably changed to \_).
Data providers must be aware that services which do not expose the IAU
designations will not return answers to queries using them.
Conversely, users must be aware that some
data of interest might not be visible if they do not use the recommended
IAU nomenclature for planetary bodies. The quaero name
resolver\footnote{\url{https://ssp.imcce.fr/webservices/ssodnet/api/quaero/}}
from IMCCE may help data providers (as well as users) to handle multiple
denominations; it is available from the VESPA portal to support queries.
Concerning celestial objects (at fixed position, i.e., stars,
galaxies…) the name should be identifiable through
Simbad\footnote{\url{http://simbad.u-strasbg.fr/simbad/}}.
Other best practices are listed below:
\begin{itemize}
\item The Exoplanet Encyclopedia provides a nearly
complete list of currently known extrasolar planets:
\url{http://exoplanet.eu/index.php}
(also available as an EPN-TAP service)
\item Meteorite catalogues can be found here:\\
\url{http://www.nhm.ac.uk/research-curation/research/projects/metcat/search/indexsing.dsml}
\\and
\url{http://www.lpi.usra.edu/meteor/index.php}.
\item The catalog of lunar samples is available here:
\url{http://www.lpi.usra.edu/lunar/samples/}
\item Other planetary samples are listed in topical web sites,
e.g., samples from the Stardust mission are described here:\\
\url{http://curator.jsc.nasa.gov/stardust/catalog/}
\item Consortia such as the IGSN\footnote{\url{https://www.igsn.org}}
issue IDs for samples
\item Asteroids: Usage is to use preferably name (if it exists) or
principal designation (the number is not used here, it can be included
in alt\_target\_name)
\item Meteors: the IAU Meteor Data Center assigns IDs to meteors and
meteor showers
\item Calibration targets: values can relate to existing names in a
given archive (e.g., the PSA contains values such as bias, checkout,
dark, flatfield, internal source…)
\end{itemize}
\textbf{\\}
\emph{Example TAP queries}:
\begin{verbatim}
SELECT TOP 1000 * FROM spectro_asteroids.epn_core
WHERE (target_name LIKE '%Ceres%' OR target_name LIKE '%Vesta%')
AND (ivo_hashlist_has(target_class, 'dwarf_planet') = 1 OR
ivo_hashlist_has(target_class, 'asteroid') = 1)
\end{verbatim}
Will return data only from 1 Ceres or 4 Vesta.
The LIKE operator is used here to handle cases when 1) the name includes a
number, either `1 Ceres' or `(1) Ceres', which (although not recommended)
may occur in some services, 2) the special case of dwarf planets, which
may be tagged as asteroids, or as `dwarf\_planet\#asteroid' (recommended,
see below).
The usage of LIKE cannot be generalized to all target\_class values,
because of `planet' which also responds to `dwarf\_planet' and `exoplanet'.
The ivo\_hashlist\_has function smoothly handles all values
of target\_class. Also beware that using LIKE on target\_name may generate false positives.
\textbf{\\}
\emph{Example}:
1P is the official IAU designation for comet Halley
\paragraph{target\_class}
The target\_class parameter identifies the type of the target. Solar
System bodies are defined without ambiguity by the couple target\_class
and target\_name. In other cases, targets may have no proper name
(i.e., samples), but the target\_class parameter must contain a value in
all cases. The possible values for target\_class are:
\begin{itemize}
\item asteroid
\item dwarf\_planet
\item planet
\item satellite
\item comet
\item exoplanet
\item interplanetary\_medium
\item sample
\item sky
\item spacecraft
\item spacejunk
\item star
\item calibration
\end{itemize}
Usage:
\begin{itemize}
\item Most Solar System bodies are expected to belong to a single target class,
and the pair of values is required to resolve homographies (such as Io).
However, classifications may evolve with time.
\item It is therefore good practice to use a hash-separated-list to
handle any ambiguity or evolution in target classes, such as dwarf
planets (dwarf\_planet\#asteroid) or transitional objects (comet\#asteroid).
This will help support queries by target\_class in particular.
\item If several target\_name values are provided for a granule,
several target\_class may be present as hash-separated values.
\item ``interplanetary\_medium'' refers in particular to interplanetary
dust observed in context (not to samples), or to simulations
(e.g., of the primordial nebula).
Details can be provided through target\_region.
\item ``sample'' refers to lunar or planetary samples, to meteorites,
but also to terrestrial samples, e.g., in laboratory studies.
\item ``satellite'' stands for natural satellites only --- artificial
satellites are handled though spacecraft or spacejunk.
\item ``star'' is used typically for calibration targets, and for the Sun.
\item ``sky'' should be used for all other celestial bodies,
usually referred to by their sky coordinates.
It also includes the Interstellar Medium.
\item ``calibration'' is used for observations only related to instrument
or signal calibration, including dark current, flat field, reference
sample (in lab), etc (use of ``calibration'' with planetary bodies is
left to data providers).
\item planetary rings are not considered a target class, but appear as
target\_region = ``ring'' with their planet indicated in target\_name.
\end{itemize}
\emph{Comment}:
All ``Types'' of objects used by the quaero resolver are included
in the EPNCore list: ``asteroid'', ``spacecraft'', ``comet'',
``exoplanet'', ``spacejunk'', ``satellite'', ``planet'', ``dwarf planet'',
``star''.\\
Additional EPNCore classes are: ``interplanetary\_medium'',
``sample'', ``sky'', ``calibration''.
\subsubsection{Axes}
EPNCore describes the data coverage along 8 main axes:
time, spectral, spatial (3 coordinates), and viewing geometry (3 angles).
Coverages along the time/space/spectral axes are described with a range
and a resolution and/or sampling step.
Each quantity is a set of 2 parameters providing min and max values. They
must both be present with the same value when min = max. Additional single
parameters provide extra information: spatial\_frame\_type, s\_region,
and the optional coverage parameter.
\paragraph{time (min/max)}
The time\_min and time\_max parameters provide the date and time of
acquisition in the observer frame.
The time parameters are always provided in UTC and formatted in Julian
Days (expressed in double precision). EPNCore uses standard JD to avoid
ambiguity with time origin (as opposed to ObsCore which uses Modified
JD). The use of double precision floats ensures an accuracy on the order
of 1 ms, which is considered sufficient for search purposes (the initial
accuracy is preserved in the data itself). A client front-end may expose
times formatted in a more convenient way for human users.
\textbf{\\}
\emph{Example TAP queries}:
\begin{description}
\item[Search data contained in a time range:]\mbox{}
\begin{verbatim}
SELECT TOP 100 * FROM spectro_asteroids.epn_core
WHERE time_min > 2455197.5 AND time_max < 2455927.5
\end{verbatim}
\item[Search data described by a start time parameter:]\mbox{}
\begin{verbatim}
SELECT TOP 100 * FROM spectro_asteroids.epn_core
WHERE time_min BETWEEN 2455197.5 AND 2455927.5
\end{verbatim}
\end{description}
Time range is, by default, provided in the observer frame (i.e., at the
observer location at that moment), which is almost always the native time
in the data. For instance, space-borne observations are usually documented
with spacecraft on-board time, which is expected here (provided as JD,
not as on-board clock timing). For other cases, the location where time
is measured must be provided through the time\_refposition parameter.
To support space-borne vs ground-based campaigns, multiple spacecraft
observations, or survey of periodic events, measurement times need to be
corrected for light path. Whenever comparisons are potentially involved,
the use of the target\_time\_min and target\_time\_max parameters is
recommended in addition to time\_min and time\_max to make this kind of
comparison simpler.
Non-compulsory parameters may be used to accommodate additional,
specially formatted time scales such as native on-board time for space
borne observations. The information of the time\_min and time\_max
parameters is however highly recommended for all observations,
as it is used by default to search data collections.
\paragraph{time\_sampling\_step (min/max)}
These parameters provide the sampling step in seconds for measurements
of dynamical phenomena, and for computations. This is the time between
2 successive measurements or data, which is mostly relevant when the
measurements are regularly spaced. This may also reflect an input
parameter, e.g., for simulations or ephemeris computations. This
parameter is intended to allow the user to search for time-resolved
observations of dynamic phenomena, but can also accommodate the
``repetition time'' for several types of instruments.
\paragraph{time\_exp (min/max)}
These parameters correspond to the integration time (or exposure
time) of measurements in seconds. They provide an estimate of the time
resolution for dynamical phenomena, as well as an indication of relative
S/N ratio inside a given data collection. This time is usually shorter than
the time\_sampling\_step if both are present. It provides the overall
integration time, i.e., individual exposure time $\times$ number of summed
frames when relevant.
\paragraph{spectral\_range (min/max)}
The spectral\_range parameters define the upper and lower bounds of the
spectral domain of the data. This quantity is conventionally expressed
on a frequency scale in Hertz, although the client front-end may propose
conversions to other units/scales. The spectral range and associated
parameters only apply to electromagnetic waves. See the optional
parameters from the particle spectroscopy extension for particle energy or mass detection.
Since this is the standard parameter to identify a spectral range, it
is recommended to fill it even for images obtained through a filter
(for instance, by giving the central wavelength $\pm$ FWHM/2,
or even just the central wavelength as both minimum and maximum,
which may already be enough to make the granule findable).
The SVO Filter Profile
Service\footnote{\url{http://svo2.cab.inta-csic.es/theory/fps3/}}
provides responses for many instrument filters;
bandwidths can be retrieved easily with a python library:
\url{https://github.com/hover2pi/svo_filters}.
Care must be taken when computing spectral ranges from metadata given as
wavelength. For instance, when the spectral range $\lambda_\mu$ is
given in micrometers, with the constants from sect.~\ref{sect:convs} one
would compute (notice the change of roles!)
\begin{eqnarray*}
\hbox{\texttt{spectral\_range\_min}} &=&
2.99792458\cdot 10^{14}/\lambda_{\mu,\textrm{max}}\\
\hbox{\texttt{spectral\_range\_max}} &=&
2.99792458\cdot 10^{14}/\lambda_{\mu,\textrm{min}}\\
\end{eqnarray*}
\paragraph{spectral\_sampling\_step (min/max)}
The spectral\_sampling\_step parameters provide the spectral separation
between the centers of two adjacent filters or channels. Similar to the
spectral\_range quantities, they are expressed on a frequency scale in Hz.
These parameters are most relevant for radio observations
(long wavelengths / low frequencies) with regular sampling.
They can also help distinguish between Nyqvist and sub-Nyqvist
sampling rates (together with resolution).
Again, care must be taken when converting sampling steps from
non-frequency metadata. For example, when the spectral sampling step is
given in micrometers of wavelength,
$\Delta\lambda_\mu$, with the constants from
sect.~\ref{sect:convs} one would compute
$$\hbox{\texttt{spectral\_sampling\_step\_*}}
= 2.99792458\cdot10^{14}\Delta\lambda_\mu/\lambda_\mu^2.$$
For wavelengths in nanometers, $\lambda_{\textrm{nm}}$, this becomes
$$\hbox{\texttt{spectral\_sampling\_step\_*}}
= 2.99792458\cdot10^{11}\Delta\lambda_{\textrm{nm}}/
\lambda_{\textrm{nm}}^2.$$
Finally, for a wave number $u$ in $\textrm{cm}^{-1}$, the conversion is
$$\hbox{\texttt{spectral\_sampling\_step\_*}} =
2.99792458\cdot10^{6}\Delta u$$
\paragraph{spectral\_resolution (min/max)}
The spectral\_resolution parameters provide the (dimensionless)
resolving power $\lambda/\Delta\lambda = \nu/\Delta\nu$.
These parameters are mostly intended to provide an order of magnitude,
e.g., to distinguish between Fourier spectrometers, grating spectrometers
or filter cameras, or between observations related to surfaces or
atmospheres. This is often most relevant for optical/infrared observations
(short wavelengths / high frequencies).
Note that when computing the spectral resolution using wavelengths,
minimum and maximum change roles, as in
\begin{eqnarray*}
\hbox{\texttt{spectral\_resolution\_min}}
&=& \lambda_\textrm{min} / \textrm{max}(\Delta\lambda)\\
\hbox{\texttt{spectral\_resolution\_max}}
&=& \lambda_\textrm{max} / \textrm{min}(\Delta\lambda)
\end{eqnarray*}
\paragraph{c1 (min/max)}
\paragraph{c2 (min/max)}
\vspace{-15pt}
\paragraph{c3 (min/max)}