From 4f483af0d4a8bb689bc1c46f3dcf0187f5fb2ab0 Mon Sep 17 00:00:00 2001 From: TR Jaffe Date: Wed, 30 Oct 2024 10:10:49 -0400 Subject: [PATCH 1/4] TJ first edits --- VOHE-Note.tex | 10 +++++++--- ivoatex | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/VOHE-Note.tex b/VOHE-Note.tex index 2f74a6f..92fa875 100644 --- a/VOHE-Note.tex +++ b/VOHE-Note.tex @@ -593,7 +593,7 @@ \subsection{Tools for data extraction and visualisation} \gls{HE} data are typically multi-dimensional ({\em e.g.\/}, 2 spatial dimensions, time, energy, possibly polarisation) and may be complex and diverse at lower levels. Therefore one may commonly find specific tools to process the data for a given facility, {\em e.g.\/}, CIAO for Chandra, \gls{SAS} for \gls{XMM-Newton}, or Gammapy for gamma-ray data, with a particular focus on Cherenkov data as foreseen by \gls{CTAO}. -However, many tools in a high energy astrophysics data analysis package may perform common tasks in a mission-independent way and can work well with similar data from other facilities. For example, one commonly needs to be able to filter and project the multi-dimensional data to select specific data subsets with manageable sizes and eliminate extraneous data. Some tool sets include built-in generic filtering and binning capabilities so that a general purpose region filtering and binning syntax is available to the end user. +However, many tools in a high energy astrophysics data analysis package may perform common tasks in a mission-independent way and can work well with similar data from other facilities. For example, one commonly needs to be able to filter and project the multi-dimensional data to select specific data subsets with manageable sizes and eliminate extraneous data. Some tool sets include built-in generic filtering and binning capabilities so that a general purpose region filtering and binning syntax is available to the end user. Examples include the HEASoft package\footnote{\url{https://https://heasarc.gsfc.nasa.gov/docs/software/heasoft/}}, Gammapy\footnote{\url{https://gammapy.org/}}, Gamma-ray Data Tools (GDT)\footnote{\url{https://astro-gdt.readthedocs.io/}}, etc. A high energy astrophysics data analysis package typically includes tools that apply or re-apply instrumental calibrations to the data, and as described above these may be observatory-specific. More general algorithms ({\em e.g.\/}, source detection) and utility tools ({\em e.g.\/}, extract an observed spectrum from a region surrounding a source) are applied to calibrated data to extract data subsets that can then be fed into modeling tools ({\em e.g.\/}, Xspec, Sherpa, or Gammapy) together with the appropriate instrumental responses (\gls{IRF}s, or \gls{RMF}s and \gls{ARF}s) to derive physical quantities. Since instrumental responses are often designed to be compliant with widely adopted standards, the tools that apply these responses in many cases will interoperate with other datasets that use the same standards. @@ -660,7 +660,7 @@ \subsection{UC4: Multi-wavelength and multi-messenger science} For both use cases, any type of data should be findable on the \gls{VO} and retrievable. And the data should have a standardised open format (\gls{OGIP}, \gls{GADF}, \gls{VODF}). -Such use case is already in use with small data sets shared by \gls{VHE} experiments. In +Such use cases are already common with numerous examples in the x-rays in the decades that missions have been contributing to the standardized HEASoft package. Other examples include small data sets shared by \gls{VHE} experiments. In \citep{2019A&A...625A..10N, 2022A&A...667A..36A}, groups of astronomers working on the Gammapy library had successfully analysed DL3 data taken on the Crab nebula by different facilities (MAGIC, \gls{HESS}, FACT, VERITAS, Fermi-LAT and HAWC). A real statistical joint analysis has been performed to derive an emitting model of the Crab pulsar wind nebula over more @@ -757,7 +757,8 @@ \subsubsection{Provenance} \subsubsection{VOEvent} -Source variability and observations of transient are common in the \gls{HE} domain, and as such, handling of alerts is generally including in the requirements of \gls{HE} observatories. Alerts are both sent and received by \gls{HE} observatories. The \gls{IVOA} recommendation VOEvent \citep{2017ivoa.spec.0320S} is thus of interest to the \gls{HE} domain. +Source variability and observations of transient are common in the \gls{HE} domain, and as such, handling of alerts is generally including in the requirements of \gls{HE} observatories. Alerts are both sent and received by \gls{HE} observatories. The \gls{IVOA} recommendation VOEvent \citep{2017ivoa.spec.0320S} is thus of interest to the \gls{HE} domain. This standard has been part of the decades-long success of of the General Coordinates Network (GCN)\footnote{\url{https://gcn.nasa.gov/}}, an alert system first created in the 1990's for BATSE\footnote{\url{https://ui.adsabs.harvard.edu/abs/1995Ap\%26SS.231..235B/abstract}} that has been through a number of technology and standards refreshes. See also \S~\ref{sec:voevent_he}. + \subsubsection{Measurements} @@ -1009,6 +1010,9 @@ \subsection{Use of Datalink for HE products} In the second option, we would provide links to the relevant information from the base "event-list" (\ref{sec:event-bundlle-or-list}) exposed in the discovery service. This could be done using Datalink and a list of links to each contextual information such as the \gls{IRF}s. The Event-list Context Data Model (see \ref{sec:EventListContext}) would provide the concepts and vocabulary to characterise the \gls{IRF}s and other information relevant to the analysis of an event-list. These specific concepts and terms describing the various flavors of \gls{IRF}s and \gls{GTI}s will be given in the semantics and content\_qualifier FIELDS of the DataLink response to qualify the links. The different links can point to different dereferencable URLs or alternbatively to different fragments of the same drefereencable URL as stated by the DataLink specification. +\subsection{VOEvent modernization}\lable{sec:voevent_he} + +The modern GCN is exploring new standards more compatible with modern event brokers such as Kafka. This includes defining metadata serialization in formats such as JSON and new schema\footnote{\url{https://gcn.nasa.gov/docs/notices/schema}} to describe how to use them. This interest group should actively engage the growing community of projects publishing transient alerts through systems that have outgrown the VOEvent standard. %\todo[inline]{To be completed: show an example ?} diff --git a/ivoatex b/ivoatex index 4cb591c..c838d62 160000 --- a/ivoatex +++ b/ivoatex @@ -1 +1 @@ -Subproject commit 4cb591cb784bd06378b537cff05a02e658aad143 +Subproject commit c838d628821446b112df6a92f5010423290db8e2 From d21583e050bb0f53d37d6d8727507a52f31ce3fd Mon Sep 17 00:00:00 2001 From: TR Jaffe Date: Wed, 30 Oct 2024 10:22:48 -0400 Subject: [PATCH 2/4] Add ref and fix typo --- VOHE-Note.bib | 16 ++++++++++++++++ VOHE-Note.tex | 4 ++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/VOHE-Note.bib b/VOHE-Note.bib index c07eb2e..bb51e0d 100644 --- a/VOHE-Note.bib +++ b/VOHE-Note.bib @@ -1,3 +1,19 @@ +% Origin of GCN +@ARTICLE{1995Ap&SS.231..235B, + author = {{Barthelmy}, S.~D. and {Butterworth}, P. and {Cline}, T.~L. and {Gehrels}, N. and {Fishman}, G.~J. and {Kouveliotou}, C. and {Meegan}, C.~A.}, + title = "{BACODINE, the Real-Time BATSE Gamma-Ray Burst Coordinates Distribution Network}", + journal = {\apss}, + keywords = {Delay Time, Scientific Community, Distribution Network, Total Delay, Distribution Method}, + year = 1995, + month = sep, + volume = {231}, + number = {1-2}, + pages = {235-238}, + doi = {10.1007/BF00658623}, + adsurl = {https://ui.adsabs.harvard.edu/abs/1995Ap&SS.231..235B}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + %HESS paper describing data format %related doi= {https://doi.org/10.48550/arXiv.1810.04516} diff --git a/VOHE-Note.tex b/VOHE-Note.tex index 92fa875..e3b980c 100644 --- a/VOHE-Note.tex +++ b/VOHE-Note.tex @@ -757,7 +757,7 @@ \subsubsection{Provenance} \subsubsection{VOEvent} -Source variability and observations of transient are common in the \gls{HE} domain, and as such, handling of alerts is generally including in the requirements of \gls{HE} observatories. Alerts are both sent and received by \gls{HE} observatories. The \gls{IVOA} recommendation VOEvent \citep{2017ivoa.spec.0320S} is thus of interest to the \gls{HE} domain. This standard has been part of the decades-long success of of the General Coordinates Network (GCN)\footnote{\url{https://gcn.nasa.gov/}}, an alert system first created in the 1990's for BATSE\footnote{\url{https://ui.adsabs.harvard.edu/abs/1995Ap\%26SS.231..235B/abstract}} that has been through a number of technology and standards refreshes. See also \S~\ref{sec:voevent_he}. +Source variability and observations of transient are common in the \gls{HE} domain, and as such, handling of alerts is generally including in the requirements of \gls{HE} observatories. Alerts are both sent and received by \gls{HE} observatories. The \gls{IVOA} recommendation VOEvent \citep{2017ivoa.spec.0320S} is thus of interest to the \gls{HE} domain. This standard has been part of the decades-long success of of the General Coordinates Network (GCN)\footnote{\url{https://gcn.nasa.gov/}}, an alert system first created in the 1990's for BATSE \citep{1995Ap&SS.231..235B} that has been through a number of technology and standards refreshes. See also \S~\ref{sec:voevent_he}. @@ -1008,7 +1008,7 @@ \subsection{Use of Datalink for HE products} In the first option, the "event-bundle" dataset (\ref{sec:event-bundlle-or-list}) exposed in the discovery service contains all the relevant information, e.g. several frames in the \gls{FITS} file, one corresponding to the event-list itself, and the others providing good/stable time intervals, or any \gls{IRF} file. This is what was done in the current \gls{GADF} data format (see \ref{sec:GADF}). In this option, the content of the event-list package should be properly defined in its description: what information is included and where is it in the dataset structure? The Event-list Context Data Model (see \ref{sec:EventListContext}) would be useful to provide that information. In the second option, we would provide links to the relevant information from the base "event-list" (\ref{sec:event-bundlle-or-list}) exposed in the discovery service. This could be done using Datalink and a list of links to each contextual information such as the \gls{IRF}s. The Event-list Context Data Model (see \ref{sec:EventListContext}) would provide the concepts and vocabulary to characterise the \gls{IRF}s and other information relevant to the analysis of an event-list. These specific concepts and terms describing the various flavors of \gls{IRF}s and \gls{GTI}s will be given in the semantics and content\_qualifier FIELDS of the DataLink response to qualify the links. The different links can point to different -dereferencable URLs or alternbatively to different fragments of the same drefereencable URL as stated by the DataLink specification. +dereferencable URLs or alternatively to different fragments of the same dereferencable URL as stated by the DataLink specification. \subsection{VOEvent modernization}\lable{sec:voevent_he} From 6309263dca49c091f3cd6950d50112cf3f93088b Mon Sep 17 00:00:00 2001 From: TR Jaffe Date: Wed, 30 Oct 2024 10:28:55 -0400 Subject: [PATCH 3/4] minor mods --- VOHE-Note.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/VOHE-Note.tex b/VOHE-Note.tex index e3b980c..acbda2e 100644 --- a/VOHE-Note.tex +++ b/VOHE-Note.tex @@ -593,7 +593,7 @@ \subsection{Tools for data extraction and visualisation} \gls{HE} data are typically multi-dimensional ({\em e.g.\/}, 2 spatial dimensions, time, energy, possibly polarisation) and may be complex and diverse at lower levels. Therefore one may commonly find specific tools to process the data for a given facility, {\em e.g.\/}, CIAO for Chandra, \gls{SAS} for \gls{XMM-Newton}, or Gammapy for gamma-ray data, with a particular focus on Cherenkov data as foreseen by \gls{CTAO}. -However, many tools in a high energy astrophysics data analysis package may perform common tasks in a mission-independent way and can work well with similar data from other facilities. For example, one commonly needs to be able to filter and project the multi-dimensional data to select specific data subsets with manageable sizes and eliminate extraneous data. Some tool sets include built-in generic filtering and binning capabilities so that a general purpose region filtering and binning syntax is available to the end user. Examples include the HEASoft package\footnote{\url{https://https://heasarc.gsfc.nasa.gov/docs/software/heasoft/}}, Gammapy\footnote{\url{https://gammapy.org/}}, Gamma-ray Data Tools (GDT)\footnote{\url{https://astro-gdt.readthedocs.io/}}, etc. +However, many tools in a high energy astrophysics data analysis package may perform common tasks in a mission-independent way and can work well with similar data from other facilities. For example, one commonly needs to be able to filter and project the multi-dimensional data to select specific data subsets with manageable sizes and eliminate extraneous data. Some tool sets include built-in generic filtering and binning capabilities so that a general purpose region filtering and binning syntax is available to the end user. Examples include the HEASoft package\footnote{\url{https://https://heasarc.gsfc.nasa.gov/docs/software/heasoft/}} (enabled by the OGIP standards mentioned above), Gammapy\footnote{\url{https://gammapy.org/}}, Gamma-ray Data Tools (GDT)\footnote{\url{https://astro-gdt.readthedocs.io/}}, etc. A high energy astrophysics data analysis package typically includes tools that apply or re-apply instrumental calibrations to the data, and as described above these may be observatory-specific. More general algorithms ({\em e.g.\/}, source detection) and utility tools ({\em e.g.\/}, extract an observed spectrum from a region surrounding a source) are applied to calibrated data to extract data subsets that can then be fed into modeling tools ({\em e.g.\/}, Xspec, Sherpa, or Gammapy) together with the appropriate instrumental responses (\gls{IRF}s, or \gls{RMF}s and \gls{ARF}s) to derive physical quantities. Since instrumental responses are often designed to be compliant with widely adopted standards, the tools that apply these responses in many cases will interoperate with other datasets that use the same standards. @@ -1007,7 +1007,7 @@ \subsection{Use of Datalink for HE products} In the first option, the "event-bundle" dataset (\ref{sec:event-bundlle-or-list}) exposed in the discovery service contains all the relevant information, e.g. several frames in the \gls{FITS} file, one corresponding to the event-list itself, and the others providing good/stable time intervals, or any \gls{IRF} file. This is what was done in the current \gls{GADF} data format (see \ref{sec:GADF}). In this option, the content of the event-list package should be properly defined in its description: what information is included and where is it in the dataset structure? The Event-list Context Data Model (see \ref{sec:EventListContext}) would be useful to provide that information. -In the second option, we would provide links to the relevant information from the base "event-list" (\ref{sec:event-bundlle-or-list}) exposed in the discovery service. This could be done using Datalink and a list of links to each contextual information such as the \gls{IRF}s. The Event-list Context Data Model (see \ref{sec:EventListContext}) would provide the concepts and vocabulary to characterise the \gls{IRF}s and other information relevant to the analysis of an event-list. These specific concepts and terms describing the various flavors of \gls{IRF}s and \gls{GTI}s will be given in the semantics and content\_qualifier FIELDS of the DataLink response to qualify the links. The different links can point to different +In the second option, we would provide links to the relevant information from the base "event-list" (\ref{sec:event-bundlle-or-list}) or product exposed in the discovery service. This could be done using Datalink and a list of links to each contextual information such as the \gls{IRF}s. The Event-list Context Data Model (see \ref{sec:EventListContext}) would provide the concepts and vocabulary to characterise the \gls{IRF}s and other information relevant to the analysis of an event-list. These specific concepts and terms describing the various flavors of \gls{IRF}s and \gls{GTI}s will be given in the semantics and content\_qualifier FIELDS of the DataLink response to qualify the links. The different links can point to different dereferencable URLs or alternatively to different fragments of the same dereferencable URL as stated by the DataLink specification. \subsection{VOEvent modernization}\lable{sec:voevent_he} From 1890122e71ccb556a28295100e4bc3dd03e8cd8b Mon Sep 17 00:00:00 2001 From: TR Jaffe Date: Tue, 5 Nov 2024 12:03:01 -0500 Subject: [PATCH 4/4] Fixing errors --- .gitmodules | 3 --- VOHE-Note.tex | 2 +- ivoatex | 1 - 3 files changed, 1 insertion(+), 5 deletions(-) delete mode 160000 ivoatex diff --git a/.gitmodules b/.gitmodules index 9785d46..e69de29 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +0,0 @@ -[submodule "ivoatex"] - path = ivoatex - url = https://github.com/ivoa-std/ivoatex diff --git a/VOHE-Note.tex b/VOHE-Note.tex index acbda2e..d90246e 100644 --- a/VOHE-Note.tex +++ b/VOHE-Note.tex @@ -1010,7 +1010,7 @@ \subsection{Use of Datalink for HE products} In the second option, we would provide links to the relevant information from the base "event-list" (\ref{sec:event-bundlle-or-list}) or product exposed in the discovery service. This could be done using Datalink and a list of links to each contextual information such as the \gls{IRF}s. The Event-list Context Data Model (see \ref{sec:EventListContext}) would provide the concepts and vocabulary to characterise the \gls{IRF}s and other information relevant to the analysis of an event-list. These specific concepts and terms describing the various flavors of \gls{IRF}s and \gls{GTI}s will be given in the semantics and content\_qualifier FIELDS of the DataLink response to qualify the links. The different links can point to different dereferencable URLs or alternatively to different fragments of the same dereferencable URL as stated by the DataLink specification. -\subsection{VOEvent modernization}\lable{sec:voevent_he} +\subsection{VOEvent modernization}\label{sec:voevent_he} The modern GCN is exploring new standards more compatible with modern event brokers such as Kafka. This includes defining metadata serialization in formats such as JSON and new schema\footnote{\url{https://gcn.nasa.gov/docs/notices/schema}} to describe how to use them. This interest group should actively engage the growing community of projects publishing transient alerts through systems that have outgrown the VOEvent standard. diff --git a/ivoatex b/ivoatex deleted file mode 160000 index c838d62..0000000 --- a/ivoatex +++ /dev/null @@ -1 +0,0 @@ -Subproject commit c838d628821446b112df6a92f5010423290db8e2