Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spurious timespans returned in response of the service #24

Open
petrrr opened this issue Nov 30, 2023 · 1 comment
Open

Spurious timespans returned in response of the service #24

petrrr opened this issue Nov 30, 2023 · 1 comment
Assignees
@jbienkowski

Comments

@petrrr
Copy link

petrrr commented Nov 30, 2023

Summary

We observe some unexpected content (i.e. not relevant to the request) in the response to queries to our availability service. From a few quick cross-checks to other endpoiNts this behaviour seems to be more general, indicating a problem with the software rather the of our specific node.

The first impression is that the query selection is not doing exactly what is expected or requested by the user.

Evidence, by one example

#Network Station Location Channel Quality SampleRate Earliest                    Latest                     
GU       BHB              HHE     D       100.0      2022-09-07T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HHE     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HHN     D       100.0      2022-09-07T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HHN     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HHZ     D       100.0      2022-09-07T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HHZ     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-07T00:00:00.000000Z 2022-09-07T07:13:12.410000Z
GU       BHB              HNE     D       200.0      2022-09-07T07:13:14.470000Z 2022-09-07T15:15:34.800000Z
GU       BHB              HNE     D       200.0      2022-09-07T15:15:36.860000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T05:33:03.170000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:03:36.570000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:23:12.830000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:23:25.190000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:28:56.850000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T23:01:07.770000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-07T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:32:48.600000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:33:03.020000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:33:11.260000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:22:58.260000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:23:27.100000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:27:24.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:27:30.180000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T12:08:13.290000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-07T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T05:32:59.855000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:22:59.215000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:27:24.955000Z 2022-09-08T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:29:12.075000Z 2022-09-08T00:00:00.000000Z

Preliminary analisis

There are actually two different cases of lines which could be considered useless:

  • Zero length time spans
  • invalid timespans where Earliest > Latest (negative length ?)

A different view of the same (last value is the segment length):

GU.BHB..HHE        2022-09-07T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z   86400.0000
GU.BHB..HHE        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HHN        2022-09-07T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z   86400.0000
GU.BHB..HHN        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HHZ        2022-09-07T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z   86400.0000
GU.BHB..HHZ        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNE        2022-09-07T00:00:00.000000Z .. 2022-09-07T07:13:12.410000Z   25992.4100
GU.BHB..HNE        2022-09-07T07:13:14.470000Z .. 2022-09-07T15:15:34.800000Z   28940.3300
GU.BHB..HNE        2022-09-07T15:15:36.860000Z .. 2022-09-08T00:00:00.000000Z   31463.1400
GU.BHB..HNE        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNE        2022-09-08T05:33:03.170000Z .. 2022-09-08T00:00:00.000000Z  -19983.1700
GU.BHB..HNE        2022-09-08T06:03:36.570000Z .. 2022-09-08T00:00:00.000000Z  -21816.5700
GU.BHB..HNE        2022-09-08T06:23:12.830000Z .. 2022-09-08T00:00:00.000000Z  -22992.8300
GU.BHB..HNE        2022-09-08T06:23:25.190000Z .. 2022-09-08T00:00:00.000000Z  -23005.1900
GU.BHB..HNE        2022-09-08T06:28:56.850000Z .. 2022-09-08T00:00:00.000000Z  -23336.8500
GU.BHB..HNE        2022-09-08T23:01:07.770000Z .. 2022-09-08T00:00:00.000000Z  -82867.7700
GU.BHB..HNN        2022-09-07T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z   86400.0000
GU.BHB..HNN        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNN        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNN        2022-09-08T05:32:48.600000Z .. 2022-09-08T00:00:00.000000Z  -19968.6000
GU.BHB..HNN        2022-09-08T05:33:03.020000Z .. 2022-09-08T00:00:00.000000Z  -19983.0200
GU.BHB..HNN        2022-09-08T05:33:11.260000Z .. 2022-09-08T00:00:00.000000Z  -19991.2600
GU.BHB..HNN        2022-09-08T06:22:58.260000Z .. 2022-09-08T00:00:00.000000Z  -22978.2600
GU.BHB..HNN        2022-09-08T06:23:27.100000Z .. 2022-09-08T00:00:00.000000Z  -23007.1000
GU.BHB..HNN        2022-09-08T06:27:24.000000Z .. 2022-09-08T00:00:00.000000Z  -23244.0000
GU.BHB..HNN        2022-09-08T06:27:30.180000Z .. 2022-09-08T00:00:00.000000Z  -23250.1800
GU.BHB..HNN        2022-09-08T12:08:13.290000Z .. 2022-09-08T00:00:00.000000Z  -43693.2900
GU.BHB..HNZ        2022-09-07T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z   86400.0000
GU.BHB..HNZ        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNZ        2022-09-08T00:00:00.000000Z .. 2022-09-08T00:00:00.000000Z       0.0000
GU.BHB..HNZ        2022-09-08T05:32:59.855000Z .. 2022-09-08T00:00:00.000000Z  -19979.8550
GU.BHB..HNZ        2022-09-08T06:22:59.215000Z .. 2022-09-08T00:00:00.000000Z  -22979.2150
GU.BHB..HNZ        2022-09-08T06:27:24.955000Z .. 2022-09-08T00:00:00.000000Z  -23244.9550
GU.BHB..HNZ        2022-09-08T06:29:12.075000Z .. 2022-09-08T00:00:00.000000Z  -23352.0750

Examing the following day's response or looking into WFcatalog, the original source of availability, these "negative timespan" entries are generated from timespans found in this following day.

From the WFcatalog, output of the relevant information:

GU.BHB..HHN, gaps: 0 overlap: 1 cseg: False
GU.BHB..HNN, gaps: 8 overlap: 1 cseg: True
  10
  2022-09-08T00:00:00.000Z .. 2022-09-08T00:00:00.360Z        0.3600  200       72 
  2022-09-08T05:32:48.600Z .. 2022-09-08T05:32:50.660Z        2.0600  200      412 
  2022-09-08T06:27:24.000Z .. 2022-09-08T06:27:28.120Z        4.1200  200      824 
  2022-09-08T05:33:03.020Z .. 2022-09-08T05:33:09.200Z        6.1800  200     1236 
  2022-09-08T06:22:58.260Z .. 2022-09-08T06:23:18.860Z       20.6000  200     4120 
  2022-09-08T06:23:27.100Z .. 2022-09-08T06:24:47.440Z       80.3400  200    16068 
  2022-09-08T05:33:11.260Z .. 2022-09-08T05:56:44.420Z     1413.1600  200   282632 
  2022-09-08T00:00:00.000Z .. 2022-09-08T05:31:57.100Z    19917.1000  200  3983420 
  2022-09-08T06:27:30.180Z .. 2022-09-08T12:08:11.230Z    20441.0500  200  4088210 
  2022-09-08T12:08:13.290Z .. 2022-09-09T00:00:00.000Z    42706.7100  200  8541342 
GU.BHB..HHE, gaps: 0 overlap: 1 cseg: False
GU.BHB..HNZ, gaps: 4 overlap: 1 cseg: True
  6
  2022-09-08T00:00:00.000Z .. 2022-09-08T00:00:02.745Z        2.7450  200      549 
  2022-09-08T06:27:24.955Z .. 2022-09-08T06:29:10.015Z      105.0600  200    21012 
  2022-09-08T06:22:59.215Z .. 2022-09-08T06:24:48.395Z      109.1800  200    21836 
  2022-09-08T05:32:59.855Z .. 2022-09-08T05:56:45.375Z     1425.5200  200   285104 
  2022-09-08T00:00:00.000Z .. 2022-09-08T05:32:57.795Z    19977.7950  200  3995559 
  2022-09-08T06:29:12.075Z .. 2022-09-09T00:00:00.000Z    63047.9250  200 12609585 
GU.BHB..HNE, gaps: 6 overlap: 0 cseg: True
  7
  2022-09-08T06:23:12.830Z .. 2022-09-08T06:23:19.010Z        6.1800  200     1236 
  2022-09-08T06:23:25.190Z .. 2022-09-08T06:28:54.790Z      329.6000  200    65920 
  2022-09-08T06:03:36.570Z .. 2022-09-08T06:23:08.710Z     1172.1400  200   234428 
  2022-09-08T05:33:03.170Z .. 2022-09-08T05:56:44.570Z     1421.4000  200   284280 
  2022-09-08T23:01:07.770Z .. 2022-09-09T00:00:00.000Z     3532.2300  200   706446 
  2022-09-08T00:00:00.000Z .. 2022-09-08T05:32:56.990Z    19976.9900  200  3995398 
  2022-09-08T06:28:56.850Z .. 2022-09-08T23:01:01.590Z    59524.7400  200 11904948 
GU.BHB..HHZ, gaps: 0 overlap: 0 cseg: False

#Network Station Location Channel Quality SampleRate Earliest                    Latest                     
GU       BHB              HHE     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HHE     D       100.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HHN     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HHN     D       100.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HHZ     D       100.0      2022-09-08T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HHZ     D       100.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T05:32:56.990000Z
GU       BHB              HNE     D       200.0      2022-09-08T05:33:03.170000Z 2022-09-08T05:56:44.570000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:03:36.570000Z 2022-09-08T06:23:08.710000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:23:12.830000Z 2022-09-08T06:23:19.010000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:23:25.190000Z 2022-09-08T06:28:54.790000Z
GU       BHB              HNE     D       200.0      2022-09-08T06:28:56.850000Z 2022-09-08T23:01:01.590000Z
GU       BHB              HNE     D       200.0      2022-09-08T23:01:07.770000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNE     D       200.0      2022-09-09T21:26:19.850000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:00.360000Z
GU       BHB              HNN     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T05:31:57.100000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:32:48.600000Z 2022-09-08T05:32:50.660000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:33:03.020000Z 2022-09-08T05:33:09.200000Z
GU       BHB              HNN     D       200.0      2022-09-08T05:33:11.260000Z 2022-09-08T05:56:44.420000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:22:58.260000Z 2022-09-08T06:23:18.860000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:23:27.100000Z 2022-09-08T06:24:47.440000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:27:24.000000Z 2022-09-08T06:27:28.120000Z
GU       BHB              HNN     D       200.0      2022-09-08T06:27:30.180000Z 2022-09-08T12:08:11.230000Z
GU       BHB              HNN     D       200.0      2022-09-08T12:08:13.290000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-09T21:26:20.900000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNN     D       200.0      2022-09-09T23:47:15.120000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T00:00:02.745000Z
GU       BHB              HNZ     D       200.0      2022-09-08T00:00:00.000000Z 2022-09-08T05:32:57.795000Z
GU       BHB              HNZ     D       200.0      2022-09-08T05:32:59.855000Z 2022-09-08T05:56:45.375000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:22:59.215000Z 2022-09-08T06:24:48.395000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:27:24.955000Z 2022-09-08T06:29:10.015000Z
GU       BHB              HNZ     D       200.0      2022-09-08T06:29:12.075000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T00:00:00.000000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T20:16:14.665000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T21:15:26.105000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T21:26:19.105000Z 2022-09-09T00:00:00.000000Z
GU       BHB              HNZ     D       200.0      2022-09-09T23:47:15.335000Z 2022-09-09T00:00:00.000000Z

Possible solution

Correct query in order to not return timespans which are not relevant to the requested time span, avoid also the non informative zero lenth timespans.
@sheimers
Copy link
Contributor

This issue is still open, I did a git pull and recreated the docker image, but still get wrong timespans.
eg:
https://eida.ethz.ch/fdsnws/availability/1/query?station=VDR&starttime=2024-08-22T12:50:00&endtime=2024-08-26T11:46:00&channel=HGE

#Network Station Location Channel Quality SampleRate Earliest                    Latest                     
CH       VDR              HGE     D       200.0      2024-08-22T12:50:00.000000Z 2024-08-22T12:46:04.915000Z
CH       VDR              HGE     D       200.0      2024-08-26T10:26:34.915000Z 2024-08-26T11:46:00.000000Z

It has the latest before the earliest.

"Earliest" on the first line is wrong, it is the requested start time, but there is no data at that time.
"Latest" is also wrong, it is the correct latest data available for that day, but ends before the requested start time.

Here is what is really available in miniseed:

sysop@eida:/rz_nas/miniseed/2024/CH/VDR/HGE.D$ qmerge -D m -n CH.VDR..HGE.D.2024.235
VDR.CH.HGE. rate=200 (2024.08.22 00:00:00.3850 to 2024.08.22 12:46:04.9150) : 9192906 points, 0.0 msec correction, (min,max,max_step = 0.0,0.0,0.0 msec)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbienkowski jbienkowski

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@petrrr @jbienkowski @sheimers