Provide GNSS vectors

[tonycanike]

File this in the Pipe Dreams drawer. :)

Subject of the issue

Provide GNSS vectors (from base to rover). A number of high-end survey-grade GNSS receivers provide GNSS vectors in addition to the coordinates of a recorded point, and corresponding data collectors collect this data. There appear to be a number of proprietary APIs and data formats between commercial data collectors and GNSS receivers to request and gather this data.

GNSS vectors can be used for high-precision processing of the survey data. An individual vector represents the line between the RTK base and the rover, and it includes error information. This data can be used, for example, in a least squares adjustment to apply statistical tests to the data and perform precision adjustments. Longer vectors (perhaps when using network bases) can be given less weight than shorter vectors. Vectors with better DOP, less noise, smaller sdevs, for example, can be given more weight than those with less-favorable characteristics.

The NGS proposed a non-proprietary format for this data:
https://geodesy.noaa.gov/data/formats/GVX/index.shtml

At least one data collector can use this vector data in real time to perform advanced averaging of the GNSS observations to determine the mark coordinates. Mark Silver says this functionality has been in SurvCE since 2013. The opening moments of his video describe its usefulness.

https://www.youtube.com/watch?v=umiRftGkd3A

I'm really not sure how the (proprietary?) interfaces between the GNSS receiver and the data collectors work; does the GNSS receiver continually transmit the vectors intermixed with the NMEA stream and the data collector grabs a vector (along with the NMEA position report) when the surveyor uses the data collector to measure a point?

There is a F9P message that provides base-rover vectors, apparently this is used when the base and rover are both fixed to a moving platform (drone) and intended to be used for heading determination. Is this the same vector that the NGA and Mark Silver are talking about?

I'm not even sure what I'm asking the RTK Firmware to do.

Well, thanks for reading this!

Expected behavior

System to provide GNSS vectors for processing by adjustment software.

[tonycanike]

Update: Yesterday and today I attended the NGS's "OPUS Project Manager Training". Some things I learned:

GNSS Vector processing is in OPUS Projects beta. The beta is available to any current OPUS Projects users. OPUS Projects beta is reading GNSS Vectors in the GVX file format.

Trimble, TopCon, and Leica current support the GVX file format. Carlson, iGage, Emlid, and Javad are reportedly working on it.

It's my understanding that the major GNSS survey receiver manufacturers will process, and have for some time now, GNSS vector data in a proprietary format using the proprietary software the manufacturers provide.

The value of GNSS Vectors is that it will allow RTK observations (local base or RTN) to be included and adjusted in least squares processing software. If the vectors are in proprietary format, s/w that reads that manufacture's format will be required. If the GNSS Vectors are in GVX format, the vector data aren't locked in the proprietary walled garden.

[tonycanike]

I found a forum post in which surveyors are discussing converting the format of GNSS vectors.
https://surveyorconnect.com/community/gnss-geodesy/converting-carlson-rw5-format-to-starnet/

It looks like the data in a GNSS Vector contains
From pointname
To pointname
The vector DX, DY, DZ (ECEF deltas I believe)
and the variance-covariance matrix. It's a symmetric 3x3 matrix, so there's 6 unique number in the matrix.

As best I can tell, here's a format recognized by Carlson and Star*Net (from the forum post)

Yes, it has all the essential info in the file. The Star*Net format is:

G0 'optional remarks
G1 from-to DX DY DZ
G2 xx yy zz
G3 xy xz yz

Specimen:

G0 'V261 Day025(3) 21:06 00008341.ssf
G1 363-362 190.482400 -30.037207 1.334055
G2 3.18378042786085E-007 1.89480531625124E-006 2.41912013830421E-006
G3 -1.50610449416695E-007 -1.20847501448019E-008 -1.54065969949142E-006

I looked at a vector file one of my Carlson software tools wrote, and here's some sample data from it. I note this sample data is a similar format, but it includes the antenna heights in the G0 remark/comment field.

My point numbers are 9001, LUDR, ECLG, AQCG, etc.

G0 '1: "9001"-"LUDR"{"from" : {"name" : "9001","antenna h/e/n" : [1.5359,0,0],"meta" : ""},"to" : {"name" : "LUDR","antenna h/e/n" : [1.4685,0,0],"meta" : ""},"solution" : {"quality" : "high","id" : 1,"svcount" : 5,"s0" : 1.3,"type" : "FixedL1st","reweightedobs" : 0,"s0_Obs" : 0.004,"PDOP" : 2.3,"ambiguityfix" : 99.7}}
G1 "9001"-"LUDR" 3716.5387 3143.5249 2861.9063
G2 0.0000000203 0.0000000700 0.0000000412
G3 -0.0000000171 0.0000000113 -0.0000000329
G0 '3: "9001"-"ECLG"{"from" : {"name" : "9001","antenna h/e/n" : [1.5359,0,0],"meta" : ""},"to" : {"name" : "ECLG","antenna h/e/n" : [1.639827,0,0],"meta" : ""},"solution" : {"quality" : "high","id" : 3,"svcount" : 4,"s0" : 1.8,"type" : "FixedL1st","reweightedobs" : 0,"s0_Obs" : 0.0055,"PDOP" : 7.9,"ambiguityfix" : 100}}
G1 "9001"-"ECLG" 1741.7201 1017.5938 714.1240
G2 0.0000000424 0.0000002697 0.0000001380
G3 -0.0000000372 0.0000000326 -0.0000001007
G0 '4: "9001"-"AQCG"{"from" : {"name" : "9001","antenna h/e/n" : [1.5359,0,0],"meta" : ""},"to" : {"name" : "AQCG","antenna h/e/n" : [0.7586,0,0],"meta" : ""},"solution" : {"quality" : "high","id" : 4,"svcount" : 4,"s0" : 2,"type" : "FixedL1st","reweightedobs" : 0,"s0_Obs" : 0.006,"PDOP" : 7.6,"ambiguityfix" : 99.5}}
G1 "9001"-"AQCG" 4740.8112 2716.0851 1909.0557
G2 0.0000007772 0.0000125193 0.0000074528
G3 -0.0000028118 0.0000021907 -0.0000094755

[tonycanike]

Here's a link to a Carlson manual describing the above format, or at least their variation of it.
https://surveyorconnect.com/wp-content/uploads/converted_files/385319=5561-Raw%20File%20Records_2%2050.pdf

SurvCE Version 2.50 Raw File records
1) Antenna Type: [RTK GPS Antenna+Receiver], RA0.0000m,SHMP0.0000m,L10.1135m,L20.0941m,--
APS-3
These are the 4 generic GPS antenna variables for Radius, Slant Height Measure Point, L1 offset,
L2 offset and the description
2) RTK Method: RTCM V3.0, Device: Internal GSM, Network: NTRIP RTCM3_MAX
This is your values used in Equip / GPS Rover / RTK tab
3) Entered HR: 6.5620, Vertical
This is the hand entered value typed in by the customer in Equip / GPS Rover / Receiver tab
along with the method selected {i.e. – Slant or Vertical}
4) HSDV:0.034, VSDV:0.075
This is a GPS variable similar to “HRMS / VRMS” for the Horizontal Standard Deviation and the
Vertical Standard Deviation
5) BP, PN733,LA30.160894090052,LN-97.471343999946,EL175.4530,AG2.000,PA0.114,--
The BP record is the Base Point record. The AG is the “Antenna to Ground” value and the PA is
the “Phase Center to Antenna” value. You can add them together to get the Phase Center to Ground
value. THE UNITS OF THE ELLIPSOID ELEVATION, PA AND AG ARE ALWAYS IN METERS.
6) GT, PN27, SW-522,ST-259200000,EW-522,ET-259200000
This is the GPS Time stamp for RW5 file if you have “Store GPS Accuracy” turned on.
GT - GPS time, PN point number, SW start week, ST Start time, EW - End week, ET - End time
GPS VECTOR RECORDS
You will get the rod height of the rover from the LS record prior to the vector records. The LS,HR value is
from phase center to the ground, we do not show the antenna offset in this record. WARNING, THE
UNITS CAN BE FEET OR METERS; YOU WILL HAVE TO LOOK AT THE MO RECORD (UN) TO TELL.
The vector information is in the G records:
G0 - Date, time, Base ID
G1 - Base point number, Rover point number, Delta X, Delta Y, Delta Z
G2 - Variance X, Variance Y, Variance Z
G3 - Covariance XY, Covariance XZ, Covariance YZ
The DX, DY and DZ values are phase center to phase center. ALL THE VALUES ARE ALWAYS IN
METERS.

Here is a sample of the top part of the RW5 Raw File from Version 2.50:
JB,NMTERRYHSE,DT01-25-2010,TM15:16:11
MO,AD0,UN2,SF1.00000000,EC0,EO0.0,AU0
--SurvCE Version 2.50
--CRD: Alphanumeric
--TX Central NAD83
--Equipment: APS-3
--Antenna Type: [RTK GPS
Antenna+Receiver],RA0.0000m,SHMP0.0000m,L10.1135m,L20.0941m,--APS-3
--Localization File: None
--Geoid Separation File: None
--GPS Scale: 1.00000000
--RTK Method: RTCM V3.0, Device: Internal GSM, Network: NTRIP RTCM3_MAX
BP,PN733,LA30.160894090052,LN-97.471343999946,EL175.4530,AG2.000,PA0.114,--
--Entered HR: 6.5620, Vertical
LS,HR6.9344
GPS,PNBWC1+A,LA30.241617091114,LN-97.441679812958,EL231.637722,--PK NAIL
--GS,PNBWC1+A,N 10120391.5553,E 3114671.1420,EL837.6091,--PK NAIL
G0,01/25/2010 20:53:02,(Average) - Base ID read at rover: 733
G1,BP733,PNBWC1+A,DX5692.192,DY6823.564,DZ12978.073
G2,VX0.00863666,VY0.02219784,VZ0.01231287
G3,XY0.00006689,XZ-0.00001147,YZ-0.00017120
--HSDV:0.034, VSDV:0.075, STATUS:FIXED, SATS:10, PDOP:1.773, HDOP:0.860,
VDOP:1.550
--DT01-25-2010
--TM14:57:08
GPS,PN3,LA42.214176546000,LN-71.095340522000,EL-8.227500,--base
--GS,PN3,N 5.4159,E 4.9849,EL10.0046,--base
--GT,PN3,SW-522,ST-259200000,EW-522,ET-259200000
--HRMS:0.013, VRMS:0.019, STATUS:FIXED, SATS:8, PDOP:1.800, HDOP:1.000,
VDOP:1.500

[gdt]

See also https://www.ngs.noaa.gov/data/formats/GVX/index.shtml

[gdt]

Sorry, saw that up top. Given SparkFun's open source alignment, ti seems the code should head for GVX and people can deal with GVX/proprietary converters.

[tonycanike]

Agreed about GVX and open formats. Hope more manufacturers adopt it. Perhaps the fact that NGS only accepts GVX vectors for OPUS Projects could drive adoption.

I wasn't clear; I put in the info on those proprietary formats as info about the data necessary to construct GNSS vectors. The examples were clearer to me than the NGS GVX spec.

Though that proprietary format is apparently the StarNet format, and StarNet appears to be widely used by the survey community for performing least squares adjustments of total station and GNSS vectors. It also appears Carlson has adopted the format, perhaps with variations, for GNSS vectors in their RW5 files (their data collectors 'electronic field book') and in their SurvNet files, SurvNet being Carlson's competitor to StarNet. Not advocating adoption this format, just sharing what I'm learning.

I've been looking at UBlox interface spec to see if it can provides this data. There's a message that provides base-rover relative position data, with variance-covariance info, but it appears to be in NED orientation and not ECEF.

[tonycanike]

Another tidbit of information, this time from the Carlson SurvPC version 7 release notes:

Vector records now include a G4 record to indicate final tilt offsets in metric. These values
can be added to the G1 record to correct a tilt adjusted APC to APC vector.

https://www.carlsonsw.com/api/wp-content/uploads/SurvPC-7.0-Release-Notes.pdf

[tonycanike]

The GPS vectors provided in OPUS extended solutions are in the "GLOBAL POSITIONING SYSTEM DATA TRANSFER FORMAT", aka the "G File" format.

https://www.ngs.noaa.gov/FGCS/BlueBook/pdf/Annex_N.pdf

[nseidle]

1. Is this request still valid? Because you've been adding to it for a few months now, I'll presume, yes it is.

2. Are you asking for the generation of a GVX file along side the standard UBX log file for any given work? Reading through the GVX format, I don't see any immediate show stoppers. I'm not sure we have all these datums available for logging, but most are.

3. Help me understand the work flow. Would the GVX file be updated with every time the user hits the 'Mark' button on the RTK setup? Or is it a constant stream of GNSS_VECTOR elements?

[tonycanike]

Yes, it's still valid. Though I don't have a conceptual design or process in mind. Got nuthin' to say to the rubber duck.

Would you be opposed to leaving this open while we figure things out?

The end goal is to get mark-to-mark GNSS vectors associated with survey points that can be incorporated into a least squares adjustment.

The vectors would be from the mark the base is over to the mark the rover is over. Surveyors commonly take redundant measurements and calculate a least squares adjustment (LSA) to determine best-fit coordinates for each mark (points) that is being surveyed. The LSA of the redundant measurements are a commonly accepted technique for a surveyor to check their work and calculate their accuracy. Surveyors need to meet certain accuracy standards and this is one way to determine if they met the standard.

It's unclear to me if we get one vector per rover mark coordinates (aka a point) collected, or multiple vectors (e.g. 1Hz for 3 minutes = 180 vectors) per point.

In a day of surveying someone might collect coordinates for 100s of marks (100s of points).

As far as I can tell, today approaches for this are proprietary. Trimble GNSS receivers send the data to Trimble data collection software which saves it in Trimble format files to be read by Trimble desktop software.

That all works well very well for Trimble users, but it doesn't help makers and open source developers and users. I believe we'd be plowing new ground here.

Your questions #2 and #3 are insightful. Those are the problems we need to solve. I don't believe many, if any, of us are hitting the mark button on the SFE RTK receivers to save a point. We're hitting the store/save point button in whichever app we're using on our device (SWMaps, QField, SurvPC, etc.)

I imagine in Trimble Land the Trimble data collector is receiving a stream of data from the GNSS Receiver, including GNSS Vector data, and the data collector s/w saves the vector information and the coordinates when user stores data for a point. GNSS surveyors generally collect data for 15 seconds or a couple minutes, so somehow that's reduced to one set of coordinates (averaging?) and one vector (least squares?) I imagine Trimble data collector also knows the height of the base over the mark and the height of the rover over the mark,, so it can calculate mark-to-mark vectors.

We're not in Trimble Land. How would a maker - open source person accomplish the same goal?

How to we get the small subset of vectors that correspond to the point we measured? Do we just save all the vectors--a days' worth--to a file on the SD care, and later use some TBD desktop software to extract out the few minutes of vectors that we need for our point? Some survey collectors (including SurvPC) save a file that specifies the time range data for a particular point was collected.

That is, if our data collector is averaging 180 sets of coordinates (say 3 minutes @ 1Hz) for a point, do we then later use desktop software to select those 180 vectors out of the hours-long files of vectors using time stamps? Throw a few out at the beginning and end to account for time sync issues? Then what do we do with, say, the remaining 160 vectors? LSA them into one vector?

How do we get mark-to-mark vectors from the base to the point we measured? The SFE RTK doesn't know the height above mark for the base or the rover. So I presume the vectors provided would be APC to APC. So would we need desktop s/w to use the heights above the mark, APC offsets, etc. to calculate mark-to-mark GNSS vectors?

I searched to see what Emlid does about GNSS Vectors. This was all I could find.
https://community.emlid.com/t/support-of-gvx-format-update/28674

From the NGS's GVX Page:

GVX elements include (but are not limited to) the following:
Mark-to-mark Earth-Centered, Earth-Fixed (ECEF) vector components
Variances and covariances of vector components
Reference frame information
Start and stop time of the observation
A-priori coordinates for the end points of each vector
Receiver and antenna types
RTK and real-time network (RTN) settings, if applicable
Quality control metadata (e.g., PDOP, number of satellites used, orbit type, etc.)

[PS - There's a time-sync-point-collection-with-data-stream-to-select-data-subset pattern here that is similar to the problem of post-processing Stop-and-Go / PPK data. The GNSS Receiver streams a set of raw GNSS observations in the UBX (and eventually RINEX) file. We'd need to use time-syncing to get a subset of the raw GNSS observations that correspond to the time we occupied the mark and collected the data in our data collector. Then we can post-process the raw obs against the base (eg using RTKPOST) and select the output coordinate positions that correspond to the time range. Emlid does have a solution for this; if you give their desktop s/w a CSV with point names and time ranges in it, it will process the Base and Rover raw observation files and determine positions (coordinates) for each of the named points. ]

[tonycanike]

Here's a Carlson video on what I want to be able to accomplish, which is using a least square adjustment (LSA) program on the GNSS vectors to improve accuracy. This video is specific to Carlson's SurvNet LSA software, but the first few minutes and the last few minutes give a good visualization of the general idea. One of the industry-leading LSA software packages is StarNet, and SurvNet uses similar file formats to StarNet.

https://www.youtube.com/watch?v=AY11_3zsRKk

The above video is #2 in a 3-video series. Here are links to the other videos in the series if you want a deep-dive.

#1/3: https://www.youtube.com/watch?v=KHqqI4WXnQE
#3/3: https://www.youtube.com/watch?v=uWBXHG8OqQk

These three videos show common use cases and demonstrate the value of using GNSS Vectors.