Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queue sending to a custom URL and send every x minutes #918

Closed
dshanske opened this issue Oct 3, 2021 · 8 comments
Closed

Queue sending to a custom URL and send every x minutes #918

dshanske opened this issue Oct 3, 2021 · 8 comments
Milestone

Comments

@dshanske
Copy link

dshanske commented Oct 3, 2021

The system already, from what I understand, will queue if offline. Instead of sending every point to a custom URL, queue points and send every X number of minutes. This is similar to the concept of the Auto-Send.

This would essentially be a way to record an entire journey, but not to be constantly sending it.

@qsysopr1
Copy link

qsysopr1 commented Oct 3, 2021

And this appears to be not working presently. Today, for instance, I had the app up on the screens with functioning GPS but data communication offline and no data reports were ever uploaded upon return to data communication.

@mendhak
Copy link
Owner

mendhak commented Oct 4, 2021

For my understanding of the goal - is it basically to have more reliable sending of Custom URL points?
Taking the auto send concept, you could have Custom URL logging enabled if you wanted, or Custom URL auto-send, or both enabled. Then every 60 minutes, or when you press stop (the auto send options), or when you use the upload menu, the Custom URL points are re-sent to the target server. This way, if some of the previously queued points got lost or never sent, at least the auto-send will re-send it to the server. Does that sound like the goal?

This is somewhat similar to how the OpenGTS feature operates right now. OpenGTS logging will queue it up right away. Then every 60 minutes, or when pressing stop, or when the user uses the 'upload' menu, the code reads the GPX file and reconstructs the OpenGTS URLs, then re-sends it to the server.

I could think of something similar to do with Custom URL 'auto send'. There are a few things different with a Custom URL 'auto send'.

First one is, on the target server, you would need to be able to handle 'duplicates'. For example, use the timestamp or epoch sent when you receive the URL. In other words on the server side the write operation needs to be idempotent.

Now the second and main one, it's the storage of the Custom URL data. Right now, the URL is constructed on t he fly, and queued up right away. There's a lot of types of data that could be sent via Custom URL.

Here's the list so far

%lat
%lon
%sat
%desc
%alt
%acc
%dir
%prov
%spd
%timestamp
%timeoffset
%time
%date
%starttimestamp
%batt
%aid
%ser
%act
%filename
%profile
%hdop
%vdop
%pdop
%dist

Unlike OpenGTS, it's not possible to reconstruct a Custom URL from a GPX file, because most of this data types shown above is simply not stored in GPX format. I would definitely need to store all these values in a separate data 'store'. So that when 60 minutes are up, or when user presses stop, or when the user chooses to send a file manually, these values are available.

So... thinking about this, what about CSV. The CSV file format is pretty much a data store. And its columns look very close to Custom URL already. Here are its columns

time,lat,lon,elevation,accuracy,bearing,speed,satellites,provider,hdop,vdop,pdop,geoidheight,ageofdgpsdata,dgpsid,activity,battery,annotation

What I could do is when auto-sending, just re-read the CSV file and reconstruct the URL and send its values out. I would need to add a few more columns to the existing CSV format, it should be possible.

That will mean, the third thing - if a user wants Custom URL auto-sending, I will need to force CSV file writing to also be enabled. And when they choose to 'upload' a file to Custom URL, I'd present the CSV files.

@dshanske
Copy link
Author

dshanske commented Oct 4, 2021

That's not a problem. It means a local backup that I could replay into the server

@qsysopr1
Copy link

qsysopr1 commented Oct 4, 2021

Agreed, the goal is more reliable sending of Custom URL points when the device is frequently offline but recording.
What you describe sounds reasonable with csv on the device and with the remote recipient responsible for handling duplicates.
I wonder why queuing up used to work more reliably in the past and now less so recently; before fdroid, I had sent my gpslogger in areas frequently out of data service, and it did an admirable job of reporting the route when it reached data service again.

@mendhak
Copy link
Owner

mendhak commented Oct 5, 2021

I think the unreliability is due to the increasing throttling that Android has been applying in each new version of the OS. Each new release there are additional restrictions on background processing, and location data. Both of which this app makes use of. Their aim is to improve battery life as well as unrestricted data siphoning, so they do this by throttling or restricting background tasks from running, which actually is understandable.

That in turn affects GPSLogger, because the background tasks used as part of the queueing and sending can be killed off and there isn't a recovery from that.

I'm also using quite an old job queueing library, and I ought to be moving onto the newer libraries for this, called JobScheduler, but even on that page are big red warnings:

Note: Beginning with API 30 (Build.VERSION_CODES.R), JobScheduler will throttle runaway applications. Calling schedule(android.app.job.JobInfo) and other such methods with very high frequency can have a high cost and so, to make sure the system doesn't get overwhelmed, JobScheduler will begin to throttle apps, regardless of target SDK version.

Note: Scheduling a job can have a high cost, even if it's rescheduling the same job and the job didn't execute, especially on platform versions before version Build.VERSION_CODES.Q. As such, the system may throttle calls to this API if calls are made too frequently in a short amount of time.

@mendhak
Copy link
Owner

mendhak commented Oct 5, 2021

I will spend some time exploring the CSV file aspect since there's more data to manage now. It's likely a library will be needed, probably the Apache Common CSV one.

@mendhak
Copy link
Owner

mendhak commented Oct 8, 2021

Custom URL can now be used as an auto sender, such as every 60 minutes or when stop is pressed.
It works by storing the values in the CSV file, so as a result the CSV file has gotten a bunch of new fields added.

You can also 'upload' the CSV file to the Custom URL sender, and the values will be read and converted to URLs and sent to the server. This should serve as a way to mitigate any lost Custom URLs or do away with offline queueing problems.

You will need to ensure that the Custom URL server can receive the same log lines multiple times.

image

I've put a test APK here: https://github.com/mendhak/gpslogger/releases/tag/v119-rc1
As usual, this APK is not compatible with the F-Droid version.
If you install this APK you will lose all GPSLogger data from the F-Droid version.

@mendhak
Copy link
Owner

mendhak commented Nov 11, 2021

v119 is now on F-Droid and in the releases. I'm going around and closing some issues.

@mendhak mendhak closed this as completed Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants