Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kill ns loop if it is running more than 10 minutes #1340

Merged
merged 1 commit into from
Dec 28, 2019
Merged

kill ns loop if it is running more than 10 minutes #1340

merged 1 commit into from
Dec 28, 2019

Conversation

tzachi-dar
Copy link
Contributor

Today, oaps stopped uploading data to NS, therefor I thought that it stopped working.

A small debug showed that ns-loop stopped running, and specifically, the following process got stuck:
root 14207 14063 0 37727 38252 0 16:23 ? 00:00:25 node /usr/local/bin/oref0-get-ns-entries cgm/ns-glucose-1h-temp.json https://snir-dev.herokuapp.com 911eea185f5c758ac275687da8b7eab50ad720e3 1

It seems that on pi-zero this process might be running up to 6-7 minutes (I saw it once running for 20 minutes), so killing it after 10 minutes seems reasonable.

@sulkaharo
Copy link
Collaborator

That run time is weird, as the socket timeouts for the operations made by the script are not long enough to cause such long runtime. Typically the script exits in seconds. Do you have any additional information under which conditions this happens? (Also, the pr was probably intended to target dev?)

@scottleibrand scottleibrand changed the base branch from master to dev December 28, 2019 20:55
@tzachi-dar
Copy link
Contributor Author

Yes, the pr was made to be for dev, moving it there in a minute.

As for your second question, If you are talking about the hang for more than two hours, this happened on "normal" conditions that is pi zero running alone, getting it's internet connection from a phone (using wifi).

If you are asking why this sometimes takes 5-6 minutes, than I guess that this is more pi-zero processing time.
Here is an example of the log:

2019-12-27 13:11:07 openaps.ns-loop Starting oref0-ns-loop at Fri 27 Dec 2019 01:11:07 PM IST:
2019-12-27 13:13:00 openaps.ns-loop Connected to 192.168.44.1, testing for xDrip API availability
2019-12-27 13:13:04 openaps.ns-loop Load from local xDrip timed out, likely not connected to xDrip hotspot
2019-12-27 13:13:11 openaps.ns-loop Got CGM results from Nightscout
2019-12-27 13:13:50 openaps.ns-loop Found recent valid BG:
2019-12-27 13:13:50 openaps.ns-loop {"glucose":111,"dateString":"2019-12-27T11:11:44.695Z"}
2019-12-27 13:13:51 openaps.ns-loop {"sgv":111,"dateString":"2019-12-27T13:11:44.695+0200"}
2019-12-27 13:14:01 openaps.ns-loop Latest NS temptargets: (NOT VALID JSON: empty)
2019-12-27 13:14:01 openaps.ns-loop Merging local temptargets: (NOT VALID JSON: empty)
2019-12-27 13:14:01 openaps.ns-loop Temptargets merged: (NOT VALID JSON: empty)
2019-12-27 13:15:24 openaps.ns-loop Refreshed carbhistory; COB: 4
2019-12-27 13:15:25 openaps.ns-loop {"batteryVoltage":4080,"battery":94}
2019-12-27 13:16:06 openaps.ns-loop (NOT VALID JSON: empty)
2019-12-27 13:16:10 openaps.ns-loop Latest NS treatment: 2019-12-27T11:03:32.000Z
2019-12-27 13:17:59 openaps.ns-loop [{"duration":120,"raw_duration":{"timestamp":"2019-12-27T13:03:32+02:00","_type":"Te$
2019-12-27 13:18:00 openaps.ns-loop Completed oref0-ns-loop at Fri 27 Dec 2019 01:17:59 PM IST

So, in this case the loop took 7 minutes.
On first look this is more processing time, then network time.

@tzachi-dar
Copy link
Contributor Author

By the way, looking at the time it takes this loop to run, we might consider changing that to 15-20 minutes.
Here are some results:
Seems like we have 4 loops taking more than 10 minutes.

grep Starting ns-loop.log

Starting oref0-ns-loop at Thu 26 Dec 2019 02:50:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 02:58:05 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:05:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:08:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:09:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:10:11 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:18:05 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:25:11 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:32:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:39:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:43:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:48:05 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 03:55:10 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:01:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:04:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:13:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:23:07 PM IST: --- 10 minutes
Starting oref0-ns-loop at Thu 26 Dec 2019 04:26:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:27:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:34:11 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:37:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:43:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:46:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:52:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 04:55:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:00:11 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:02:09 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:03:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:08:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:15:09 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:25:10 PM IST: ---- 10 minutes
Starting oref0-ns-loop at Thu 26 Dec 2019 05:27:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:28:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:34:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:38:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:43:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:46:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:47:07 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 05:53:12 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:01:04 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:04:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:13:06 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:17:06 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:21:05 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:25:08 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:33:09 PM IST:
Starting oref0-ns-loop at Thu 26 Dec 2019 06:18:19 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 12:51:12 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 12:55:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 12:57:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:00:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:05:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:11:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:18:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:20:04 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:25:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:28:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:29:06 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:34:06 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:40:03 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:47:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:53:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 01:59:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:02:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:03:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:08:06 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:14:04 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:21:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:24:06 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:25:04 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:30:04 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:33:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:38:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:45:11 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:47:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:48:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:51:10 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:53:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 02:57:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:03:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:06:11 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:07:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:08:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:13:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:18:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:34:09 PM IST: --- 16 minutes
Starting oref0-ns-loop at Fri 27 Dec 2019 03:41:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:46:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:52:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 03:58:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:00:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:01:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:04:05 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:11:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:17:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:20:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:21:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:22:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:26:09 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:28:08 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:33:07 PM IST:
Starting oref0-ns-loop at Fri 27 Dec 2019 04:36:19 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 01:41:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 01:44:10 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 01:51:12 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 01:54:06 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 01:59:04 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:05:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:10:11 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:17:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:23:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:26:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:32:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:35:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:41:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:48:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 02:54:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:00:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:06:04 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:08:10 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:15:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:17:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:24:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:27:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:33:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:36:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:37:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:41:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:44:15 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:45:10 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:49:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 03:56:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 04:09:08 PM IST: --- 13 minutes
Starting oref0-ns-loop at Sat 28 Dec 2019 04:23:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:04:04 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:11:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:17:12 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:24:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:27:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:34:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:41:06 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:47:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:49:10 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:50:06 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:54:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 08:59:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:05:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:14:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:19:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:22:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:28:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:34:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:39:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:45:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:51:04 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 09:56:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:01:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:06:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:13:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:19:06 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:24:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:27:08 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:32:07 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:37:05 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:39:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:42:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:48:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 10:54:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 11:02:09 PM IST:
Starting oref0-ns-loop at Sat 28 Dec 2019 11:04:11 PM IST:

scottleibrand
scottleibrand approved these changes Dec 28, 2019
View reviewed changes
Copy link
Contributor

@scottleibrand scottleibrand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also saw the same and made a similar change on one of our rigs.

@scottleibrand scottleibrand merged commit 2808a14 into openaps:dev Dec 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottleibrand scottleibrand scottleibrand approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tzachi-dar @sulkaharo @scottleibrand