update auth process and other improvements

[sedy89]

Carelink:

• 2nd Gen login and reauth process, based on @ondrej1024's python library
  • requires inital login data file from external script
• updated devcontainer setup
• implemented scan interval parameter to integration setup page

Nightscout Uploader:

• resuses customer timezone

@yo-han those changes should address most of the open Issues. I am currently testing the changes and it is working as expected so far. Another beta release would be very helpful to figure out remaining issues

[yo-han]

@sedy89 Nice, I merged the pr and created a new beta release.

[Trenar]

Hi @sedy89 , thanks for your hard work.
I just installed this rc and want to share my findings:

1. The python script to get the logindata did not work out-of-the-box, I had to change some things according to this comment. I'm running on Ubuntu 23.10, so maybe it's not worth changing anything for most of the users.
2. The timezone thing is still not resolved. In nightscout, the data is still displayed as "9 hours ago". I also tried changing the timezone in the nightscout-"profile editor", but that did not change anything unfortunately (does it have any effect at all?).
   
   However, checking the NS logs, I see that an "utcOffset" is correctly supplied (I picked an exemplary meal entry here)

2024-02-22T15:08:01.409601036Z got data-received event, requesting reload
2024-02-22T15:08:01.409634507Z REST API treatment created [
2024-02-22T15:08:01.409639774Z   {
2024-02-22T15:08:01.409643606Z     device: 'Home Assistant Carelink',
2024-02-22T15:08:01.409647281Z     timestamp: 1708580907000,
2024-02-22T15:08:01.409650927Z     enteredBy: 'Home Assistant Carelink',
2024-02-22T15:08:01.409654782Z     created_at: '2024-02-22T05:48:27.000Z',
2024-02-22T15:08:01.409658618Z     eventType: 'Correction Bolus',
2024-02-22T15:08:01.409662266Z     insulin: 0.3,
2024-02-22T15:08:01.409665789Z     utcOffset: -480
2024-02-22T15:08:01.409669184Z   }
2024-02-22T15:08:01.409672633Z ]

The logging timestamp (15:08) ist 14:08 UTC+1 (server location), corresponding to 7:08 UTC-8 (client location).

For a glucose entry however, something different is happening:

2024-02-22T15:14:50.176529277Z DEBUG nightscout-connect data-loaded {
2024-02-22T15:14:50.176560289Z   entries: 2024-02-22T06:13:00.000Z,
2024-02-22T15:14:50.176563633Z   sgvs: {
2024-02-22T15:14:50.176566204Z     _id: '65d764caa0a86f8120f530ba',
2024-02-22T15:14:50.176568737Z     mgdl: 137,
2024-02-22T15:14:50.176571215Z     mills: 1708582380000,
2024-02-22T15:14:50.176573713Z     device: 'Home Assistant Carelink',
2024-02-22T15:14:50.176576189Z     direction: 'FortyFiveDown',
2024-02-22T15:14:50.176578718Z     filtered: undefined,
2024-02-22T15:14:50.176581281Z     unfiltered: undefined,
2024-02-22T15:14:50.176583805Z     noise: 1,
2024-02-22T15:14:50.176586254Z     rssi: undefined,
2024-02-22T15:14:50.176602587Z     type: 'sgv',
2024-02-22T15:14:50.176605002Z     scaled: 137
2024-02-22T15:14:50.176607451Z   },
2024-02-22T15:14:50.176609823Z   treatments: 2024-02-22T06:13:35.000Z,
2024-02-22T15:14:50.176612331Z   devicestatus: 2024-02-22T15:14:16.540Z,
2024-02-22T15:14:50.176614827Z   profile: undefined
2024-02-22T15:14:50.176633426Z }

here, the devicestatus and treatments lines have an offset of ~ 9 hrs.

Please let me know, if you need more / additional information.

Thanks again :)

[sedy89]

Hi @Trenar ,
thank you for testing and reporting your findings.
I updated my branch.
Could you update your files manually and test again?
The changes are quite small -> changes

Once you confirm that it is working, I will create another PR with the fix.
Thanks

edit: maybe it will work for you, but I see now a shifting regarding my timezone.. still need to check.

[Trenar]

Hi @sedy89
wow, you're fast. Unfortunately, it did not solve my problem, but changed the offset from "9 hours ago" to "8 hours ago".

In the NS-logs, let's quickly review this entry (same as in my post above):

got data-received event, requesting reload
REST API treatment created [
  {
    device: 'Home Assistant Carelink',
    timestamp: 1708580907000,
    enteredBy: 'Home Assistant Carelink',
    created_at: '2024-02-22T05:48:27.000Z',
    eventType: 'Correction Bolus',
    insulin: 0.3,
    utcOffset: -480
  }
]

I think the problem is the "created_at" timestamp. It is not the UTC time stamp (as indicated by the "Z"), but instead it's the local pump / uploader time, which already includes the tz-offset (8hrs). I assume, that the utcOffset variable is then applied again, yielding a double-correction of the time zone. Could that be possible?

Thanks,
Trenar

[Trenar]

Addendum: I checked the timestamp value, it translates to the same values as created_at, i.e. including the tz-offset, and not in UTC.

[Trenar]

Regarding the python-script, it still does not launch the Firefox window with the error message "Message: binary is not a Firefox executable". It really needs the path to the executables. However, as this is system-specific, I understand that you don't want to integrate it here.

[sedy89]

@Trenar please check again with the lastet changes. The login script should have linux support (if your path is the same link in the stackoverflow thread). And the ns uploader might use the correct timezone now. At least for my timezone it is working again. Best of luck

[Trenar]

@sedy89 now the reportings are again 9 hours ago, i.e. the correct offset between client/uploader and server. I left a comment on the commit in your branch with my thoughts: I assume NS expects UTC timestamps, but we actually provide local timestamps. Is that possible?

EDIT:

Now it got very confusing.. SGV and treatments now have a different offset: while SGV has an offset of -8hrs (UTC vs local client time), the treatments have an offset of -9hrs (server vs local client time). Not sure why this happens...

[Trenar]

@sedy89 I think I worked it out.. Hope that it also works on your side :-)

I started with your branch and changed the __getDataStringFromIso function to this:

    def __getDataStringFromIso(self, time, tz):
        dt = datetime.fromisoformat(time.replace(".000-00:00", ""))
        dt = dt.replace(tzinfo=tz)
        dt = dt.astimezone(tz)
        timestamp = dt.timestamp()
        date = int(timestamp * 1000)
        date_string = dt.isoformat()
        return date, date_string

What it now does is creating a datetime object from the string (as before). Note, that here the string is in "local time", while the datetime object intrinsically assumes UTC. Therefore, the tzinfo is replace'd to represent the time in the correct timezone. The replace command does not apply the hour-offset to the time itself, but only adds the tzinfo. That does the trick, hence dt.timestamp() is now a correct unix timestamp referenced to UTC.

Also the issue that SGV and treatments have a relative offset is gone now.

[sedy89]

Have you seen the PR I created in your own carelink fork ? :)

[Trenar]

Now I did 😄 I think we basically found the very same thing. Which one do you want me to test now, the PR in my fork, or the branch in yours?

[sedy89]

I already tested your's and it is wokring 👍
I will create another PR for Yo-han with the updated script and the update in the ns uploader.
Thank you for your help and testing

edit: btw.: you can enable the Pump plugin in your Nightscout settings to see also the active insulin.. if you haven't. I just did not see it in your screenshot. It will show the active insulin instead of the remaining units of the reservoir

[Trenar]

Perfect, thanks a lot :)