OS X, podman machine time stops sometime

[dm3ch]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Date output is wrong for both containers and podman machine itself on OS X.

Steps to reproduce the issue:
I'm not sure 100% in my reproduction guide

1. Install podman on OSX
2. Create podman machine
3. Wait couple of days
4. Get date from podman machine

Describe the results you received:
OS X date:

$ date
Sun Sep 12 15:55:30 MSK 202

Podman container and podman machine ssh date:

$ date
Fri Sep 10 19:47:08 UTC 2021

I have ran same command again and time was completely the same.
After stoping and starting machine again time started to go.

Describe the results you expected:
Date inside podman machine should be right

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 22:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 23:46:36 2021
OS/Arch:      linux/amd64

OS X info:

[afbjorklund]

I think KVM does this (ntp) in "hardware", but it seems to be missing from HVF ?

Might have to run an ntpd in the VM, before it is supported by the virtualization

EDIT: I think I got RTC and NTP confused in my head there.

[    0.249404] PM: RTC time: 17:44:05, date: 2021-09-13
[    0.769992] rtc_cmos 00:00: RTC can wake from S4
[    0.773422] rtc_cmos 00:00: registered as rtc0
[    0.774279] rtc_cmos 00:00: setting system clock to 2021-09-13T17:44:06 UTC (1631555046)
[    0.775629] rtc_cmos 00:00: alarms up to one day, y3k, 114 bytes nvram, hpet irqs

[guillaumerose]

There is a long blog post about this on the Docker blog.
https://www.docker.com/blog/addressing-time-drift-in-docker-desktop-for-mac/

Docker Desktop runs an embedded NTP server on the host. It takes his source on the system time. A NTP client in the VM keeps it in sync.

[dm3ch]

JFYI: Maybe my case is connected with that, but in my case it wasn't a slowdown because time completely stoped.

But it looks like NTP hack could workaround this particular issue too

[rhatdan]

@dustymabe Is this something we should turn on in Fedora CoreOS?

[rhatdan]

@ashley-cui PTAL

[dustymabe]

hmm.. NTP should be utilized on Fedora CoreOS by default

What's the output of the following commands run on the Fedora CoreOS host?

• timedatectl
• systemctl status chronyd
• sudo journalctl -b 0 -u chronyd

[afbjorklund]

My "podman machine" CoreOS seems synced to 2.fedora.pool.ntp.org

[guillaumerose]

I think podman machine should be self-contained and not require an Internet access. What if you code in the plane? The time should not drift.

[guillaumerose]

This situation can also happen after suspend/resume of the system. See a solution here: https://github.com/linuxkit/linuxkit/tree/master/pkg/host-timesync-daemon

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@dm3ch @baude @ashley-cui Is this still a bug?

[ashley-cui]

I think so

[rhatdan]

Should we grab the code from linuxkit/linuxkit@d24d0bd and add it to podman system service. So that if this device exists we could grab the latest time and set it when podman service starts up.

[rhatdan]

@guillaumerose @dustymabe WDYT?

[vrothberg]

@guillaumerose @dustymabe friendly ping

[dustymabe]

I'm no expert here. So mac/windows don't have a good way to keep VM clocks in sync with the host clock but the code in linuxkit/linuxkit@d24d0bd knows how to extract that information from the hypervisor and apply it to the VM?

If there is a way to extract the host time from the guest I'm surprised linux (or some builtin userspace daemon) doesn't already know how to get that information. Is there any open issues against some builtin linux components that discuss this issue?

[fingon]

This is still an issue, an example from a podman machine started early yesterday:

[core@localhost ~]$ timedatectl
               Local time: Tue 2021-11-30 19:48:01 UTC
           Universal time: Tue 2021-11-30 19:48:01 UTC
                 RTC time: Wed 2021-12-01 03:58:20
                Time zone: UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
..
mstenber@kobuta ~>TZ=UTC date
Wed Dec  1 05:38:01 UTC 2021

Funnily enough even the 'RTC' is lagging but not as much as the system time. systemd's ntp thing is clearly not working, I'm not sure why (it is supposed to have sane defaults, and running ntpdate inside container works, but cannot set time due to permissions I guess).

NOTE: The local time roughly correlates to not having moved when machine was suspended; RTC time lag I have no idea, I guess it just doesn't get updated from the host.

After sudo reboot it is fine, and I presume it will be too until if I suspend it again:

[core@localhost ~]$ timedatectl
               Local time: Wed 2021-12-01 05:48:52 UTC
           Universal time: Wed 2021-12-01 05:48:52 UTC
                 RTC time: Wed 2021-12-01 05:48:53
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

[rhatdan]

Run it privileged, and it should work.

[acdha]

I just ran into this and strongly suspect it's related to system sleep events, which should be easier to trigger at wake in the VM, since the clock hardware on the host has very low drift and I saw a ~3.5 hour lag in the last 6 wall-clock hours.

One factor which is almost certainly exacerbating this: I'm on a network which firewalls arbitrary NTP and podman does not pass through the host's NTP /etc/ntp.conf configuration. I'm not sure it'd be worth spending time on that rather than improving synchronization from the host.

[fingon]

Run it privileged, and it should work.

How? At least with podman machine, it does not apparently help.

mstenber@kobuta ~>podman run --privileged -it fedora:latest
...
[root@af04e3c8dbd3 /]# sudo dnf install -y ntpsec
...
[root@af04e3c8dbd3 /]# ntpdate 0.pool.ntp.org
{"time":"2021-12-02T09:18:19.836900+0000","offset":-0.000026,"precision":0.009240,"host":"0.pool.ntp.org","ip":"95.216.150.202","stratum":2,"leap":"no-leap","adjusted":false}
CLOCK: adj_systime: Operation not permitted
[root@af04e3c8dbd3 /]# 

[rhatdan]

It does require a rootfull containers. Rootless users are not allowed to adjust machine time.

[konstruktoid]

Is there anyway to easily configure the VM?
No real need to install additional services as one possible issue is to add NTP servers to the systemd-timesyncd service.
Will report back after the weekend if this hasn't worked.

[core@localhost ~]$ sudo sed -i 's/#NTP=.*/NTP=0.fedora.pool.ntp.org 1.fedora.pool.ntp.org/g' /etc/systemd/timesyncd.conf 
[core@localhost ~]$ grep -v '^#' /etc/systemd/timesyncd.conf 

[Time]
NTP=0.fedora.pool.ntp.org 1.fedora.pool.ntp.org
[core@localhost ~]$ sudo systemctl restart systemd-timesyncd.service 

This of course won't help if there's firewalls blocking stuff, as in #11541 (comment)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[xRomZak]

Got this issue with Minio container (after Mac sleep/awaik).
Minio rejects requests when there is a time diff between client and server.

ErrorResponse(
  code = RequestTimeTooSkewed,
  message = The difference between the request time and the server's time is too large.
)

[stevenharrisBG]

There is a long blog post about this on the Docker blog. https://www.docker.com/blog/addressing-time-drift-in-docker-desktop-for-mac/

Docker Desktop runs an embedded NTP server on the host. It takes his source on the system time. A NTP client in the VM keeps it in sync.

I'm not certain these are the same issues - though they could be, given other variables.

I did not have time drift in my postgres containers using Docker. Using rootless podman, it is a very serious problem related to power saving and sleep, as far as I can tell.

[jamesmortensen]

Here's another method I came across to reset the clock manually. I wasn't able to use one of the other methods, so I did this. I stopped the ntp service (which didn't seem to be working anyway) and then set the time manually:

# SSH into the VM
$ podman machine ssh

# From within the VM:
$ sudo timedatectl set-ntp false
$ sudo timedatectl set-time "10:02”
$ exit

[jordansissel]

I ran into this and used this as a workaround:

podman machine ssh 'sudo hwclock -s'

[sc68cal]

I got hit with this recently when trying to push a container to an internal registry and SSL failed because the clock in the podman VM on MacOS was a couple weeks behind.

Rebooting it fixed the issue but we should try and get the VM to stay in sync timewise

[crahan]

I installed sleepwatcher via brew and then added the following to my ~/.wakeup script to update the date and time whenever my Mac wakes up:

#!/bin/bash

# Fix Podman Machine's time drift on wake
if [[ $(/usr/local/bin/podman machine info) ]]; then
    the_date=$(date +'%Y-%m-%dT%H:%M:%S')
    /usr/local/bin/podman machine ssh sudo date --set $the_date
    echo "Updated Podman Machine date: $the_date" >> ~/podman_debug.log
fi

So far, so good (until a long term fix becomes available).

[simon-friedberger]

I edited /etc/chrony.conf and added maxchange 100000 1 2 which lets chrony fix the time when it is very off. (Disclaimer: Not properly researched. Just seems to work for now.)

[fnkr]

makestep seems to fit this use-case exactly:

in some cases (e.g. a computer without an RTC or virtual machine which can be suspended and resumed with an incorrect time) it might be necessary to allow the step on any clock update. The example above would change to

makestep 1 -1

Source

To update the default Podman-Machine:

podman machine ssh --username root -- sed -i 's/^makestep\ .*$/makestep\ 1\ -1/' /etc/chrony.conf
podman machine ssh --username root -- systemctl restart chronyd

[xordspar0]

Makestep does indeed seem like the right solution. It looks like makestep 1 -1 was discussed in a Fedora CoreOS bug and the conclusion was that it should be the default.

I'm not sure why that doesn't show up in our images. Oh, I see. The change spawned by that discussion is only for cloud VM images. Either we should make Podman do the same for its image or we should bring this up in the Fedora CoreOS project.

[stephennimmo]

This is still an issue. I ran into it as I was using Keycloak as a Quarkus devservices container and the tokens issued were using the VM time and therefore already expired upon issuance.

[mrkvh]

Same issue here, noticed today after my mac hibernated over the weekend. Nothing helped except restarting the machine.

[xordspar0]

Either we should make Podman do the same for its image or we should bring this up in the Fedora CoreOS project.

I asked Fedora CoreOS if it would make sense to use makestep 1 -1 as the default, but it doesn't appear to be a good default for all qemu VMs. In that case I propose that we add this configuration to the default Podman machine image.

[baude]

it could easily be added to the ignition file that is used to configure the coreos instance.

[xordspar0]

it could easily be added to the ignition file that is used to configure the coreos instance.

Ok, I wondered if Podman might already had its own Ignition configuration. I found it and did just that in #17661.

[jgresham]

it could easily be added to the ignition file that is used to configure the coreos instance.

Ok, I wondered if Podman might already had its own Ignition configuration. I found it and did just that in #17661.

Thanks for the PR. I have been experiencing this issue numerous times and this seems like a good fix! Hoping it goes thru soon.