[Bookworm] Timezone setting is not propagated to containers #20925

vivekrnv · 2024-11-26T21:45:26Z

Description

Timezone setting is expected to propagate to the containers. But after bookworm upgrade in SONiC, it is broken

202311

Set a new timezone: config clock timezone Asia/Pyongyang
Updated on host immediately:

timedatectl show
Timezone=Asia/Pyongyang
TimeUSec=Wed 2024-11-27 06:17:03 KST

config save & config reload for the change to be updated in the container.

docker exec eventd zdump /etc/localtime
/etc/localtime  Wed Nov 27 06:20:09 2024 KST

docker exec eventd cat /etc/timezone
Asia/Pyongyang

docker exec eventd date
Wed Nov 27 06:20:39 KST 2024

202405 and above:

config clock timezone Asia/Pyongyang

timedatectl show
Timezone=Asia/Pyongyang
TimeUSec=Wed 2024-11-27 06:37:01 KST

zdump /etc/localtime
/etc/localtime  Wed Nov 27 06:37:15 2024 KST

After config reload, containers are expected to have timezone change but it is not seen

docker exec eventd zdump /etc/localtime
/etc/localtime  Tue Nov 26 21:39:00 2024 UTC

docker exec eventd date
Tue Nov 26 21:39:09 UTC 2024

docker exec eventd cat /etc/timezone
Etc/UTC

cat /etc/timezone
Etc/UTC (Even on host, this file is not updated)

Triage

Issue lies in timedatectl command not updating /etc/timezone in systemd version used on bookworm
Ref: https://unix.stackexchange.com/questions/451709/timedatectl-set-timezone-doesnt-update-etc-timezone

/etc/timezone appears to be mostly Debian-specific (including derivatives). On Debian systems, timedatectl set-timezone also updates /etc/timezone, in version of systemd older than 252.6-1 (so until Debian 11 included).

This PR #14000 (Which fixes the timesync issue between host and container) uses /etc/timezone as reference to update the timezone in container.

The text was updated successfully, but these errors were encountered:

vivekrnv · 2024-11-27T16:57:39Z

@nmoray can the logic be updated instead to use /etc/localtime directly from host instead of using /etc/timezone

nmoray · 2024-12-02T12:20:53Z

st instead of using /etc/timezone

@vivekrnv It is not recommended to mount the symlink. That is why, I mounted only the timezone and is creating the symlink at docker runtime.

vivekrnv · 2024-12-02T19:31:38Z

Hmm, so one way i could think is this. @nmoray Can you try if this solves the problem?

-TZ=$(cat /etc/timezone)
+TZ=$(timedatectl show -p Timezone --value)

prabhataravind · 2024-12-04T16:14:52Z

@nmoray, could you please take a look at Vivek's comments and rework your patch?

nmoray · 2024-12-05T06:07:52Z

@prabhataravind Okay, let me look into it.
@vivekrnv BTW did you try using "dpkg-reconfigure tzdata" command? It does update the /etc/timezone file

vivekrnv · 2024-12-05T08:23:43Z

@prabhataravind Okay, let me look into it. @vivekrnv BTW did you try using "dpkg-reconfigure tzdata" command? It does update the /etc/timezone file

I haven't but looks like that command is used in older systems or systems without systemd. Besides "config clock" command, in our deployments we use DHCP for mgmt ip. When eth0 recieved mgmt-ip, it also invokes the timedatectl command and changes the timezone.

Thus, we should consider timedatectl as the command that is used to change timezone and work upon it to solve this issue

nmoray · 2024-12-05T08:49:18Z

@vivekrnv Okay. IMO, following change will not work out. When the container is rebooted, "timedatectl show -p Timezone --value" will always return the timezone set inside the container only. So it will eventually create the symlink based on the default timezone only.

-TZ=$(cat /etc/timezone)
+TZ=$(timedatectl show -p Timezone --value)

Presently, I can think of three options to solve this issue.

Use "dpkg-reconfigure tzdata" in place of timedatectl
Update /etc/timezone file on the host at the time of updating timezone
Directly mount /etc/localtime on to the container (It is not a recommended option :-) )

nmoray · 2024-12-05T09:08:15Z

@vivekrnv There is one better way. The method timezone_update() from hostcfgd is responsible for updating the timezone. We can update the /etc/timezone file inside this method.

    def timezone_update(self, data):
        """
        Apply timezone handler.
        Run the following command in Linux: timedatectl set-timezone <timezone>
        Args:
            data: Read table's key's data.
        """
        new_timezone = data.get('timezone')
        syslog.syslog(syslog.LOG_DEBUG,
                      f'DeviceMetaCfg: timezone update to {new_timezone}')

        if new_timezone is None:
            syslog.syslog(syslog.LOG_DEBUG,
                          f'DeviceMetaCfg: Recieved empty timezone')
            return

        if new_timezone == self.timezone:
            syslog.syslog(syslog.LOG_DEBUG,
                          f'DeviceMetaCfg: No change in timezone')
            return

        # run command will print out log error in case of error
        run_cmd(['timedatectl', 'set-timezone', new_timezone])
        self.timezone = new_timezone

        run_cmd(['systemctl', 'restart', 'rsyslog'], True, False)
        syslog.syslog(syslog.LOG_INFO, 'DeviceMetaCfg: Restart rsyslog after '
                      'changing timezone')

nmoray · 2024-12-05T09:10:44Z

@vivekrnv Please try out this fix and let me know if it works.

vivekrnv · 2024-12-05T19:00:31Z

@nmoray, I'm in favour of option 3, did some preliminary testing and the change is working. I know its not ideal but this should work as long as the container has tzdata package installed.

Problem with option 1 & 2, they can solve the "config cli" flow but not the timezone setting through DHCP flow.

Eg for the dhcp flow:

2024 Nov 20 11:44:05.949094 sonic INFO dhclient[13076]: XMT: Solicit on eth0, interval 18420ms.
2024 Nov 20 11:44:24.387003 sonic INFO dhclient[13076]: XMT: Solicit on eth0, interval 36390ms.
2024 Nov 20 11:44:33.473125 sonic INFO dbus-daemon[1089]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.4' (uid=0 pid=14055 comm="timedatectl set-timezone Israel")
2024 Nov 20 11:44:33.509363 sonic INFO systemd[1]: Starting systemd-timedated.service - Time & Date Service...
2024 Nov 20 11:44:33.620416 sonic INFO dbus-daemon[1089]: [system] Successfully activated service 'org.freedesktop.timedate1'
2024 Nov 20 11:44:33.620665 sonic INFO systemd[1]: Started systemd-timedated.service - Time & Date Service.
2024 Nov 20 11:44:33.623377 sonic INFO systemd-timedated[14056]: Changed time zone to 'Israel' (IDT).

Once there is a config reload, the rsyslog also adjusts. 

2024 Nov 20 11:45:10.009245 sonic INFO systemd[1]: Starting backend-acl.service - Enable backend acl on storage backend ToRs...
2024 Nov 20 13:45:10.009965 sonic INFO systemd[1]: Stopping rsyslog.service - System Logging Service...
2024 Nov 20 13:45:10.010806 sonic INFO rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="7380" x-info="https://www.rsyslog.com"] exiting on signal 15.

Containers were adjusting till 2311 but broken from 202405.

For both flows to work, i could only think of option 3. I will raise a fix for this

nmoray · 2024-12-06T12:23:59Z

@vivekrnv one way to fix your DHCP flow is by overriding systemd-timedated.service. We can write the new timezone in /etc/timezone file as a part of ExecStartPost. This way, we can achieve our goal with minimal change.

nmoray · 2024-12-06T14:41:35Z

@vivekrnv for your reference,

root@CLS-DS2000-Leaf-1:~# cat /etc/nik_timezone 
America/Adak
root@CLS-DS2000-Leaf-1:~# timedatectl set-timezone America/Los_Angeles 
root@CLS-DS2000-Leaf-1:~# cat /etc/nik_timezone 
America/Los_Angeles
root@CLS-DS2000-Leaf-1:~# 
root@CLS-DS2000-Leaf-1:~# systemctl cat systemd-timedated.service 
# /lib/systemd/system/systemd-timedated.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Time & Date Service
Documentation=man:systemd-timedated.service(8)
Documentation=man:localtime(5)
Documentation=man:org.freedesktop.timedate1(5)

[Service]
BusName=org.freedesktop.timedate1
CapabilityBoundingSet=CAP_SYS_TIME
DeviceAllow=char-rtc r
ExecStart=/lib/systemd/systemd-timedated
IPAddressDeny=any
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProtectProc=invisible
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectSystem=strict
ReadWritePaths=/etc
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM
SystemCallFilter=@system-service @clock
WatchdogSec=3min
ExecStartPost=/bin/bash -c 'echo "$(timedatectl show -p Timezone --value)" > /etc/nik_timezone'

Note: I tested it on the earlier debian version so I used nik_timezone as a temporary file in place of /etc/timezone.

vivekrnv · 2024-12-07T00:33:45Z

/etc/timezone is deprecated in Debian. In SONiC, we should try to stay close to Debian for generic config and thus I’m not in favor in patching systemd to update /etc/timezone.

dgsudharsan added the Issue for 202405 label Nov 27, 2024

prabhataravind assigned nmoray Dec 4, 2024

prabhataravind added the Triaged this issue has been triaged label Dec 4, 2024

vivekrnv assigned vivekrnv and unassigned nmoray Dec 5, 2024

vivekrnv added the NVIDIA label Dec 5, 2024

vivekrnv mentioned this issue Dec 6, 2024

[Bookworm] Fix timezone propagation to containers #21079

Merged

11 tasks

liat-grozovik closed this as completed in #21079 Dec 11, 2024

liat-grozovik closed this as completed in ade34bf Dec 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bookworm] Timezone setting is not propagated to containers #20925

[Bookworm] Timezone setting is not propagated to containers #20925

vivekrnv commented Nov 26, 2024

vivekrnv commented Nov 27, 2024

nmoray commented Dec 2, 2024 •

edited

Loading

vivekrnv commented Dec 2, 2024

prabhataravind commented Dec 4, 2024

nmoray commented Dec 5, 2024 •

edited

Loading

vivekrnv commented Dec 5, 2024 •

edited

Loading

nmoray commented Dec 5, 2024 •

edited

Loading

nmoray commented Dec 5, 2024

nmoray commented Dec 5, 2024

vivekrnv commented Dec 5, 2024 •

edited

Loading

nmoray commented Dec 6, 2024 •

edited

Loading

nmoray commented Dec 6, 2024 •

edited

Loading

vivekrnv commented Dec 7, 2024

[Bookworm] Timezone setting is not propagated to containers #20925

[Bookworm] Timezone setting is not propagated to containers #20925

Comments

vivekrnv commented Nov 26, 2024

Description

202311

202405 and above:

Triage

vivekrnv commented Nov 27, 2024

nmoray commented Dec 2, 2024 • edited Loading

vivekrnv commented Dec 2, 2024

prabhataravind commented Dec 4, 2024

nmoray commented Dec 5, 2024 • edited Loading

vivekrnv commented Dec 5, 2024 • edited Loading

nmoray commented Dec 5, 2024 • edited Loading

nmoray commented Dec 5, 2024

nmoray commented Dec 5, 2024

vivekrnv commented Dec 5, 2024 • edited Loading

nmoray commented Dec 6, 2024 • edited Loading

nmoray commented Dec 6, 2024 • edited Loading

vivekrnv commented Dec 7, 2024

nmoray commented Dec 2, 2024 •

edited

Loading

nmoray commented Dec 5, 2024 •

edited

Loading

vivekrnv commented Dec 5, 2024 •

edited

Loading

nmoray commented Dec 5, 2024 •

edited

Loading

vivekrnv commented Dec 5, 2024 •

edited

Loading

nmoray commented Dec 6, 2024 •

edited

Loading

nmoray commented Dec 6, 2024 •

edited

Loading