want paravirtualized clock enlightenment #328
Assignees
Labels
customer
For any bug reports or feature requests tied to customer requests
Milestone
Comments
|
guest is and the propolis version is 6282016343573335026e1f1adb5d4dedbcd8b079 |
|
if I grep out all lines in the dmesg that contain the string |
|
full dmesg: |
|
answering some out-of-band questions:
|
|
Jotting down some notes about my cursory research into this so far looking at the linux 5.10 source:
I have a couple of follow up questions / things to explore, but I need to step away for a bit and wanted to write this down before I did. |
|
How loaded was the host system when this occurred? We've seen behavior like this before when the host is busy. The thread(s) running vCPUs struggle to be scheduled in a timely manner. @jordanhendricks's analysis helps shed light on how that could manifest itself in a reading which was reportedly unstable: If we took an excessively long exit to service the second ( Using the values listed in the dmesg output, we can clean what sort of delay was observed: PM timer: 6c50c5 to f4b7d (wrapping at 24 bits) is 10681015 ticks or 2983902982ns (@ 3579545Hz) That puts the PM timer reading at ~5600us off the TSC. Jordan noted a threshold of 65ms, but that assumes the default uncertainty margins. I believe both the TSC and PM timer clocksources have their margins set according to their frequency, with a lower bound of |
Pretty loaded, and there were several guests running at the time. They were all running a ffmpeg encoding test workload. |
|
This is still kicking around, I think--I saw it hit while running a local PHD test run in its This test usually passes (this is the first time I've seen this flake in many recent runs), and I don't know of any reason this would have regressed recently, but wanted to make a note to keep this issue from getting any more stale. |
twinfees
added
customer
|
I'm able to force the TSC to be abandoned on a Debian 11 guest using the DTrace BEGIN {
i = 0;
}
fbt:vmm:vpmtmr_handler:entry {
i = i + 1;
printf("[%d] [%u] PM timer read", i, walltimestamp);
}
fbt:vmm:vpmtmr_handler:return /(i % 3) == 1/ {
chill(1000000);
printf("[%d] [%u] delayed return from timer read!", i, walltimestamp);
}The guest is using the ACPI PM timer as a watchdog for the TSC. Its watchdog sequence is
If the differences between any pair of consecutive PM timer reads is too large, the guest concludes that there's too much scheduling or I/O weather for the watchdog to be reliable and abandons it. The script inserts a delay before returning from the first PM timer read--i.e. it inserts a delay between the moment the first PM timer value is captured and the moment the guest reads the TSC. With a proper chill value (1ms here) this gets the watchdog to fire reliably. |
gjcolombo
changed the title
This was referenced Feb 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@gjcolombo edit: This is the primary tracking issue for cases where a guest decides to stop using the TSC as a reference time source. We propose to address this by implementing a paravirt clock that allows the guest to use the TSC reliably. For more details about what can cause guests to abandon the TSC, see the discussion below.
one of my guests is showing some dmesg messages relating to clock skew:
The text was updated successfully, but these errors were encountered: