RPS count drops when master and slaves drift in time

[hrosenhorn]

It looks like the master trusts the slaves to report the correct time and if the system clock drifts the rps value will be reported incorrectly.

During a 24 hour run, running 10k rps this value will drop to about 6k rps.

Doing a ntpdate ntp.kth.se on master and the slaves restores the rps count back to 10k

[heyman]

I think it would be hard to remove the requirement that the different machines should have synced system clocks. The alternative would be to rely on the time when the "reports" arrive to the master from the slave nodes, which I think would be less reliable.

[cgbystrom]

Sounds reasonable.

@hjlarsson Do you recall how much the clock drifted during those 24 hours? It seems odd that the clock should be drifting that much between master/slaves in the first place.

[Jahaja]

Obviously the simple way of handling this is having ntpd setup properly. However I see the following options as a way forward on this issue.

1. Making the slaves report in their local time when they connect to the master and warn if a significant clock skew is detected.
2. In addition to 1, the master then compensates for the detected clock skew when receiving reports.

With the latter option feeling quite naive and error prone, I'd suggest doing just the first - which in the worst case, horrendous connect time, will result in a false warning.

[cgbystrom]

Sticking to KISS, I think it is fair to expect the OS to provide an accurate system clock.

[heyman]

I'm closing this since we agree that doing some auto compensation is a bad idea.

Issuing a warning in the master node, if a slave connects and reports a time that is off by more than a certain timespan, isn't related to the reported problem and can be discussed in a new ticket if we think it should be done :).