Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dataman: Add client sync perf counter and increase default timeout to 5s #22845

Merged
merged 1 commit into from
Mar 13, 2024

Conversation

niklaut
Copy link
Contributor

@niklaut niklaut commented Mar 6, 2024

Solved Problem

Dataman requests occasionally time out during boot. My suspicion is that its a combination of slow SDCard and unfortunate scheduling order during boot when the CPU is very busy. The timeout of 1s may be reached more quickly than expected.

However, it is notoriously difficult to reproduce this issue under controlled and debuggable/traceable conditions. I was only able to catch it once and the dataman: write perf counter "only" recorded most=310ms. However, on unrelated traces I've seen scheduling latencies on the dataman thread in the hundreds of milliseconds, here 271ms:

Solution

This increases the dataman client timeout to 5s and logs the time it took to service the sync requests.
There are already two perf counters dataman: write and dataman: read, however, those are running in the dataman thread, while the client is running on another thread. Thus adding this perf counter will show scheduling latencies between the dataman and its client threads.

Changelog Entry

For release notes:

Bugfix Increase dataman client timeout

Alternatives

Dataman request timeouts are bad anyways, since the data is needed to fulfill the mission. There really isn't any alternative to not getting any data.

Context

See also #22778.

@niklaut niklaut requested a review from bkueng March 6, 2024 10:31
@niklaut niklaut force-pushed the wip/dataman-client-perf-counter branch from e963c30 to 774b7b8 Compare March 6, 2024 10:48
@niklaut niklaut force-pushed the wip/dataman-client-perf-counter branch from 774b7b8 to 2c370bc Compare March 7, 2024 09:33
@niklaut niklaut force-pushed the wip/dataman-client-perf-counter branch from 2c370bc to fec27ba Compare March 7, 2024 09:33
@bkueng bkueng merged commit 23c5c0b into PX4:main Mar 13, 2024
88 of 89 checks passed
@niklaut niklaut deleted the wip/dataman-client-perf-counter branch March 26, 2024 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants