Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new(userspace): plugin api to dump async events #2152

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

FedeDP
Copy link
Contributor

@FedeDP FedeDP commented Nov 7, 2024

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area libsinsp

Does this PR require a change in the driver versions?

What this PR does / why we need it:

This PR adds a new dump() API under the async events plugin capability.
It is used to dump plugin state (as list of PPME_ASYNCEVENT_E events) when a scap file dump is requested.
Note:

  • only PPME_ASYNCEVENT_E can be dumped
  • PPME_ASYNCEVENT_E dumped must have a name supported by the plugin (see get_async_events() API)

Moreover, enable initialstate consuming of PPME_ASYNCEVENT_E events while opening the inspector.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

new(userspace): plugin api to dump async events

@FedeDP
Copy link
Contributor Author

FedeDP commented Nov 7, 2024

/milestone 0.19.0
/cc @jasondellaluce @mrgian

@poiana poiana added this to the 0.19.0 milestone Nov 7, 2024
@poiana poiana added the size/L label Nov 7, 2024
@poiana
Copy link
Contributor

poiana commented Nov 7, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@@ -159,7 +159,7 @@ bool sinsp::is_initialstate_event(scap_evt* pevent) const {
return pevent->type == PPME_CONTAINER_E || pevent->type == PPME_CONTAINER_JSON_E ||
pevent->type == PPME_CONTAINER_JSON_2_E || pevent->type == PPME_USER_ADDED_E ||
pevent->type == PPME_USER_DELETED_E || pevent->type == PPME_GROUP_ADDED_E ||
pevent->type == PPME_GROUP_DELETED_E;
pevent->type == PPME_GROUP_DELETED_E || pevent->type == PPME_ASYNCEVENT_E;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial state events parsing enablement for PPME_ASYNCEVENT_E.

Copy link

github-actions bot commented Nov 7, 2024

Perf diff from master - unit tests

     8.88%     +1.31%  [.] sinsp_parser::reset
     6.66%     +1.14%  [.] sinsp::next
     2.28%     -0.73%  [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>
     1.39%     -0.67%  [.] std::_Hashtable<long, std::pair<long const, std::shared_ptr<sinsp_threadinfo> >, std::allocator<std::pair<long const, std::shared_ptr<sinsp_threadinfo> > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find
     0.72%     +0.59%  [.] std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, libsinsp::state::dynamic_struct::field_info>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, libsinsp::state::dynamic_struct::field_info> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find
     0.72%     -0.53%  [.] libsinsp::state::dynamic_struct::field_infos::add_field_info
     2.28%     -0.50%  [.] std::_Hashtable<long, std::pair<long const, std::shared_ptr<sinsp_threadinfo> >, std::allocator<std::pair<long const, std::shared_ptr<sinsp_threadinfo> > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node
     0.52%     +0.46%  [.] std::vector<sinsp_evt_param, std::allocator<sinsp_evt_param> >::emplace_back<sinsp_evt*, unsigned int&, char const*, unsigned long&>
     0.67%     +0.42%  [.] sinsp_parser::event_cleanup
     0.76%     -0.42%  [.] std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, libsinsp::state::dynamic_struct::field_info>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, libsinsp::state::dynamic_struct::field_info> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, libsinsp::state::dynamic_struct::field_info> >

Heap diff from master - unit tests

peak heap memory consumption: -816B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Heap diff from master - scap file

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Benchmarks diff from master

Comparing gbench_data.json to /root/actions-runner/_work/libs/libs/build/gbench_data.json
Benchmark                                                         Time             CPU      Time Old      Time New       CPU Old       CPU New
----------------------------------------------------------------------------------------------------------------------------------------------
BM_sinsp_split_mean                                            +0.0008         +0.0008           143           143           143           143
BM_sinsp_split_median                                          +0.0011         +0.0011           143           143           143           143
BM_sinsp_split_stddev                                          -0.3731         -0.3735             2             1             2             1
BM_sinsp_split_cv                                              -0.3737         -0.3740             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_mean                  -0.0037         -0.0037            60            60            60            60
BM_sinsp_concatenate_paths_relative_path_median                +0.0040         +0.0040            59            60            59            60
BM_sinsp_concatenate_paths_relative_path_stddev                -0.8123         -0.8130             1             0             1             0
BM_sinsp_concatenate_paths_relative_path_cv                    -0.8116         -0.8123             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_mean                     -0.0078         -0.0078            25            25            25            25
BM_sinsp_concatenate_paths_empty_path_median                   -0.0209         -0.0210            25            25            25            25
BM_sinsp_concatenate_paths_empty_path_stddev                   +0.0759         +0.0755             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_cv                       +0.0844         +0.0840             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_mean                  -0.0112         -0.0112            63            62            63            62
BM_sinsp_concatenate_paths_absolute_path_median                -0.0120         -0.0120            63            62            63            62
BM_sinsp_concatenate_paths_absolute_path_stddev                +0.3490         +0.3511             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_cv                    +0.3643         +0.3664             0             0             0             0
BM_sinsp_split_container_image_mean                            -0.0126         -0.0126           385           381           385           381
BM_sinsp_split_container_image_median                          -0.0121         -0.0121           385           380           385           380
BM_sinsp_split_container_image_stddev                          +0.3297         +0.3292             3             4             3             4
BM_sinsp_split_container_image_cv                              +0.3467         +0.3462             0             0             0             0

@FedeDP
Copy link
Contributor Author

FedeDP commented Nov 7, 2024

/hold
fixing a couple of things.

@FedeDP
Copy link
Contributor Author

FedeDP commented Nov 7, 2024

/unhold

Copy link

codecov bot commented Nov 7, 2024

Codecov Report

Attention: Patch coverage is 72.54902% with 28 lines in your changes missing coverage. Please review.

Project coverage is 74.84%. Comparing base (2e1c5b6) to head (7fcd4dc).

Files with missing lines Patch % Lines
userspace/libsinsp/plugin.cpp 53.48% 20 Missing ⚠️
userspace/libsinsp/dumper.cpp 50.00% 3 Missing ⚠️
userspace/libsinsp/test/plugins/syscall_async.cpp 83.33% 3 Missing ⚠️
userspace/libsinsp/test/plugins.ut.cpp 94.11% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2152      +/-   ##
==========================================
+ Coverage   74.82%   74.84%   +0.02%     
==========================================
  Files         254      255       +1     
  Lines       33519    33620     +101     
  Branches     5749     5779      +30     
==========================================
+ Hits        25079    25162      +83     
- Misses       8440     8458      +18     
Flag Coverage Δ
libsinsp 74.84% <72.54%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

Copy link
Contributor

@jasondellaluce jasondellaluce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely! Left few comments

// pointer is not retained by the handler after it returns.
//
// Return value: A ss_plugin_rc with values SS_PLUGIN_SUCCESS or SS_PLUGIN_FAILURE.
ss_plugin_rc (*dump)(ss_plugin_t* s,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ss_plugin_rc (*dump)(ss_plugin_t* s,
ss_plugin_rc (*dump_state)(ss_plugin_t* s,

Not sure what's the right name for this... but I think that giving a more scoped name (I'll trust you with that) would save us some headaches maybe in the future. @gnosek since you envisioned this to happen too some time ago, you have any naming suggestions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with the naming, gonna change it asap!

@@ -1053,6 +1053,25 @@ typedef struct {
ss_plugin_rc (*set_async_event_handler)(ss_plugin_t* s,
ss_plugin_owner_t* owner,
const ss_plugin_async_event_handler_t handler);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't fully enjoy the fact that this is tied to the async events capability. For instances, certain plugins may collect some state through parse_event and make it available through the table API too, but maybe will never emit async events.

However, I see the point. It also makes sense to "recycle" get_async_event_sources and get_async_events, since they are indeed useful for this use case. Given that this symbol is optional, my interpretation is that defining set_async_event_handler will be a requirement even if not used. If so, we'll have the opportunity later on to change our minds and loosen the requirements without incurring in a breaking change. So I think we can sleep peacefully with this for now 👍

userspace/libsinsp/test/plugins.ut.cpp Show resolved Hide resolved
userspace/libsinsp/plugin.h Outdated Show resolved Hide resolved
return false;
}

m_async_dump_handler = [&dumper](auto e) { dumper.dump(e.get()); };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we don't have thread safety issue here compared to the async event handler, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly!

Also, ASYNCEVENT_E will not be correctly pre-parsed at init time while reading from captures.

Signed-off-by: Federico Di Pierro <[email protected]>
We now expect `PPME_ASYNCEVENT_E` whose `name` matches one of the
plugin supported ones (get_async_events() API).

The new API is not required for async capability.

Added also a test.

Signed-off-by: Federico Di Pierro <[email protected]>
Other tests using no_driver engine and async plugin are also disabled on it.

Signed-off-by: Federico Di Pierro <[email protected]>
@FedeDP
Copy link
Contributor Author

FedeDP commented Nov 22, 2024

Pushed with the proposed changes!

@FedeDP
Copy link
Contributor Author

FedeDP commented Nov 25, 2024

Moving to next milestone.
/milestone 0.20.0

@poiana poiana modified the milestones: 0.19.0, 0.20.0 Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants