feat: support host monitor #1890

Abingcbc · 2024-11-18T02:08:43Z

打通进程元信息采集链路

TODO：

支持更多字段
可观测指标

core/common/timer/HostMonitorTimerEvent.cpp

core/models/PipelineEventGroup.h

core/runner/sink/http/HttpSink.cpp

core/host_monitor/collector/MockCollector.cpp

core/host_monitor/collector/CollectorManager.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/constants/EntityConstants.cpp

core/host_monitor/SystemInformationTools.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.cpp

core/common/FileSystemUtil.h

core/common/StringTools.cpp

core/common/timer/HostMonitorTimerEvent.cpp

core/plugin/input/InputHostMeta.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/HostMonitorInputRunner.cpp

core/common/timer/Timer.h

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/runner/sink/http/HttpSink.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.h

core/host_monitor/collector/ProcessCollector.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.cpp

core/common/MachineInfoUtil.cpp

core/common/timer/Timer.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/collector/ProcessEntityCollector.cpp

core/plugin/input/InputHostMeta.cpp

core/host_monitor/HostMonitorInputRunner.cpp

messixukejia · 2024-12-29T07:04:11Z

core/host_monitor/HostMonitorInputRunner.h

+    ThreadPool mThreadPool;
+
+    mutable std::shared_mutex mRegisteredCollectorMapMutex;
+    std::unordered_map<std::string, bool> mRegisteredCollectorMap;


std::unordered_map<std::string, bool> mRegisteredCollectorMap; std::unordered_map<std::string, std::shared_ptr<BaseCollector>> mCollectorInstanceMap;

这两个变量有点不合理。
包括
1、HostMonitorInputRunner::GetCollector的使用上。应该是自闭环合法性
2、锁的关系等
3、RegisterCollector 操作的变量是 mCollectorInstanceMap，但是另一个没操作的变量却叫reg

可能的优化
1、保持现有逻辑。一组功能不用过多发散变量，以上改成一个map。value用结构体表示。整体实现结构优化下。
2、UpdateCollector时再注册ProcessEntityCollector实例，这样能省一个变量。

core/host_monitor/collector/ProcessEntityCollector.cpp

core/common/timer/Timer.cpp

messixukejia · 2024-12-30T08:56:13Z

core/host_monitor/collector/ProcessEntityCollector.cpp

+
+const std::string ProcessEntityCollector::sName = "process_entity";
+
+ProcessEntityCollector::ProcessEntityCollector() : mProcessSilentCount(INT32_FLAG(process_collect_silent_count)) {


任务执行异步
超时影响
不同任务类型见例如Prometheus、host entity、processo不影响

线程池放入不会堵塞，runner拿到任务后，可以立刻放入线程池

在一个任务执行结束后，再放入下一个任务。一个任务超时，会计算从上一次执行时间开始，增加n个周期，到达下一个未来可执行的时间点。
nextExecTime = execTime + n * interval

core/common/MachineInfoUtil.cpp

core/common/MachineInfoUtil.h

docs/cn/plugins/input/native/input-host-meta.md

core/host_monitor/collector/ProcessEntityCollector.cpp

messixukejia · 2025-01-07T01:48:46Z

core/constants/EntityConstants.cpp

+// process entity
+const std::string DEFAULT_CONTENT_VALUE_ENTITY_TYPE_ECS_PROCESS = "acs.ecs.process";
+const std::string DEFAULT_CONTENT_VALUE_ENTITY_TYPE_HOST_PROCESS = "infra.host.process";
+const std::string DEFAULT_CONTENT_KEY_PROCESS_PID = "pid";


字段是如何跟安全保持一致的？

找讯飞看了一下ebpf那边代码还比较乱，自己维护了一套constant，没有用公共的。
是否可以先合入，然后等ebpf合入后，两边再对齐一下？

core/host_monitor/HostMonitorInputRunner.cpp

core/models/PipelineEventGroup.h

core/host_monitor/HostMonitorInputRunner.cpp

core/host_monitor/collector/ProcessEntityCollector.cpp

core/app_config/AppConfig.cpp

core/host_monitor/HostMonitorInputRunner.cpp

core/models/PipelineEventGroup.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/collector/ProcessEntityCollector.cpp

core/constants/EntityConstants.cpp

core/host_monitor/HostMonitorInputRunner.cpp

core/common/timer/Timer.h

core/application/Application.cpp

core/unittest/host_monitor/1/stat

core/host_monitor/Constants.cpp

core/host_monitor/HostMonitorInputRunner.h

core/plugin/input/InputHostMeta.cpp

core/plugin/input/InputHostMeta.h

core/application/Application.cpp

core/host_monitor/HostMonitorInputRunner.h

core/host_monitor/HostMonitorInputRunner.cpp

core/host_monitor/SystemInformationTools.cpp

core/host_monitor/HostMonitorInputRunner.h

core/host_monitor/HostMonitorInputRunner.cpp

core/plugin/input/InputHostMeta.cpp

messixukejia · 2025-02-26T07:47:37Z

core/host_monitor/collector/ProcessEntityCollector.cpp

+
+const size_t ProcessTopN = 20;
+
+const std::string ProcessEntityCollector::sName = "process_entity";


core/host_monitor/HostMonitorInputRunner.h

Abingcbc force-pushed the host_monitor branch 6 times, most recently from 6150e52 to bfdd9c2 Compare November 19, 2024 06:27

messixukejia requested changes Nov 25, 2024

View reviewed changes

henryzhx8 requested changes Nov 26, 2024

View reviewed changes

feat: support host monitor

3e12438

Abingcbc force-pushed the host_monitor branch from 4110e3d to 3e12438 Compare November 27, 2024 06:22

Abingcbc added 2 commits November 29, 2024 00:01

fix

8f9f8ee

Merge remote-tracking branch 'alibaba/main' into host_monitor

0455a6c

messixukejia requested changes Nov 30, 2024

View reviewed changes

messixukejia requested changes Dec 4, 2024

View reviewed changes

core/common/MachineInfoUtil.cpp Outdated Show resolved Hide resolved

core/common/MachineInfoUtil.cpp Outdated Show resolved Hide resolved

core/common/timer/Timer.h Show resolved Hide resolved

Abingcbc added 7 commits December 27, 2024 00:34

fix

5435d03

Merge remote-tracking branch 'alibaba/main' into host_monitor

6f8d8d4

fix

f7b71ab

fix

cc1f455

fix

0693bb1

fix

77f1384

fix

719fb39

messixukejia reviewed Dec 29, 2024

View reviewed changes

catdogpandas reviewed Dec 30, 2024

View reviewed changes

core/common/timer/Timer.cpp Outdated Show resolved Hide resolved

Abingcbc added 2 commits January 5, 2025 22:03

fix

84332c9

Merge remote-tracking branch 'alibaba/main' into host_monitor

3fd57b3

messixukejia reviewed Jan 6, 2025

View reviewed changes

Abingcbc added 3 commits January 6, 2025 09:59

fix

f2e1885

lint

3b27c1b

fix

2cb4b7b

Abingcbc force-pushed the host_monitor branch from 8fdce38 to 2cb4b7b Compare January 6, 2025 07:45

fix

c4614a8

messixukejia requested changes Jan 7, 2025

View reviewed changes

core/app_config/AppConfig.cpp Outdated Show resolved Hide resolved

core/host_monitor/HostMonitorInputRunner.cpp Outdated Show resolved Hide resolved

core/models/PipelineEventGroup.h Outdated Show resolved Hide resolved

Abingcbc added 3 commits February 10, 2025 14:24

Merge branch 'main' into host_monitor

5ee6a23

fix

b9200b1

fix lint

910d709

messixukejia reviewed Feb 14, 2025

View reviewed changes

henryzhx8 requested changes Feb 21, 2025

View reviewed changes

Abingcbc added 4 commits February 24, 2025 11:46

fix

2ee7e5c

Merge remote-tracking branch 'alibaba/main' into host_monitor

a31ab65

fix

6fc97ea

fix

e46bcf2

henryzhx8 requested changes Feb 25, 2025

View reviewed changes

Abingcbc added 2 commits February 26, 2025 12:26

fix

573eaeb

fix

352f7da

messixukejia approved these changes Feb 26, 2025

View reviewed changes

henryzhx8 requested changes Feb 26, 2025

View reviewed changes

core/host_monitor/HostMonitorInputRunner.h Outdated Show resolved Hide resolved

fix

67283a8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support host monitor #1890

feat: support host monitor #1890

Abingcbc commented Nov 18, 2024

messixukejia Dec 29, 2024

messixukejia Dec 30, 2024

messixukejia Dec 30, 2024

Abingcbc Jan 6, 2025 •

edited

Loading

messixukejia Jan 7, 2025

Abingcbc Feb 12, 2025

messixukejia Feb 26, 2025


		const std::string ProcessEntityCollector::sName = "process_entity";

		ProcessEntityCollector::ProcessEntityCollector() : mProcessSilentCount(INT32_FLAG(process_collect_silent_count)) {


		const size_t ProcessTopN = 20;

		const std::string ProcessEntityCollector::sName = "process_entity";

feat: support host monitor #1890

Are you sure you want to change the base?

feat: support host monitor #1890

Conversation

Abingcbc commented Nov 18, 2024

messixukejia Dec 29, 2024

Choose a reason for hiding this comment

messixukejia Dec 30, 2024

Choose a reason for hiding this comment

messixukejia Dec 30, 2024

Choose a reason for hiding this comment

Abingcbc Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

messixukejia Jan 7, 2025

Choose a reason for hiding this comment

Abingcbc Feb 12, 2025

Choose a reason for hiding this comment

messixukejia Feb 26, 2025

Choose a reason for hiding this comment

Abingcbc Jan 6, 2025 •

edited

Loading