Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support to process tag #1806

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Conversation

Abingcbc
Copy link
Collaborator

@Abingcbc Abingcbc commented Oct 14, 2024

支持重命名和修改Tag

配置

  • Input级别配置(Reader实现):控制插件级tag pathinode、容器元信息、ExternalEnvTag、ExternalK8sLableTag
  • Global级别配置(内置插件实现):控制Pipeline级tag File Tag、ENV Tag、sourcehostnameuser_defined_id

内部数据结构定义调整

  • metadata:仅用于存储处理和发送阶段需要使用的信息
  • tag:用于保存所有输出字段

@Abingcbc Abingcbc changed the title feat: support to rename and delete tag feat: support to process tag Oct 16, 2024
Comment on lines +288 to +289
bool IsKeyExist(const Json::Value& config, const std::string& key);

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

删了

const std::string HOST_IP_DEFAULT_KEY = "host.ip";
#endif

} // namespace logtail
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

文件末尾空行,所有自查一下

bool enableContainerDiscovery);
StringView GetFileTagKeyName(TagKey key) const;


Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change


#include "file_server/FileTagOptions.h"

#include "common/ParamExtractor.h"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpp文件都直接用using namespace std吧,代码里的std可以删了

const Json::Value* tagConfig = config.find(tagKey, tagKey + strlen(tagKey));
if (tagConfig) {
if (!tagConfig->isObject()) {
PARAM_WARNING_DEFAULT(context.GetLogger(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里应该是warning ignore

@@ -475,6 +475,11 @@ void Pipeline::CopyNativeGlobalParamToGoPipeline(Json::Value& pipeline) {
Json::Value& global = pipeline["global"];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里优化一下吧,这个Copy函数不要从pipeline去调用了,把initglobal函数增加两个入参,就是两个gopipeline,直接在那个里面添加,这样global config里根本不需要存这些参数了,这些参数所有权本来就在tag插件里

Copy link
Collaborator

@henryzhx8 henryzhx8 Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 整体实现需要重新看一下文档
  2. 这个插件应该是pipeline级别的了
  3. C++和Go串联的情况下,应该全部由C++这边来完成处理,Go应该啥都不做,只有Aggregator的Topic是个例外

Comment on lines +252 to +258
if (group.HasMetadata(EventGroupMetaKey::TOPIC)) {
logGroup.set_topic(group.GetMetadata(EventGroupMetaKey::TOPIC).to_string());
}
for (const auto& tag : group.GetTags()) {
if (tag.first == LOG_RESERVED_KEY_TOPIC) {
logGroup.set_topic(tag.second.to_string());
} else {
auto logTag = logGroup.add_logtags();
logTag->set_key(tag.first.to_string());
logTag->set_value(tag.second.to_string());
}
auto logTag = logGroup.add_logtags();
logTag->set_key(tag.first.to_string());
logTag->set_value(tag.second.to_string());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里不用改

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不用改

@@ -79,7 +79,7 @@ func flushOutStore[T FlushData, F FlusherWrapperInterface](lc *LogstoreConfig, s
}

func loadAdditionalTags(globalConfig *config.GlobalConfig) models.Tags {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个函数还需要?

@henryzhx8 henryzhx8 added enhancement Feature enhancement core Core feature labels Oct 22, 2024
@henryzhx8 henryzhx8 added this to the v2.2 milestone Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core feature enhancement Feature enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants