Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Naïve fix for optimize bug with line_format and json expression parser #6375

Merged

Conversation

dannykopping
Copy link
Contributor

@dannykopping dannykopping commented Jun 13, 2022

Signed-off-by: Danny Kopping [email protected]

What this PR does / why we need it:
This PR implements a naïve fix for the bug listed below.

Which issue(s) this PR fixes:
Fixes #6314

Special notes for your reviewer:
The more correct way will be to unify the JSONExpressionParser and LabelParserExpr.

func (e *LabelParserExpr) Stage() (log.Stage, error) {
	switch e.Op {
	case OpParserTypeJSON:
		return log.NewJSONParser(), nil
	case OpParserTypeLogfmt:
		return log.NewLogfmtParser(), nil
	case OpParserTypeRegexp:
		return log.NewRegexpParser(e.Param)
	case OpParserTypeUnpack:
		return log.NewUnpackParser(), nil
	case OpParserTypePattern:
		return log.NewPatternParser(e.Param)
	default:
		return nil, fmt.Errorf("unknown parser operator: %s", e.Op)
	}
}

Ultimately the JSONExpressionParser should be instantiated in here, since this is how all other parsers work. I must've missed this somehow when initially implementing the JSONExpressionParser.

It'll take a bit of work because in pkg/logql/syntax/expr.y

labelParser:
    JSON           { $$ = newLabelParserExpr(OpParserTypeJSON, "") }
  | LOGFMT         { $$ = newLabelParserExpr(OpParserTypeLogfmt, "") }
  | REGEXP STRING  { $$ = newLabelParserExpr(OpParserTypeRegexp, $2) }
  | UNPACK         { $$ = newLabelParserExpr(OpParserTypeUnpack, "") }
  | PATTERN STRING { $$ = newLabelParserExpr(OpParserTypePattern, $2) }
  ;

all of the LabelParserExprs are expecting a string argument, while the JSONExpressionParser expects a slice of log.JSONExpression.

I took a swing at this last week but there's a bit of complexity here which I don't have time right now to address, so in the interim I've implemented this simple fix which will make the problem go away, however hacky it may be.

Checklist

  • Documentation added
  • Tests updated
  • Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.1%
+               loki	0%

@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.1%
+               loki	0.7%

@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.1%
+               loki	0%

Danny Kopping added 2 commits June 13, 2022 13:11
Signed-off-by: Danny Kopping <[email protected]>
Signed-off-by: Danny Kopping <[email protected]>
@dannykopping dannykopping force-pushed the dannykopping/optimize-bug-json branch from 866d973 to 35c1438 Compare June 13, 2022 11:11
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.1%
+               loki	0%

@dannykopping dannykopping marked this pull request as ready for review June 13, 2022 11:46
@dannykopping dannykopping requested a review from a team as a code owner June 13, 2022 11:46
Copy link
Contributor

@chaudum chaudum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think it's ok to do the quick fix now. Have a suggestion for the changelog entry, though.

CHANGELOG.md Outdated Show resolved Hide resolved
Co-authored-by: Christian Haudum <[email protected]>
@dannykopping
Copy link
Contributor Author

Think it's ok to do the quick fix now. Have a suggestion for the changelog entry, though.

That's indeed more clear, thanks @chaudum 👍
Can you approve pls?

@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.1%
+               loki	0%

Copy link
Contributor

@chaudum chaudum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@dannykopping dannykopping merged commit a649e0d into grafana:main Jun 13, 2022
@dannykopping dannykopping deleted the dannykopping/optimize-bug-json branch June 13, 2022 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using json expressions with line_format in a metric query leads to unexpected results
3 participants