Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lambda/promtail: lambda/promtail fails to send logs with "panic: invalid label name" #11005

Closed
spedersen-emailage opened this issue Oct 23, 2023 · 2 comments · Fixed by #11074
Labels
lambda-promtail type/bug Somehing is not working as expected

Comments

@spedersen-emailage
Copy link

Describe the bug

The newest version of lambda-promtail (since Oct. 9) fails to send logs with a "panic: invalid label name" error.

To Reproduce

  1. Send logs using latest version of lambda-promtail from ec54c72.

Expected behavior

I expect lambda-promtail to send logs. ;-)

Environment:

  • Infrastructure: AWS Lambda
  • Deployment tool: Docker

Screenshots, Promtail config, or terminal output

Lambda logs:

2023-10-20T13:30:59.331-07:00	START RequestId: <REQUEST_ID_REMOVED> Version: $LATEST
2023-10-20T13:30:59.350-07:00	RequestId: <REQUEST_ID_REMOVED> Error: Runtime exited with error: exit status 2 Runtime.ExitError
2023-10-20T13:30:59.350-07:00	END RequestId: <REQUEST_ID_REMOVED>
2023-10-20T13:30:59.350-07:00	REPORT RequestId: <REQUEST_ID_REMOVED> Duration: 152.42 ms Billed Duration: 153 ms Memory Size: 128 MB Max Memory Used: 6 MB
2023-10-20T13:31:05.890-07:00	write address: https://logs-prod3.grafana.net/loki/api/v1/push
2023-10-20T13:31:05.909-07:00	extra labels: {appname="aws-grafana-forwarder", env="dev", region="us-west-2"}
2023-10-20T13:31:05.911-07:00	panic: invalid label name
2023-10-20T13:31:05.911-07:00	goroutine 1 [running]:
2023-10-20T13:31:05.911-07:00	main.setupArguments()
2023-10-20T13:31:05.911-07:00	/src/lambda-promtail/lambda-promtail/main.go:66 +0x47d
2023-10-20T13:31:05.911-07:00	main.main()
2023-10-20T13:31:05.911-07:00	/src/lambda-promtail/lambda-promtail/main.go:241 +0x13

I believe it's related to this feature update: [feat(lambda/promtail): support dropping labels] (ec54c72)

Note: We do not use the new DROP_LABELS environment variable.

The source of the panic error is getDropLabels(); it doesn't look like it can handle blank/null values, but I don't know Go so I can't say with certainty.

@spedersen-emailage
Copy link
Author

Thanks @hainenber !

chaudum pushed a commit that referenced this issue Nov 6, 2023
@spedersen-emailage
Copy link
Author

I can confirm this resolves the issue. Thanks!

rhnasc pushed a commit to inloco/loki that referenced this issue Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lambda-promtail type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants