-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to parse ISO8601 #1528
Comments
Is specifying |
Is doesn't work. ISO8601 can be formatted in different ways, just check out the source for |
It means your log has different formatted ISO8601 format in one stream/file? |
Yes. We have a generic log format parser for all of out docker containers. Some of them are outputting timestams with milliseconds or not depending on user input on particular request (users are allowed to pass timestamps in ISO 8601). |
I see. |
@repeatedly, actually, I've got a similar problem, but the
log.log:
Output (fractional seconds are now zero in the last "time" field):
|
I forgot about this. Maybe someone else could pick up from here and add the |
@ChipmunkV Do you use fluentd v0.12? |
@repeatedly, it's not very obvious from the docs that v0.12 doesn't support anything except seconds and you need v0.14. Other non-obvious thing is the versioning of the td-agent: you would think that v2.3.5 is greater than v0.12, but it's not and not related at all. |
Support %iso8601 special case for time_format. fix #1528
Will it be backported to v0.12? Or is it safe to use v0.14 now? We tried v0.14 a year ago and have had problems, so we've switched back to v0.12. |
Currently fluentd branch 0.12 does not contain code to allow parsing ISO8601 time format.
For some reason unknown to me, when
time_format
is omitted in JSON parser, time is interpreted as integer value (UNIX-time). However, there is no way to parse ISO 8601: it cannot be described as aTime#strptime
format, since it allows ambiguities. On the onther hand, it is trivial to implement UNIX-time parising withTime#strptime
:Time.strptime(value.to_s, '%s')
. This may be further optimized to parse integers directly (like it's done currently in the code), but it's only applicable iftime_format == '%s'
!It just seems like a poor design decision to make default format different from ISO8601 like in the rest of the parsers and inputs.
Can this be fixed now, or should this maintain backward-compatibility with the previous design? In case we have to maintain backwards compatibility, maybe we could add a special case to allow proper ISO8601 parsing? Maybe add logic for
%iso8601
format string to force parsing withTime.iso8601
?The text was updated successfully, but these errors were encountered: