Help with slow logs getting cut off

[nik9000]

One of our key features for debugging is the slow log - but when folks have very large _searches we can end up with cut off slow logs. Something chops them. I recall that we had something to do that but I can't find it at the moment. Maybe its some external tool doing the chopping

Anyway! Maybe we could make the slow logs shorter by not outputting default parameters when we toXContent the search's source. Or something. I dunno. I've just been frustrated lately when looking at slow logs from folks that are cut off and want to do something about it.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

@nik9000 I guess you are talking about plaintext slow logs in version before 7.x? or the config is stale?
In versions <7.0 we were truncating size of the message to 1000
#36833

for indexing slow log we still truncate and it is configurable by index.indexing.slowlog.source

can you confirm the version and the provide the logging config?

[jpountz]

We discussed it in a fix-it meeting. Having requests cut off in the slowlog is a big deal for troubleshooting production issues, and even if we had it entirely by raising the maximum length of a slowlog entry, the fact that it's so verbose makes it hard to read by a human. So the preferred route would be to stop serializing parameters when they have the default value.

We couldn't think of a component that would break if we stopped serializing default values. @nik9000 will try out by no longer serializing the size parameter of search requests to see if it breaks anything. In the case where some other components rely on having default values, we could still rely on the context that is passed to xcontent serialization so that default values would only be omitted when serializing requests to the slowlog.

[nik9000]

@nik9000 will try out by no longer serializing the size parameter of search requests to see if it breaks anything. In the case where some other components rely on having default values, we could still rely on the context that is passed to xcontent serialization so that default values would only be omitted when serializing requests to the slowlog.

It looks like it's been omitting the default size for a long, long time. So I picked a common query, match, and tried to omit it's defaults. That's #83338.

[nik9000]

• Queries
  • Compound
    • Boolean
    • Boosting
    • Constant score
    • Disjunction max
    • Function score
  • Full text Don't output default values in text query xcontent representations #86979
    • Intervals
    • Match (Shrink slow log for match query #83338)
    • Match boolean prefix
    • Match phrase
    • Match phrase prefix
    • Combined fields
    • Multi-match
    • Query string
    • Simple query string
  • Geo
    • Geo-bounding box
    • Geo-distance
    • Geo-polygon
    • Geoshape
  • Shape
  • Joining
    • Nested
    • Has child (Shrink slow log for has_child query #83870)
    • Has parent (Shrink join queries in slow log #83914)
    • Parent ID (Shrink join queries in slow log #83914)
  • Match all
  • Span Remove default values from term and span query xcontent #86985
    • Span containing
    • Span field masking
    • Span first
    • Span multi-term
    • Span near
    • Span not
    • Span or
    • Span term
    • Span within
  • Specialized
    • Distance feature
    • More like this
    • Percolate
    • Rank feature (Shrink slow log for rank_feature query #83847)
    • Script
    • Script score
    • Wrapper
    • Pinned Query
  • Term level
    • Exists
    • Fuzzy
    • IDs
    • Prefix
    • Range
    • Regexp
    • Term Remove default values from term and span query xcontent #86985
    • Terms
    • Terms set
    • Wildcard
• Aggregations
  • Bucketing
    • Adjacency matrix
    • Auto-interval date histogram
    • Categorize text
    • Children
    • Composite
    • Date histogram
    • Date range
    • Diversified sampler
    • Filter
    • Filters
    • Geo-distance
    • Geohash grid
    • Geotile grid
    • Global
    • Histogram
    • IP range
    • Missing
    • Multi Terms
    • Nested
    • Parent
    • Range
    • Rare terms
    • Reverse nested
    • Sampler
    • Significant terms
    • Significant text
    • Terms
    • Variable width histogram
  • Metric
    • Avg
    • Boxplot
    • Cardinality
    • Extended stats
    • Geo-bounds
    • Geo-centroid
    • Geo-Line
    • Matrix stats
    • Max
    • Median absolute deviation
    • Min
    • Percentile ranks
    • Percentiles
    • Rate
    • Scripted metric
    • Stats
    • String stats
    • Sum
    • T-test
    • Top hits
    • Top metrics
    • Value count
    • Weighted avg
  • Pipeline
    • Average bucket
    • Bucket script
    • Bucket count K-S test
    • Bucket correlation
    • Bucket selector
    • Bucket sort
    • Cumulative cardinality
    • Cumulative sum
    • Derivative
    • Extended stats bucket
    • Inference bucket
    • Max bucket
    • Min bucket
    • Moving function
    • Moving percentiles
    • Normalize
    • Percentiles bucket
    • Serial differencing
    • Stats bucket
    • Sum bucket
• The rest of the search request
  • inner_hits Shrink slow log for inner_hits #84143

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)