Extend redis plugin capabilities

[mathieuboniface]

We are massively using redis lists (and telegraf too). We would like export using telegraf the sizes of thoses lists stored into redis instances.

Currently, the redis input plugin gather metrics from the INFO command. And there is no way to get metrics stored into redis database.

For us, a good solution could be to permit the redis plugin to gather metrics from command given in the configuration file, something like this :

[[inputs.redis]]
  servers = ["tcp://localhost:6379"]
  commands = ["LLEN myList", "GET myKey", "CLUSTER COUNT-FAILURE-REPORTS myNode"]

WDYT ?

[radcheb]

+1

[sparrc]

is this the same as a redis subscriber plugin? #759

[mathieuboniface]

Hi sparrc, and thanks for your quick reply.

#759 is focused on pub/sub mechanism of redis. This new feature is more like the postgresql_extended plugin principle. The idea is to permit the execution of some commands (queries) to redis in order to gather metrics.

[UserNotFound]

@mathieuboniface Did you find a solution? I am about to just starting writing something using the telegraf exec input, and some scripting using redis-cli llen {listname}

[athoune]

you need something more dynamic like http_json.

A list of requests is not enough. You need a key per request (with mandatory number value), and handling hash response. With hash, you can handle dynamic key/value.

Here is the hash example for sidekiq, for watching queues length :

EVAL "local a={}; local queues=redis.call('smembers', 'resque:gitlab:queues'); for i,v in ipairs(queues) do a[i*2-1] = 'queue:' .. v; a[i*2] = redis.call('llen', 'resqueue:gitlab:queue:' .. v) end; return a" 0

Do you need some PR ?

[danielnelson]

We do need someone to put together a pull request still, but let's finalize an example plugin configuration first. Can you mock this up and show how the output would look given some input?

[athoune]

Something like that :

# Read Redis's basic status information
[[inputs.redis]]
  ## specify servers via a url matching:
  ##  [protocol://][:password]@address[:port]
  ##  e.g.
  ##    tcp://localhost:6379
  ##    tcp://:[email protected]
  ##
  ## If no servers are specified, then localhost is used as the host.
  ## If no port is specified, 6379 is used
  servers = ["tcp://localhost:6379"]

  [[inputs.redis.command]]
  	command = "LLEN toto"
  	tag = "name=bob"
  	field = "size" # default is "value"
  	measurement = "redis_toto" # default measurement is introspected, here, it's "toto"
  	# will become : redis_toto,name=bob size=42

  [[inputs.redis.command]]
  	commands = [
  		"GET resque:gitlab:stat:failed",
  		"GET resque:gitlab:stat:processed"
  	]
  	template_separator = ":"
  	separator = "_" 
  	templates = [
  		"measurement:queue:field:status",
	]
	# resque,queue=gitlab,status=failed stat=146

  [[inputs.redis.command]]
  	command = "HGETALL tata"
  	# With hash, measurement is key, and can use templates

Reusing things like graphite path templates.

For EVAL command, some introspection will be required : one response must be a number, multiple responses must be hash, with number as values.

[danielnelson]

Interesting idea to use the graphite template system. One thing I think might be a challenge is making this work with a proper redis client library. The current plugin implements the protocol manually, but I'd like to switch to a redis library and we have several in progress redis plugins using the go-redis library.

Just thinking aloud, but if we used this library we would need to convert the string "GET foo" to client.Get("foo"), which would be a bit of maintenance headache, depending on how many commands we want to support. Maybe we will need to EVAL all commands? This might make determining the return type harder.

Another trick would be what argument to use as the key, since not all commands have a key or have it as the first argument. We should also try to encourage using field names instead of many measurement names.

It will be a bit of work to extract the graphite template code to a location where it can be used, but this is work we are already wanting for other tasks, namely the dropwizard parser.

Other than that I would just say let's make this an independent plugin instead of being part of the redis plugin.

[athoune]

redis-go expose direct command access with https://godoc.org/github.com/go-redis/redis#NewStringCmd . Commands are described here https://github.com/antirez/redis/blob/unstable/src/server.c#L75 . It's easy to generate a go file, from an extract of "r" commands.
The hard part is return typing. Hum, using go-redis source is the best way.

For read commands, key seems to be always the first argument.

Would you like some code example, generated from go-redis and redis.c sources ?

[danielnelson]

The hard part is return typing.

It seems that we need to push this to the Telegraf config. There is no defined return type for the GET command, well maybe you could consider it a string, but then you will want the ability to convert to int/float anyway. So we could do something like this:

[[inputs.redis_command]]
  [inputs.redis_command.tag]
  	commands = ["GET c"]

  [inputs.redis_command.int]
  	commands = ["GET a"]

  [inputs.redis_command.float]
  	commands = ["GET b"]

I'm not sure how we would handle hashes though, this seems tricky especially if it contains mixed types.

For read commands, key seems to be always the first argument.

There are a few exceptions to the first arg is key rule: DBSIZE, CONFIG GET, KEYS, but maybe it is close enough, and some commands operate on several keys such as EXISTS.

Here is an alternative config that is similar to the proposed xml input configuration but does require you to specify the field name manually:

[[inputs.redis_command]]
  [inputs.redis_command.tag]
  	a = "GET a"

  [inputs.redis_command.int]
  	b = "GET b"

  [inputs.redis_command.float]
  	c = "GET c"

# redis_command,a=1 b=2i,c=3

Would you like some code example, generated from go-redis and redis.c sources?

Yeah sure, lets see how it shakes out.

[athoune]

Would you like some code example, generated from go-redis and redis.c sources?

Yeah sure, lets see how it shakes out.

The result is strange, some commands are missing :

https://github.com/athoune/redis-moi-le

[danielnelson]

I think this problem could be very complex if we don't limit our scope, and I don't want it to be a maintenance burden. With that in mind, I think it makes sense to only support int/float/string single value return types, at least in an initial plugin. We could do this all with NewStringCmd as you mentioned and the calling the requested conversion function. What do you think about the last config example?

[athoune]

I think this problem could be very complex if we don't limit our scope

Sure.

What do you think about the last config example?

Explicit key name for simple return is more predictable than guessing with redis command.

Automagical introspection is a trap. But, initial question was about complex output, from an EVAL : #1027 (comment)

For correct point in time values, MULTI/EXEC or EVAL must be used.
Simple commands can use implicit transaction, with explicit name and type as you suggest.

EVAL is important for handling dynamic responses. Something more naïve, like httpjson does : everything is a float, and more naïve, complex value can only be key => float value.

[danielnelson]

I think we can use NewStringCmd to pass an eval command and parse it in the same way so long as we know from the config if it should be an int/float/string.

I wonder if we need support for a list of commands per key? I would like to think of a concrete example when we would need this but the commands could be pipelined which would support simple multi/exec:

  [inputs.redis_command.int]
    value = ["SET value 42", "GET value"]

I think treating hash responses as key -> float would be a reasonable solution, then we need to choose between using NewStringCmd and NewStringStringMapCmd.

There is also a format similar to what we use in the snmp plugin, this might allow for converting all values in a hash to a particular type:

  [inputs.redis_command.value]
    name = "value"
    commands = ["SET value 42", "GET value"]
    conversion = "int"

  [inputs.redis_command.hash]
    name = "hash_values"
    commands = ["HGETALL foo"]
    conversion = "float"

# redis_command value=42i
# hash_values x=1,y=2,z=3

[athoune]

You can do pipeline with differents inputs, you list questions and get a list of answers https://redis.io/commands/exec

I can't imagine an example with a metric as last value. If you need to do things, then return value, EXEC is a better way.

With a hash, pattern on key for typing is simpler, with a default float type.

[danielnelson]

I can't imagine an example with a metric as last value. If you need to do things, then return value, EXEC is a better way.

I don't follow this part, can you give an example of how the config would look for MULTI/EXEC?

[athoune]

I don't follow this part, can you give an example of how the config would look for MULTI/EXEC ?

Oups, I wrote EXECand meant EVAL

[danielnelson]

I think I'm partial to placing the return type in the table name, it just seems a little more compact. One downside is it is a little bit less extensible...

[[inputs.redis_command]]
  # Using the redis:// urls that are accepted by go-redis
  servers = ["redis://localhost:6379"]

  [inputs.redis_command.tag]
    hostname = "GET hostname"
    address = "GET address"

  [inputs.redis_command.int]
    answer = "GET value"

  [inputs.redis_command.hash]
    stats = "HMGETALL stats"

redis_command,hostname=foo,address=127.0.0.1 answer=42i,stats_x=1,stats_y=2,stats_z=3

[brendan-rius]

That would be a very cool feature indeed.
My personal use case is limited to watching Redis lists length because I use them as brokers.

Any work being done on this right now?

[danielnelson]

@brendan-rius I'm not aware of anyone working on this, is this something you are interested in helping with?

[yquansah]

Hello All,

This PR was merged around 4-5 weeks ago: #8196 to extend redis capabilities.
Let me know if this works for y'all.

[sjwang90]

Closed by #8196