Listen for metrics on a separate port than the API

[mwhittington21]

Currently the API is unauthenticated so everyone who has access to the API port can get Flux to make changes to the cluster. Unfortunately, if you want metrics you have to expose that very same port (3030 by default) to the network. This is not a good idea for security.

This issue is about moving the /metrics endpoint to its own port so that it can be exposed separately.

[squaremo]

Yep, good point, thanks for bringing this up.

[ariefrahmansyah]

Hi, I would like to do this. Can you give me some guidance on how we want to do this?
I'm thinking to add a new flag for this, what is the better name for the flag? And what's the default port for /metrics?

[squaremo]

Hi, I would like to do this.

Splendid!

An outline of what needs doing:

• make an argument for the metrics port; so that it's not a breaking change, we may need to make it "opt in", i.e., if you supply the argument, it will serve metrics on the different port, otherwise it will do as it does now. I suggest the name --listen-metrics for the argument, since we have --listen already.

When the argument is supplied:

• in cmd/fluxd/main.go (right at the end), create another HTTP listener and add the promhttp.Handler() to that instead of the default mux. I forget the exact details of how listeners and muxers fit together -- consult the godocs :-)

Things to watch out for:

• at present the listener runs in a go routine and sends its (eventual) result to a channel to trigger shutdown; the metrics listener should probably do the same.
• the example deployment should be updated to use the new argument, and to declare the port, and to have the annotation that tells prometheus to scrape that port.

[squaremo]

Done in #1325.