cli: debug zip should continue past errors

[bdarnell]

Some of the information in debug zip requires the cluster to be "up" and able to serve consistent reads. But much of it does not (for example, lower-level details like goroutine stacks), and these parts are often the most useful. The debug zip command currently hangs on clusters that are unavailable. Instead, it should try to gather as much as it can and log what it couldn't get. Every step in the data collection should have a timeout, and a timeout shouldn't prevent the rest of the process from being run.

[dianasaur323]

Thanks Ben - I think we also want to add heap profiles to the debug zip, so perhaps we do these two issues together in one go.

[petermattis]

This should be relatively straightforward. I might try to get a PR out tomorrow.

@dianasaur323 Heap profiles have already been added to debug zip.

[dianasaur323]

Thanks for the heads up, woohoo!

On Fri, Mar 16, 2018 at 7:25 PM Peter Mattis ***@***.***> wrote: This should be relatively straightforward. I might try to get a PR out tomorrow. @dianasaur323 <https://github.com/dianasaur323> Heap profiles have already been added to debug zip. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23954 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFgDWDPBR8iof77VhWH8D4BxtVvtrxNIks5tfEn1gaJpZM4St4Ex> .

-- Diana Hsieh [email protected] 407-690-9048

[petermattis]

Adding timeouts to the various RPCs is easy, but doesn't get us very far. The problem is that statusServer.Nodes does a scan of the node status keys which will block if the range containing those keys is unavailable. I think I need to do a larger restructuring so that we determine the node IDs via the Gossip endpoint.

[petermattis]

The problem with statusServer.Nodes will also affect the node status command. Do we keep an in-process cache of the NodeStatus descriptors? Note that gossip keeps a cache of the NodeDescriptor which is different. For debug zip we could get away with inspecting gossip, but node status actually uses some of the fields from NodeStatus.

[tbg]

You may want to dump the `crdb_internal.gossip_*` tables instead. It's not all the same, but it's always available. They were introduced to address similar shortcomings in the cli status commands that also rely on `Nodes`.

On Mon, Mar 26, 2018 at 3:27 PM Peter Mattis ***@***.***> wrote: The problem with statusServer.Nodes will also affect the node status command. Do we keep an in-process cache of the NodeStatus descriptors? Note that gossip keeps a cache of the NodeDescriptor which is different. For debug zip we could get away with inspecting gossip, but node status actually uses some of the fields from NodeStatus. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#23954 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE135OuZgNU_7gqnJof32LlRuKxNkTFxks5tiUEogaJpZM4St4Ex> .

--

…

-- Tobias

[petermattis]

@tschottdorf Thanks. Perhaps both debug zip and node status should use crdb_internal.gossip_nodes to get the list of node IDs (which internally uses gossip) and then use statusServer.Node to retrieve the node status. Seems workable.