Properly truncate and randomly load balance large answers

[jdef]

wondering if we need some kind of max-record-count that limits the number if IPs returned for A records, or addresses returned for SRVs in large clusters (10k). for example, the "slave.mesos" name that maps to all slave IPs in the cluster probably doesn't scale very well to large cluster sizes. same goes for 'task.framework.name' for very large numbers of tasks with the same name.

what's the value in getting back 10k addresses or 10k IPs?

[tsenart]

@jdef: I think our answers should be limited to what fits in a DNS UDP datagram without truncation. I don't think we need a limit the number of IPs in the server memory, only in DNS answers. We should randomize the which IPs are included in the answer.

[discordianfish]

@tsenart But this would break for example the prometheus DNS based SD. That expects that we return all slaves.

[tsenart]

@discordianfish: We need to do this regardless of whether we support EDNS(0), but that should alleviate the problem for Prometheus, as long as it has EDNS(0) enabled DNS client. The upper bound of records we can have in the answer section is 65535 due to the ANCOUNT field in the header being an uint16. If the client won't support EDNS(0) then it could try over TCP, which is done by default in the native Go DNS client if the returned message over UDP is truncated. Over TCP, we could always serve up to 65535 answers in a single message.

[discordianfish]

@tsenart Yes, that's what we should do but you said above that we should always limit the number so it fits into UDP. Instead we should return all, even if this means we need to truncate and have clients fall back to tcp.

[tsenart]

@discordianfish: By definition, when you truncate, you can't return all :-)

[discordianfish]

Ok, let me rephrase: We should truncate the responses to whatever the client supports (512 without EDNS, whatever is indicated with EDNS) and 65535 records for TCP.

[tsenart]

👍

[sargun]

@discordianfish Is this what your PR that you merged today does?

[jdef]

#330 (merged today) should take care of proper truncation. is there a need
for a separate follow-up ticket re: the "randomly load balance" part?

On Tue, Nov 17, 2015 at 12:08 PM, Sargun Dhillon [email protected]
wrote:

Closed #237 #237.

—
Reply to this email directly or view it on GitHub
#237 (comment).

[sargun]

@jdef So, digging through the code, it looks like shuffleAnswers gets called in HandleMesos before reply, truncate in reply. This should take care of the "randomly load balance" requirement. What do you think?

P.S. perhaps truncate should be called in HandleMesos?

[jdef]

if we're already shuffling then it sounds like we're all set with this
ticket.

On Tue, Nov 17, 2015 at 3:45 PM, Sargun Dhillon [email protected]
wrote:

@jdef https://github.com/jdef So, digging through the code, it looks
like shuffleAnswers gets called in HandleMesos before reply, truncate in
reply. This should take care of the "randomly load balance" requirement.
What do you think?

P.S. perhaps truncate should be called in HandleMesos?

—
Reply to this email directly or view it on GitHub
#237 (comment)
.

[sargun]

😸