proposal: mesos-dns HTTP API watches

[jdef]

some clients will want to know right away when service records change. it would be ideal if they could watch (ala k8s, or consul) a service for changes. this would address a major pain point of typical DNS systems -- latency between record updates when apps move across hosts (or otherwise experience a shift in IPs).

/cc @karlkfi

[tsenart]

While I understand the use case, I have a few remarks regarding this idea in the context of Mesos-DNS:

1. Due to lack of better options, Mesos-DNS currently needs to periodically poll the state of the Mesos master. This inherently delays state propagation.
2. With TTL=0 the cost of an extra-round trip to the DNS server is still incurred at request time but it effectively eliminates staleness, just like watches.
3. Mesos-DNS is a DNS server and I'd like to keep it that way. Adding watch semantics to a DNS server over HTTP is smelling like feature creep. If we want a way to push Mesos state changes downstream, we should build that separately. In addition, at least in DCOS context, we're evaluating Consul for service discovery which, if it moves forward, would effectively provide this functionality with its Raft backed K/V store, not their DNS server component.

[jdef]

Some thoughts:

R(1): The delayed state propagation is a problem for DNS and
HTTP/watch. This is an orthogonal issue, should be resolvable once mesos
master supports event streams for non-framework clients (WIP). The nice
thing about watch semantics is that as soon as state propagation happens,
service record updates could be immediately sent downstream (vs. client
polling (poor scale) or client caching (TTL)).

R(2): TTL=0 is fine except for (a) broken DNS clients that ignore TTL, and;
(b) when your cluster is very large and clients are constantly querying the
DNS system because TTL=0 and they can't effectively cache record lookups at
all. watch semantics don't have this particular problem: TTL's can be
non-zero.

R(3): Mesos-DNS implements an HTTP API (currently) for a reason: not all
clients speak SRV but they do want access to service records. Adding
support for 'watch'-style semantics isn't a huge leap to make, doesn't add
a new protocol to mesos-dns (HTTP already supported), and from an API
perspective could be an extension to the existing endpoints (these could
accept a watch=true parameter if they wanted streaming updates).

More over, I think that it's reasonable to imagine a world wherein each
slave node has a mesos-dns-agent that coalesces watches so that Tasks could
simply establish a service watch directly on the mesos-dns-agent, which
establishes a single connection to a "master" mesos-dns instance (meaning,
it's a top-level instance that's consuming an event stream directly from
mesos master).

I'm really not convinced that the stateful store of a consul kv-backed
solution is the right answer. It's yet another consensus system on top of
a cluster that already requires at least two consensus systems to function
(ZK and Mesos). Proceed with a high degree of caution here.

On Thu, Sep 24, 2015 at 6:25 AM, Tomás Senart [email protected]
wrote:

While I understand the use case, I have a few remarks regarding this idea
in the context of Mesos-DNS:

Due to lack of better options, Mesos-DNS currently needs to
periodically poll the state of the Mesos master. This inherently delays
state propagation.
2.

With TTL=0 the cost of an extra-round trip to the DNS server is still
incurred at request time but it effectively eliminates staleness, just like
watches.
3.

Mesos-DNS is a DNS server and I'd like to keep it that way. Adding
watch semantics to a DNS server over HTTP is smelling like feature creep.
If we want a way to push Mesos state changes downstream, we should build
that separately. In addition, at least in DCOS context, we're evaluating
Consul for service discovery which, if it moves forward, would effectively
provide this functionality with its Raft backed K/V store, not their DNS
server component.

—
Reply to this email directly or view it on GitHub
#283 (comment)
.

[tsenart]

It seems my email reply didn't get here earlier today. Replying inline:

Some thoughts:

R(1): The delayed state propagation is a problem for DNS and
HTTP/watch. This is an orthogonal issue, should be resolvable once mesos
master supports event streams for non-framework clients (WIP). The nice
thing about watch semantics is that as soon as state propagation happens,
service record updates could be immediately sent downstream (vs. client
polling (poor scale) or client caching (TTL)).

I understand the benefits. I was just noting that this issue affects this proposal.

R(2): TTL=0 is fine except for (a) broken DNS clients that ignore TTL, and;
(b) when your cluster is very large and clients are constantly querying the
DNS system because TTL=0 and they can't effectively cache record lookups at
all. watch semantics don't have this particular problem: TTL's can be
non-zero.

With watch semantics, do we have TTLs are all? That's a DNS concept, no?

R(3): Mesos-DNS implements an HTTP API (currently) for a reason: not all
clients speak SRV but they do want access to service records. Adding
support for 'watch'-style semantics isn't a huge leap to make, doesn't add
a new protocol to mesos-dns (HTTP already supported), and from an API
perspective could be an extension to the existing endpoints (these could
accept a watch=true parameter if they wanted streaming updates).

Mesos-DNS exposes an HTTP interface to the current Mesos "service registry" which at the moment is completely coupled with this DNS server. Architecturally, this is undesirable and ought to be decoupled. I'd be reticent to add this kind of features before that happens.

More over, I think that it's reasonable to imagine a world wherein each
slave node has a mesos-dns-agent that coalesces watches so that Tasks could
simply establish a service watch directly on the mesos-dns-agent, which
establishes a single connection to a "master" mesos-dns instance (meaning,
it's a top-level instance that's consuming an event stream directly from
mesos master).

Totally agree that a per-node agent architecture (a la Consul) is a very solid direction. That's one of the reasons that led me to evaluate Consul, which I'm currently doing. There are a number of challenges that rise in such scenarios, the first of which, scalable state propagation, is very elegantly solved with constant load Gossip protocols.

I'm really not convinced that the stateful store of a consul kv-backed
solution is the right answer. It's yet another consensus system on top of
a cluster that already requires at least two consensus systems to function
(ZK and Mesos). Proceed with a high degree of caution here.

I'm not convinced either (yet), but not for that reason. After evaluation, our findings will be well synthesised and made available for consumption.

With that said, Mesos-DNS is currently in hardening, testing and general quality improvement phase,
not architectural or functional evolution, at least until we ruled out the alternative solutions to service discovery in DCOS.

[jdef]

RE: mesos-dns project phase - understood.

My concerns re: consul stand. Gossip seems like overkill for this
particular problem. Unless you have a complex distributed system like
consul, and I really hope that we don't go down this path. It's such a
heavy hammer and adds unnecessary complexity: it is not "doing one thing
and doing it well". Let's make sure that we're thinking about huge scales
here.

On Thu, Sep 24, 2015 at 12:17 PM, Tomás Senart [email protected]
wrote:

It seems my email reply didn't get here earlier today. Replying inline:

Some thoughts:

R(1): The delayed state propagation is a problem for DNS and
HTTP/watch. This is an orthogonal issue, should be resolvable once mesos
master supports event streams for non-framework clients (WIP). The nice
thing about watch semantics is that as soon as state propagation happens,
service record updates could be immediately sent downstream (vs. client
polling (poor scale) or client caching (TTL)).

I understand the benefits. I was just noting that this issue affects this
proposal.

R(2): TTL=0 is fine except for (a) broken DNS clients that ignore TTL, and;
(b) when your cluster is very large and clients are constantly querying the
DNS system because TTL=0 and they can't effectively cache record lookups at
all. watch semantics don't have this particular problem: TTL's can be
non-zero.

With watch semantics, do we have TTLs are all? That's a DNS concept, no?

R(3): Mesos-DNS implements an HTTP API (currently) for a reason: not all
clients speak SRV but they do want access to service records. Adding
support for 'watch'-style semantics isn't a huge leap to make, doesn't add
a new protocol to mesos-dns (HTTP already supported), and from an API
perspective could be an extension to the existing endpoints (these could
accept a watch=true parameter if they wanted streaming updates).

Mesos-DNS exposes an HTTP interface to the current Mesos "service
registry" which at the moment is completely coupled with this DNS server.
Architecturally, this is undesirable and ought to be decoupled. I'd be
reticent to add this kind of features before that happens.

More over, I think that it's reasonable to imagine a world wherein each
slave node has a mesos-dns-agent that coalesces watches so that Tasks could
simply establish a service watch directly on the mesos-dns-agent, which
establishes a single connection to a "master" mesos-dns instance (meaning,
it's a top-level instance that's consuming an event stream directly from
mesos master).

Totally agree that a per-node agent architecture (a la Consul) is a very
solid direction. That's one of the reasons that led me to evaluate Consul,
which I'm currently doing. There are a number of challenges that rise in
such scenarios, the first of which, scalable state propagation, is very
elegantly solved with constant load Gossip protocols.

I'm really not convinced that the stateful store of a consul kv-backed
solution is the right answer. It's yet another consensus system on top
of
a cluster that already requires at least two consensus systems to function
(ZK and Mesos). Proceed with a high degree of caution here.

I'm not convinced either (yet), but not for that reason. After evaluation,
our findings will be well synthesised and made available for consumption.

With that said, Mesos-DNS is currently in hardening, testing and general
quality improvement phase,
not architectural or functional evolution, at least until we ruled out the
alternative solutions to service discovery in DCOS.

—
Reply to this email directly or view it on GitHub
#283 (comment)
.

[tsenart]

@jdef: We're weighting pros and cons. There's nothing set in stone.