[docker_daemon] [ecs] introspection resilience

[truthbk]

Note: Please remember to review the Datadog Contribution Guidelines
if you have not yet done so.

What does this PR do?

ECS agent container no longer has the IP address or ports available when we inspect its container to infer the introspection point. The docker gateway, typically at 172.17.0.1 is fine for constructing the url to query the API. We can set it as a parameter in the config file, or we can extract it from the agent's own information.

Also, although rare, if the agent is not run dockerized, we can grab it @localhost

Motivation

Our ECS task reporting broke due to our inability to query the endpoint.

Testing Guidelines

An overview on testing
is available in our contribution guidelines.

Additional Notes

Anything else we should know when reviewing?

[truthbk]

Still untested - just verified manually. VERIFIED working - conatiner's default gateway works just fine because the ecs agent exposes its port.

[truthbk]

We might need some fallback for ipv6 only systems... Docker doesn't support ipv6-only, so we're good (for now).

[hkaj]

You could refactor the conditions to look like:

if not ip:
    port = ECS_INTROSPECT_DEFAULT_PORT
    if DockerUtil.is_dockerized() and self.docker_gateway:
        ip = self.docker_gateway
    else:
        ip = 'localhost'

which is 1 line shorter \o/ (kidding aside I think it's also a bit clearer)

[hkaj]

2 minor comments, feel free to merge once they're addressed 👍

[kuvia]

I've pulled the image docker-dd-agent:latest but this change is not in there yet. When can I expect an updated image in the docker hub?
Thanks!

[kuvia]

I've tried the nightly build and I get an exception in refresh_ecs_tags
2016-09-09 11:16:57 UTC | CRITICAL | dd.collector | utils.dockerutil(dockerutil.py:167) | Unable to find docker host hostname. Trying default route 2016-09-09 11:17:00 UTC | ERROR | dd.collector | checks.docker_daemon(__init__.py:787) | Check 'docker_daemon' instance #0 failed Traceback (most recent call last): File "/opt/datadog-agent/agent/checks/__init__.py", line 770, in run self.check(copy.deepcopy(instance)) File "/opt/datadog-agent/agent/checks.d/docker_daemon.py", line 245, in check self.refresh_ecs_tags() File "/opt/datadog-agent/agent/checks.d/docker_daemon.py", line 451, in refresh_ecs_tags if DockerUtil.is_dockerized() and self.docker_gateway(): TypeError: 'str' object is not callable

[hkaj]

Hi @himawri
That's a known issue, it's been addressed by #2825 which will go out in the next nightly.