Missing connections in ECS demo

[2opremio]

I am missing edges when using Scope 0.9.0 with the Weave ECS guide demo (using weave 1.2.0 with WEAVE_NO_FASTDP).

I tested this before the 0.9.0 release so I am not sure if there was a problem with my testing then or a non-systematic problem (I have been able reproduce it the two times I tried today though ...)

Here is what I observe with Scope v0.9.0:

The edge between the internet node and the httpserver is correctly captured, however, edges are missing between the httpserver and the dataproducers.

Report: https://gist.github.com/2opremio/f257d739377416aabdc4

And here is the expected result, which I observe with commit 1577b90 (I picked this one because I knew master was showing these edges around Oct 15th for my talk at AWS Container day)

Report: https://gist.github.com/2opremio/2793dae94ed499f97c83

I replaced Scope without destroying the instances so it may make sense to diff the reports (IPs etc ... should be the same).

Similar to #446 and #489

[tomwilkie]

Interestingly, tests have started failing today, even though they worked last night (with the same code):

https://circleci.com/gh/weaveworks/scope/1981

This points to it not being scope, but some dependancy. Will investigate.

[tomwilkie]

Test failure was coincidence, unrelated.

The only missing thing from the report for the bug is that containers have no weave ip addresses. Did scope start before weave? The scope logs will show this.

[2opremio]

I am closing this. As @tomwilkie suspected, it seems to be due to #510 since Scope can start after Weave in the Weave ECS AMI (their Upstart jobs run in parallel), which is also a good explanation for why I couldn't see this problem before.

I have created weaveworks/integrations#35 to work this around in the ECS AMI.

[2opremio]

One last clarification, there was nothing special about 1577b90 and it's not free from this problem.

The only reason for which it worked was because I replaced the Scope without restarting weave. Since I started 1577b90 after Weave, it worked (and the same would had happened if I simply restarted Scope 0.9)