Use conntrack to detect short lived connections. #386

tomwilkie · 2015-08-21T14:11:52Z

Fixes #356

Status: Good to go. Review feedback welcome.

Steps are:

Refactor the rendering pipeline such that mappers can emit multiple nodes
Use conntrack to populate the endpoint table (in addition to procspy). NB nodes from conntrack will not have a pid
Enhance the container render with an extra pipeline which joins containers to endpoints via their IP addresses

Makefile

+$(DOCKER_DISTRIB):
+	curl -o $(DOCKER_DISTRIB) $(DOCKER_DISTRIB_URL)
+
+docker/weave:


peterbourgon · 2015-08-24T16:36:28Z

To be clear: short-lived connections are visible, for processes that are running in containers, and talking to other containers?

probe/endpoint/reporter.go

 	"strconv"
 	"time"

 	"github.com/prometheus/client_golang/prometheus"
+	"github.com/typetypetype/conntrack"


tomwilkie · 2015-08-24T18:40:40Z

To be clear: short-lived connections are visible, for processes that are running in containers, and talking to other containers?

Short lived connections will only be visible for containers with a unique IP address in the containers and container image view. This rules out some of the kubernetes use cases as containers share a network namespace and IP address (although we can do the same thing for pods in the k8n views, so its not too bad), and also rules out --net=host containers (ie scope itself).

These short lived connections will not be visible in the process view (nothing to join them on). We could back-propagate the links for singleton process in containers (again, not scope).

Container-to-container links are covered, as are/will be pseudo nodes (both random ips and the internet node)

probe/endpoint/conntrack.go

+	} else if err != nil {
+		log.Printf("conntrack error: %v", err)
+		return
+	}


probe/endpoint/conntrack_test.go

+		})
+		return result
+	}
+	ts := 1000 * time.Millisecond


tomwilkie · 2015-08-27T12:21:01Z

I solved #403 by ensure each edge is only included once, by construction.

render/mapping.go

@@ -48,20 +45,20 @@ type PseudoFunc func(srcNodeID, dstNodeID string, srcIsClient bool, local report
 //
 // As with LeafMapFunc, if the final output parameter is false, the node
 // shall be omitted from the rendered topology.
-type MapFunc func(RenderableNode) (RenderableNode, bool)
+type MapFunc func(RenderableNode) RenderableNodes


peterbourgon · 2015-08-27T16:28:32Z

After reading this a little bit, very eager to go ahead and merge RenderableNode/NodeMetadata/.... as we discussed...

…g it for NAT mappings. Also use conntrack to populate the endpoint table.

… endpoints, via IP address foreign key.

tomwilkie · 2015-08-27T16:31:44Z

Yeah I've got a WIP branch to get there - https://github.com/weaveworks/scope/tree/357-adjacencies-to-nodemetadata

Will do it tomorrow.

probe/endpoint/conntrack.go

+	"strings"
+	"sync"
+
+	"github.com/weaveworks/scope/test/exec"


peterbourgon · 2015-08-27T17:25:49Z

Alright, those comments aside, this LGTM-ish. It's a shame that we have to exploit IP-per-container to get this info, but now that I understand it, I think it's reasonable enough.

tomwilkie · 2015-08-27T17:34:19Z

Thanks! Any suggestions on where to put exec? Maybe a lib package?
Theres more to follow; we've got bits and pieces of a mock filesystem that
should probably be brought together.

On Thu, Aug 27, 2015 at 6:25 PM, Peter Bourgon [email protected]
wrote:

Alright, those comments aside, this LGTM-ish. It's a shame that we have to
exploit IP-per-container to get this info, but now that I understand it, I
think it's reasonable enough.

—
Reply to this email directly or view it on GitHub
#386 (comment).

peterbourgon · 2015-08-27T21:35:10Z

Any suggestions on where to put exec?

common/{exec,foo,bar}?

Use conntrack to detect short lived connections.

tomwilkie self-assigned this Aug 21, 2015

tomwilkie force-pushed the 356-conntrack branch 3 times, most recently from c68bb82 to 0cc54b0 Compare August 24, 2015 12:18

2opremio reviewed Aug 24, 2015
View reviewed changes

tomwilkie force-pushed the 356-conntrack branch from 44f7b78 to 35d8c02 Compare August 24, 2015 13:59

tomwilkie mentioned this pull request Aug 24, 2015

We need a way to capture short lived connections. #356

Closed

tomwilkie force-pushed the 356-conntrack branch from 35d8c02 to a7d4350 Compare August 24, 2015 15:33

peterbourgon reviewed Aug 24, 2015
View reviewed changes

probe/endpoint/reporter.go

"strconv"

"time"

"github.com/prometheus/client_golang/prometheus"

"github.com/typetypetype/conntrack"

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

tomwilkie force-pushed the 356-conntrack branch from a7d4350 to d5516c7 Compare August 25, 2015 09:25

tomwilkie mentioned this pull request Aug 25, 2015

Add GET /ip handler to return all IPAM mappings. weaveworks/weave#1350

Merged

tomwilkie force-pushed the 356-conntrack branch 5 times, most recently from 5155a9d to eb30946 Compare August 26, 2015 13:36

tomwilkie force-pushed the master branch 2 times, most recently from 66be8bd to d9ab9d8 Compare August 26, 2015 14:25

tomwilkie force-pushed the 356-conntrack branch from eb30946 to 46b1a6f Compare August 26, 2015 14:30

tomwilkie changed the title ~~[WIP] Use conntrack to detect short lived connections.~~ Use conntrack to detect short lived connections. Aug 26, 2015

tomwilkie force-pushed the 356-conntrack branch 2 times, most recently from 532a8b3 to 2a56ec9 Compare August 26, 2015 16:46

tomwilkie mentioned this pull request Aug 26, 2015

We double count edge meta data is we merge in the endpoints topology more than once. #403

Closed

peterbourgon reviewed Aug 27, 2015
View reviewed changes

probe/endpoint/conntrack.go

} else if err != nil {

log.Printf("conntrack error: %v", err)

return

}

This comment was marked as abuse.

Sign in to view

peterbourgon reviewed Aug 27, 2015
View reviewed changes

probe/endpoint/conntrack_test.go

})

return result

}

ts := 1000 * time.Millisecond

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

tomwilkie force-pushed the 356-conntrack branch 2 times, most recently from 2b44189 to b805e84 Compare August 27, 2015 12:19

tomwilkie removed their assignment Aug 27, 2015

tomwilkie assigned peterbourgon Aug 27, 2015

tomwilkie force-pushed the 356-conntrack branch from b805e84 to e53ae8d Compare August 27, 2015 15:38

peterbourgon reviewed Aug 27, 2015
View reviewed changes

Tom Wilkie added 3 commits August 27, 2015 16:29

Give mapping functions the ability to return multiple nodes.

e4970f9

Use an exec'd conntrack in 'events' mode instead of repeatedly execin…

1a286ea

…g it for NAT mappings. Also use conntrack to populate the endpoint table.

Enhance the containers topology with information directly joined from…

55c77c6

… endpoints, via IP address foreign key.

tomwilkie force-pushed the 356-conntrack branch from e53ae8d to 55c77c6 Compare August 27, 2015 16:30

peterbourgon reviewed Aug 27, 2015
View reviewed changes

probe/endpoint/conntrack.go

"strings"

"sync"

"github.com/weaveworks/scope/test/exec"

This comment was marked as abuse.

Sign in to view

peterbourgon assigned tomwilkie and unassigned peterbourgon Aug 27, 2015

Review feedback

903a513

tomwilkie force-pushed the 356-conntrack branch from 27d3266 to 903a513 Compare August 28, 2015 09:20

tomwilkie added a commit that referenced this pull request Aug 28, 2015

Merge pull request #386 from weaveworks/356-conntrack

9edd507

Use conntrack to detect short lived connections.

tomwilkie merged commit 9edd507 into master Aug 28, 2015

tomwilkie deleted the 356-conntrack branch August 28, 2015 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use conntrack to detect short lived connections. #386

Use conntrack to detect short lived connections. #386

tomwilkie commented Aug 21, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

peterbourgon commented Aug 24, 2015

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Aug 24, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Aug 27, 2015

This comment was marked as abuse.

peterbourgon commented Aug 27, 2015

tomwilkie commented Aug 27, 2015

This comment was marked as abuse.

peterbourgon commented Aug 27, 2015

tomwilkie commented Aug 27, 2015

peterbourgon commented Aug 27, 2015

Use conntrack to detect short lived connections. #386

Use conntrack to detect short lived connections. #386

Conversation

tomwilkie commented Aug 21, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

peterbourgon commented Aug 24, 2015

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Aug 24, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Aug 27, 2015

This comment was marked as abuse.

peterbourgon commented Aug 27, 2015

tomwilkie commented Aug 27, 2015

This comment was marked as abuse.

peterbourgon commented Aug 27, 2015

tomwilkie commented Aug 27, 2015

peterbourgon commented Aug 27, 2015