⛵️ Move k8s.io/test-infra over to dep

[cblecker]

I have no idea how this will go, but let's try it!
This actually seems like it could work!

What I did:

dep init -no-examples
rm -rf Godeps/ _vendor-* vendor/
# Edit Gopkg.toml to ignore github.com/aws/aws-sdk-go* and remove it as a contraint
dep ensure && dep prune
verify/update-bazel.sh

@BenTheElder @stevekuznetsov @fejta @Q-Lee

[cjwagner]

/retest

[stevekuznetsov]

I wouldn't necessarily have expected the BUILD files to change or for us to pull in 200k more LoC -- is that a misinterpretation of the current vendor state by dep? Are we pulling in sub-packages we don't need?

[cblecker]

Doing this without the prune is even worse:

+6,512,203 −19,285

[spxtr]

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

[BenTheElder]

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md#ignored

[BenTheElder]

what changed here? why isn't it just "//vendor:all-srcs"?

[cblecker]

I updated the PR body with the commands I used. This was changed by verify/update-bazel.sh.

[cblecker]

Found what I needed to fix: kubernetes/kubernetes#51229

[BenTheElder]

this is why we should make verify/ match k/k hack/ :-P

[spxtr]

Probably we should add some basic instructions to the readme.

[cblecker]

Okay. So this works as-is. I was tinkering with also trying to break the dependency on k/k, as the imports we use are all ones that have been split off into the apimachinery/apiserver repos. However there was some bazel funkiness with that, so it'll have to wait for another PR. This one is big enough!

@kubernetes/test-infra-maintainers PTAL :)

[BenTheElder]

LGTM, thanks for all your work on this! Hopefully this will move us to dep sanity :-)
/lgtm
/hold
This PR is big enough that I'd love someone else to /lgtm and /hold cancel after reviewing it themseleves @cjwagner @Kargakis.

[krzyzacy]

/cc @ixdy @rmmh
LGTM, and try to get more eyes on this

[0xmichalis]

It seems that most of the additions are test code, BUILD, or other documentation files. There is also some interesting cleanup.

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, cblecker, kargakis

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• OWNERS [BenTheElder]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[stevekuznetsov]

100k LOC diff is still weird -- looks like we were not vendoring tests in the past, but now we are? Is that ok? Do we want them?

[stevekuznetsov]

Can we link to this from the main README so contributors can find it without digging too much?

[cblecker]

Done

[stevekuznetsov]

Of the additions, the majority are tests:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | wc -l
432
$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -E '_test.go$' | wc -l
281

Then we also pull in a lot of new non-code files:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | xargs -L 1 basename | sort | uniq -c | sort -rn | head -n 15
     18 README.md
     16 .gitignore
     12 CONTRIBUTING.md
      6 .travis.yml
      6 BUILD
      6 AUTHORS
      5 CONTRIBUTORS
      4 OWNERS
      4 Makefile
      4 .gitattributes
      4 codereview.cfg
      3 README
      3 CHANGELOG.md
      3 AUTHORS.md
      2 util.go

So updated vendored code is small:

$ git diff --diff-filter=AM --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | grep -E '.go$' | xargs wc -l
   829 vendor/github.com/influxdata/influxdb/client/influxdb.go
   501 vendor/github.com/influxdata/influxdb/client/v2/client.go
    44 vendor/github.com/influxdata/influxdb/errors.go
     1 vendor/github.com/influxdata/influxdb/influxdb.go
    27 vendor/github.com/influxdata/influxdb/models/inline_fnv.go
    38 vendor/github.com/influxdata/influxdb/models/inline_strconv_parse.go
  1916 vendor/github.com/influxdata/influxdb/models/points.go
   116 vendor/github.com/influxdata/influxdb/node.go
    95 vendor/github.com/influxdata/influxdb/pkg/escape/bytes.go
    39 vendor/github.com/petar/GoLLRB/llrb/avgvar.go
    93 vendor/github.com/petar/GoLLRB/llrb/iterator.go
    46 vendor/github.com/petar/GoLLRB/llrb/llrb-stats.go
   456 vendor/github.com/petar/GoLLRB/llrb/llrb.go
    17 vendor/github.com/petar/GoLLRB/llrb/util.go
    13 vendor/github.com/shurcooL/githubql/doc.go
    11 vendor/github.com/shurcooL/graphql/doc.go
    23 vendor/golang.org/x/crypto/curve25519/doc.go
    21 vendor/golang.org/x/crypto/ssh/doc.go
   156 vendor/golang.org/x/net/context/context.go
   341 vendor/golang.org/x/oauth2/oauth2.go
    76 vendor/golang.org/x/sys/unix/syscall.go
   100 vendor/golang.org/x/tools/go/gcexportdata/gcexportdata.go
  1041 vendor/golang.org/x/tools/go/gcimporter15/gcimporter.go
   112 vendor/google.golang.org/appengine/appengine.go
   355 vendor/google.golang.org/appengine/internal/urlfetch/urlfetch_service.pb.go
   210 vendor/google.golang.org/appengine/urlfetch/urlfetch.go
    20 vendor/k8s.io/kubernetes/pkg/util/doc.go
    56 vendor/k8s.io/kubernetes/pkg/util/string_flag.go
    48 vendor/k8s.io/kubernetes/pkg/util/template.go
    72 vendor/k8s.io/kubernetes/pkg/util/trace.go
    27 vendor/k8s.io/kubernetes/pkg/util/umask_windows.go
   147 vendor/k8s.io/kubernetes/pkg/util/util.go
  7047 total

So that actually makes me feel like this is a lot smaller of a move than it seems. Seems like we are updating those dependencies?

[cblecker]

@stevekuznetsov dep and godep function differently. When godep saves things to vendor/, it only copies the *.go code itself, and the LICENSE files (omits tests and non-go files). dep on the other hand, checks out the whole git repo to vendor. dep prune then prunes back to only the packages you're actually importing and using + LICENSE files.. but it leaves the whole directory (including non-go files and tests).

[cblecker]

/hold cancel

[stevekuznetsov]

Right -- the majority of this is due to test and non-code changes, but that is still a couple transitive version bumps in the dependencies from my quick search. Should be fine but did want to point it out.

[sdboyer]

also just noting over here that our prune logic is going to become a) automated b) more powerful and c) more granular pretty soon: golang/dep#944

[stevekuznetsov]

@sdboyer that sounds awesome!