[dnm,wip] *: bazel-ify attempt

[irfansharif]

Don't look, seeing what sticks.

Release note: None

[cockroach-teamcity]

This change is 

[petermattis]

@irfansharif I looked. I salute your bravery.

Reviewable status: complete! 0 of 0 LGTMs obtained

[irfansharif]

Not checking in the gazelle generated files just yet (just so the diffs here are understandable). I haven't quite figured out the right incantation to get bazel+gazelle to simply pick up what's present under vendor without generating BAZEL.build files for everything under vendor/. So far I got the simplest thing running.

# clears out all the gazelle-generated BUILD files
make bazel-reset 

# generates BUILD files, in one of two methods: 
# - raw `gazelle`, or 
# - through `bazelle` controlling gazelle behavior through directives
make {bazel,gazelle}-generate 

# build and run pkg/cmd/uptodate
bazel run //pkg/cmd/uptodate:uptodate

[kenliu]

exciting to see you taking a swing at this @irfansharif !

[irfansharif]

Ok got the vendor thing figured out. Gonna check out for a bit, but trying to get the following running next (building roachtest/roachprod through bazel):

make bazel-clean; make gazelle-generate; bazel run //pkg/cmd/roachtest:roachtest -- -h

Which runs into:

ERROR: /Users/irfansharif/Software/src/github.com/cockroachdb/cockroach/pkg/server/BUILD.bazel:3:11: no such package '@io_etcd_go_etcd//raft': BUILD file not found in directory 'raft' of external repository @io_etcd_go_etcd. Add a BUILD file to a directory to mark it as a package. and referenced by '//pkg/server:go_default_library'
ERROR: Analysis of target '//pkg/cmd/roachtest:roachtest' failed; build aborted: Analysis failed
INFO: Elapsed time: 2.012s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (306 packages loaded, 1180 targets configured)
FAILED: Build did NOT complete successfully (306 packages loaded, 1180 targets configured)

[irfansharif]

With some more gross hacks in I'm now about build cockroach (and everything else under bin) through bazel:

# nuke everything
$ make maintainer-clean

# generate all the c-deps, and generated code (could also use `make c-deps;
# make generate` which are sufficient for `buildshort`).
$ make build

# need this to generate BUILD files for ui (there are better ways to do it -
# this is also how we generated all the BUILD files in the first place so it's
# nice and hermetic)
$ make bazel-generate

# basically `make buildshort`
$ bazel build @cockroach//pkg/cmd/cockroach-short

# basically `make build`
$ bazel build @cockroach//pkg/cmd/cockroach

# could also do this right away
$ bazel run @cockroach//pkg/cmd/cockroach -- start-single-node --insecure
INFO: Analyzed target //pkg/cmd/cockroach:cockroach (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/cockroach:cockroach up-to-date:
  bazel-bin/pkg/cmd/cockroach/cockroach_/cockroach
INFO: Elapsed time: 0.144s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
*
* WARNING: RUNNING IN INSECURE MODE!
*
* - Your cluster is open for any client that can access <all your IP addresses>.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
# ...

(That was a bit of a lie, I'm not building the C++ deps or generating code, but they're doable).

So what's "remaining" is getting all the c-dependency knowledge captured within bazel, and set up those custom build rules. Right now it just expects the find the artifacts to be present and will fail to link them if not. We'd also want to get the equivalent of make generate (protos and other random generated code things we do today) into bazel by writing custom genrules: https://docs.bazel.build/versions/3.4.0/be/general.html#genrule.cmd. These two things should let us start stubbing out our makefile targets with bazel underneath the hood.

There's some funkiness going on with how CGO flags get interpreted into the generated BUILD files. I had to hack in the right include paths, and but they should be relative to repo root which didn't seem to work. Right now they're pointing paths on my macbook, so there are just a few lines you'd have to adjust to get it working on yours. They are in:

pkg/storage/BUILD.bazel
pkg/ccl/storageccl/engineccl/BUILD.bazel
pkg/geo/geos/BUILD.bazel
pkg/geo/geoproj/BUILD.bazel
vendor/github.com/knz/go-libedit/BUILD.bazel

Also our artifacts today get placed under $GOPATH/native/, which because it exists outside our repo root, might not play super well with Bazel's idea of a workspace (but I could be wrong). I think it wants everything to be addressable under the workspace/repo root.

One other thing that I'm not so sure about is how bazel/gazelle should treat the vendor directory. Right now I've let it bazel-ify everything, and I think it wants to, but I'm not sure. I took precedence from https://github.com/paulbellamy/golanguk2017 and few other links online but this seems like a wart. I'm sure there's a better way.

---

I think getting bazel to start working with the most common build paths our engineers use may not be as far away as we think. As a bonus, you can also run go tests through bazel now. Here's one of our more involved packages that makes use of datadriven.

$ bazel test @cockroach//pkg/kv/kvserver/concurrency/...
INFO: Analyzed 4 targets (19 packages loaded, 170 targets configured).
INFO: Found 3 targets and 1 test target...
INFO: Elapsed time: 40.434s, Critical Path: 40.16s
INFO: 32 processes: 32 darwin-sandbox.
INFO: Build completed successfully, 37 total actions
@cockroach//pkg/kv/kvserver/concurrency:go_default_test                  PASSED in 31.1s

INFO: Build completed successfully, 37 total actions

[irfansharif]

So, next steps here:

• For each of the c-dependencies, we're going to need to construct a BUILD file
  for it to represent the dependency. This should be able to shell out and run
  cmake or whatever else it is that we do today to compile those artifacts.
  We have various cgo linking directives today in our go code that expect to
  find these artifacts generated and placed in certain directories. These were
  used when auto-generating the BUILD.bazel files per directory. As long as
  we set up these per-pkg BUILD files to include the aforementioned
  c-depedencies as dependencies (which would hopefully generate the artifacts
  and place them at the right directories), it should just work.
• For every "codegen" facility we have today (optgen, execgen, protobuf), we're
  going to need to write custom build rules to generate those files and then
  depend against those targets before compiling CRDB. See genrule.
• In the branch above we let bazel take over the vendor directory and bazel-ify
  each dependency. I'm not sure that that's the best way to do it. What we want
  is to tell bazel to treat the vendor directory as an artifact of sorts
  (perhaps we we want to write custom build rules to check out the right
  submodule SHA), and just have the go compiler (as run through bazel) to look
  towards there without expecting everything to be similarly bazel-ified.
• Write custom build files to generate our UI assets.
• See comment above about the manual BUILD.bazel include path changes I had to
  make. I think they should be relative to repo root, but I couldn't get that
  to work. I hacked in an absolute path on my machine, but that'll need to get
  changed.
• Gazelle auto-generated the dependencies.bzl file from go.mod. We should make
  sure our process for bumping dependencies also appropriately update this file (or
  check that they're in-sync, something like teamcity-assert-clean.

Given we're proposing having a dedicated hack-week to get all this out the
door, we should first aim for the developer experience around our current
Makefile. We can invoke bazel underneath our Makefile targets (gated by an env
var USE_BAZEL=1) and aim for the following targets (roughly in the order we
should want to do them):

• make {generate,protobuf}
• make bin/{...}
• make build{,oss,short} (the latter is easiest)
• make clean{,short}, ui-{,maintainer-}clean, unsafe-clean
• make install
• make test (including supporting PKG=... TESTS=... TESTFLAGS=...)
• make bench (including supporting PKG=... BENCHES=...)
• make stress (including
• make testlogic
• ...everything else

At all times we should be generating artifacts that are byte-by-byte identical
with those we would otherwise generate through our current make-based system.

@jordanlewis, @otan, @petermattis: thoughts?

[otan]

We can invoke bazel underneath our Makefile targets (gated by an env var USE_BAZEL=1)

This seems messy given Makefile dependencies will probably have a bad time if you switch between them both. I'd rather the concept of make be gone when this ends -- instead of make commands, we run something like bazel tool //build:lint, or something of the ilk work (face it, who wants to maintain a Makefile :P).

[irfansharif]

That works for me, I guess the real motivation was to have eager folks try just one build system at a time and report badness as they come up. I'd be happy with just having the bazel equivalents as easy to remember shorthands.

[otan]

In the branch above we let bazel take over the vendor directory and bazel-ify
each dependency.

I think that's the correct philosophy long term. I haven't thought about this too deep.

I'm not sure that that's the best way to do it. What we want
is to tell bazel to treat the vendor directory as an artifact of sorts
(perhaps we we want to write custom build rules to check out the right
submodule SHA), and just have the go compiler (as run through bazel) to look
towards there without expecting everything to be similarly bazel-ified.

But open to this as a first step, if it's possible. But if it's easier to do the above, I'd prefer it.

---

In an ideal world, I envisioned we don't have vendor directories at all, and bazel will pull these down at build time. Ideally we can get rid of submodules entirely :P but I'm not familiar with bazel vendoring.

[irfansharif]

@ajwerner, @miretskiy, @knz: word around the block is y'all have some prior experience with bazel. Mind sanity checking our plan above? We're thinking to work on moving crdb over the bazel later this month. +cc @jlinder.

[knz]

have eager folks try just one build system at a time and report badness as they come up.

The Makefile fallback is important to keep around for more than just the eager folk - there will be enough combinations of build platforms, debug flags, testing commands etc which you won't have thought about and will need adjustments.

[irfansharif]

Subsumed by #55258 and then #55687.