incompatible_depset_union: Disallow union operator on depsets

[c-parsons]

This is a tracking issue for offering a migration solution for
--incompatible_depset_union

Using +, | or .union on depsets is deprecated. Users should use the new
constructor instead (see https://docs.bazel.build/versions/master/skylark/depsets.html)

[dslomov]

What is missing for migration: migration docs, length of migration window. After these are done, please add "migration-ready" label.

[davido]

@laurentlb @c-parsons

It turns out that depset union feature is used very frequently in Gerrit Build tool chain. For example when running latest buildifier (0.20.0) in warn mode, we get this (on stable-2.14 branch in gerrit tree):

gerrit $ find . \( -name BUILD -o -name "*.bzl" \) -print | xargs buildifier --lint=warn
./tools/bzl/javadoc.bzl:23: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/javadoc.bzl:24: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/gwt.bzl:204: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/gwt.bzl:205: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/gwt.bzl:208: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/gwt.bzl:210: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/pkg_war.bzl:81: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/pkg_war.bzl:83: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/pkg_war.bzl:92: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/pkg_war.bzl:104: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/pkg_war.bzl:106: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:146: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:150: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:153: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:157: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:198: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:246: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:250: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/js.bzl:254: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/classpath.bzl:5: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/classpath.bzl:6: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)
./tools/bzl/classpath.bzl:8: depset-union: Depsets should be joined using the depset constructor. (https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#depset-union)

It also seems that at least in context of Polymer/Closure custom rules, that were written by @hanwen , it is not really trivial to avoid/re-write usage of depset-union feature.

I wonder if someone from Bazel team could help us to resolve those usages? Or give us more time and postpone deprecation of that feature? Or even re-consider and abandon the deprecation entirely?

Thanks!

[hanwen]

AFAICT, this is just a syntactic change, where we can change

a.union(b)

to

a = depset(transitive = [a ,b])

[dslomov]

Is it something we could easily implement in buildifier @c-parsons @vladmos ?

[laurentlb]

Hanwen's solution is the simplest fix.

However, in many cases, the depsets are not properly used. For example, using union or + in a loop can lead to performance issues (https://docs.bazel.build/versions/master/skylark/performance.html#use-depsets).

If we update the code automatically, most users won't have the chance to look at the code and decide if it's a good thing to do.

For example,

d = depset()
for item in mylist:
  d += item.field

may be instead transformed to:

d = depset(transitive = [item.field for item in mylist])

[c4urself]

Truly don't understand this change, if the workaround is a = depset(transitive=[a, b]) why not just make that the implementation for .union() or + behind the scenes instead of making people jump through hoops every few releases with these nonsensical changes. Python is the inspiration for Skylark and these changes are making everyone write/change more code for no reason (less Pythonic, more Javaesque)...

[laurentlb]

Let me expand from my previous comment. Each call to depset has a cost. The goal is to make that cost more obvious for users. We did this change because the old behavior was the cause of lots of performance issues.

Old code (now forbidden):

all_deps = depset()
for dep in ctx.attr.deps:
    all_deps = all_deps + dep.files

Discouraged fix:

all_deps = depset()
for dep in ctx.attr.deps:
    all_deps = depset(transitive = [all_deps, dep.files])

Recommended fix:

all_deps = depset(transitive = [dep.files for dep in ctx.attr.deps])

In some cases, keeping the loop is useful (e.g. if you have a condition or complex constructs). You can also accumulate the dependencies in a list, before constructing the depset:

deps_builder = []
for dep in ctx.attr.deps:
    deps_builder.append(dep.files)

all_deps = depset(transitive = deps_builder)

For a Python user, the old code might look fine and many people were writing this kind of code. If you blindly replace .union by its equivalent, the problem is more apparent. You are creating lots of depsets (it will be slow to iterate on those). The suggested fix is not longer and performs better.

So the design decision was made to prevent this tempting error.

[c4urself]

Hmm, that last example makes a lot more sense to me -- will update to that, thanks for the quick reply!