Build a compiler extension and then use it

[gergelyfabian]

In rules_scala's scalac BUILD file, we have the following.

SCALAC_DEPS = [
    "//scala/private/toolchain_deps:scala_compile_classpath",
    "//src/java/io/bazel/rulesscala/io_utils",
    "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/jar",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/worker",
    "@io_bazel_rules_scala//src/protobuf/io/bazel/rules_scala:diagnostics_java_proto",
    "//src/java/io/bazel/rulesscala/scalac/compileoptions",
    "//src/java/io/bazel/rulesscala/scalac/reporter",
]

java_binary(
    name = "scalac",
    srcs = [
        ":scalac_files",
    ],
    javacopts = [
        "-source 1.8",
        "-target 1.8",
    ],
    main_class = "io.bazel.rulesscala.scalac.ScalacWorker",
    visibility = ["//visibility:public"],
    deps = (select_for_scala_version(
        any_2 = [
            "//third_party/dependency_analyzer/src/main/io/bazel/rulesscala/dependencyanalyzer/compiler:dep_reporting_compiler",
        ],
    ) if ENABLE_COMPILER_DEPENDENCY_TRACKING else []) + SCALAC_DEPS,
)

"//scala/private/toolchain_deps:scala_compile_classpath" is the critical part in my perspective. It's a:

common_toolchain_deps(
    name = "scala_compile_classpath",
    deps_id = "scala_compile_classpath",
    visibility = ["//visibility:public"],
)

That is in turn a:

_toolchain_type = "@io_bazel_rules_scala//scala:toolchain_type"

def _common_toolchain_deps(ctx):
    return expose_toolchain_deps(ctx, _toolchain_type)

common_toolchain_deps = rule(
    implementation = _common_toolchain_deps,
    attrs = {
        "deps_id": attr.string(mandatory = True),
    },
    toolchains = [_toolchain_type],
    incompatible_use_toolchain_transition = True,
)

Where:

def _required_deps_id_message(target, toolchain_type_label, deps_id):
    return target + " requires mapping of " + deps_id + " provider id on the toolchain " + toolchain_type_label

def java_info_for_deps(deps):
    return [java_common.merge([dep[JavaInfo] for dep in deps])]

def _lookup_provider_by_id(ctx, toolchain_type_label, dep_providers, deps_id):
    for dep_provider in dep_providers:
        if dep_provider[DepsInfo].deps_id == deps_id:
            return dep_provider
    fail(_required_deps_id_message(ctx.attr.name, toolchain_type_label, deps_id))

def find_deps_info_on(ctx, toolchain_type_label, deps_id):
    dep_providers = getattr(ctx.toolchains[toolchain_type_label], "dep_providers")

    print("find_deps_info_on dep providers: " + str(dep_providers))

    return _lookup_provider_by_id(ctx, toolchain_type_label, dep_providers, deps_id)[DepsInfo]

def expose_toolchain_deps(ctx, toolchain_type_label):
    deps_id = ctx.attr.deps_id
    deps_info = find_deps_info_on(ctx, toolchain_type_label, deps_id)
    print("expose_toolchain_deps %s deps_info: %s" % (toolchain_type_label, deps_info))

    return java_info_for_deps(deps_info.deps)

So it comes out that the scala toolchain is the only way to add custom dependencies for the scalac runner.

I guess I can have just a single Scala toolchain active at a time. I'd like to compile two types of targets:

1. A compiler extension (example is a wartremover extension), that would later be used as a compiler dependency
2. All other targets that would need that extension as a compiler dependency

For nr. 1 I need a toolchain that does NOT include the compiler extension (to build the compiler extension).
For nr. 2. I need a toolchain that does include the compiler extension.

The two requirements are mutually exclusive, as I understand, in rules_scala's current implementation.

Is that correct? And is there any way to extend rules_scala, to create another mechanism to provide additional (optional) dependencies for the scalac java_binary, that would be used just for some of the compiled targets?

[gergelyfabian]

I think a scala_library_for_plugin_bootstrapping would be very much in the direction of what I need, just that it also would use the compiler deps.

[gergelyfabian]

I'd need a way to provide a different set of deps for scalac_bootstrap (modify //scala/private/toolchain_deps:scala_compile_classpath to something else for it).

[liucijus]

Can it be solved with macros? Eg. scala_library_to_compile_extension and another like scala_library_with_extensions, which adds additional extension deps.

[gergelyfabian]

I doubt it can be solved with macros. What I'd need is two different versions of scalac java_binary, one that is using one set of dependencies, and another that uses another set of dependencies.

[gergelyfabian]

I managed to solve it. Steps:

1. Remove the compiler extension, that I want to build with Bazel, from my toolchain's compile dependencies (this enables compiling it, removing the cycle)
2. Create a scala_library_to_compile_extensions (thank @liucijus for the idea) that will work as previously my scala_library did
3. Create an extension (like one for phases, or scalafmt) and use that to generate all my other scala_library, etc. macros. This will override the _scalac used in those, to my own scalac, that will have the extension added as a dependency. Details below.

This works without changing rules_scala.

tools/compile-with-extensions/compile_with_extensions.bzl:

compile_with_extensions_custom_scalac = {
    "attrs": {
        "_scalac": attr.label(
            executable = True,
            cfg = "exec",
            # We need to explicitly specify the "main" repository, as this will be executed by rules_scala (otherwise it would
            # interpret it as being in the rules_scala repository).
            default = Label("@//tools/compile-with-extensions:scalac_with_extensions"),
            allow_files = True,
        ),
    },
    "phase_providers": [],
}

tools/compile-with-extensions/BUILD:

load("@io_bazel_rules_scala//scala:scala_cross_version_select.bzl", "select_for_scala_version")
load("@io_bazel_rules_scala_config//:config.bzl", "ENABLE_COMPILER_DEPENDENCY_TRACKING")

SCALAC_DEPS = [
    # Add my compiler extensions here...
    "@//my/compiler/extension:library",
    # The rest is copied from rules_scala...
    "@io_bazel_rules_scala//scala/private/toolchain_deps:scala_compile_classpath",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/io_utils",
    "@bazel_tools//src/main/protobuf:worker_protocol_java_proto",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/jar",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/worker",
    "@io_bazel_rules_scala//src/protobuf/io/bazel/rules_scala:diagnostics_java_proto",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac/compileoptions",
    "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac/reporter",
]

java_binary(
    name = "scalac_with_extensions",
    srcs = [
        "@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac:scalac_files",
    ],
    javacopts = [
        "-source 1.8",
        "-target 1.8",
    ],
    main_class = "io.bazel.rulesscala.scalac.ScalacWorker",
    visibility = ["//visibility:public"],
    deps = (select_for_scala_version(
        any_2 = [
            "@io_bazel_rules_scala//third_party/dependency_analyzer/src/main/io/bazel/rulesscala/dependencyanalyzer/compiler:dep_reporting_compiler",
        ],
    ) if ENABLE_COMPILER_DEPENDENCY_TRACKING else []) + SCALAC_DEPS,
)

My macro definitions:

load("//tools/compile-with-extensions:compile_with_extensions.bzl", "compile_with_extensions_custom_scalac")

scala_library = make_scala_library(..., compile_with_extensions_custom_scalac)
scala_library_to_compile_extensions = make_scala_library(...)
scala_macro_library = make_scala_macro_library(..., compile_with_extensions_custom_scalac)

To compile my extension:

-scala_library(
+scala_library_to_compile_extensions(

[gergelyfabian]

With the latest changes from #1677, tools/compile-with-extensions/BUILD is changed to:

load("@io_bazel_rules_scala//src/java/io/bazel/rulesscala/scalac:definitions.bzl", "DEFAULT_SCALAC_DEPS", "define_scalac")

EXTENDED_SCALAC_DEPS = DEFAULT_SCALAC_DEPS + [
    "@//utils/wartremover-extended:library",
]

define_scalac(
    name = "scalac_with_extensions",
    deps = EXTENDED_SCALAC_DEPS,
)