[IntelliJ] Export only modulizable targets when in export-dep-as-jar

[blorente]

Problem

As an optimization for exporting source dependencies as jars, we want to export target entries only for targets that are either:

• Target Roots, or
• Between target roots (e.g. if A -> B -> C and we ./pants export-dep-as-jar :A :C, we want :B to also be in the output).

Solution

Split the function generate_targets_map into three parts:
1.- We iterate over all targets to create the resource map. This is O(|all_targets|)
2.- We iterate over all targets to fill up the graph_info['libraries'] section. This is also O(|all_targets|).
3.- We calculate the printable targets (that fulfil the constraints above), and populate targets_map with entries for those. Figuring out the printable targets boils down to a call to BuildGraph.transitive_dependees_of_addresses, which is a DFS graph traversal. Populating targets_map can be quadratic, because for each modulizable target we must walk its transitive dependency graph to include all of those dependencies as libraries.

TODO before merging:

• There are still a couple of failing tests, that will either have to be changed, or addressed before merging.
• We need to include library entries for all the transitive deps that are not included in the final json in the targets that we print, to import them as library dependencies in IntelliJ (as opposed to module dependencies).

Result

The output json will only include the targets that fulfil the constraints defined in Problem in the targets section, as opposed to all the targets in play.

[wisechengyi]

Thanks, @blorente! Looks reasonable overall. Please kindly see comments.

[blorente]

Addressed comments and updated description.

[blorente]

I found another problem:
(in the A->B->C->D ./pants export :A :C case) If I import a target from the IntelliJ UI (say A), and then add a project using the pants menu in IntelliJ, B will not be imported.
I think this will need some tweaking from the plugin side, very possibly to include all imported targets in calls to refresh (which should have negligible overhead)

[blorente]

This is now ready for review.

In particular the last commit (Modify tests to accomodate for the new structure) gives a good intuition on how the output has changed. (cc\ @wisechengyi @olafurpg)

[blorente]

Corresponding plugin changes: pantsbuild/intellij-pants-plugin#465

[blorente]

Before change (dependencies are modules):

After change (dependencies are libraries, only roots are modules):

Project structure:

The library looks like this:

[wisechengyi]

Found some issues after a closer look.

With export-dep-as-jar examples/tests/java/org/pantsbuild/example/hello/greet/ examples/src/java/org/pantsbuild/example/hello/greet/

The target relation is correct: https://gist.github.com/wisechengyi/75e00245542efdbae26abcf009936b64#file-gistfile1-txt-L6-L8

But examples.src.java.org.pantsbuild.example.hello.greet.greet should not be there (https://gist.github.com/wisechengyi/75e00245542efdbae26abcf009936b64#file-gistfile1-txt-L13), because examples/src/java/org/pantsbuild/example/hello/greet/ is already a target root.

Neither should these two entries exist: https://gist.github.com/wisechengyi/75e00245542efdbae26abcf009936b64#file-gistfile1-txt-L121-L126 because they are target roots.

[wisechengyi]

Also to be clear on the high level, given A -> B -> C -> D

$ export-dep-as-jar A
targets: {
A: { targets: [], libraries [B,C,D] }
}
libraries: {
B: // just a jar with no dep
C: // just a jar with no dep
D: // just a jar with no dep
}
$ export-dep-as-jar A B
targets: {
A: {targets: [B], libraries: []} 
B: {targets: [], libraies: [C D]}
}
libraries: {
C: // just a jar with no dep
D: // just a jar with no dep
}