option to output all command lines for all subprocesses

[cosmicexplorer]

Potentially relevant: #7350, #8450, #8765.

Background

In one case in our monorepo, a user wanted to have all of the source roots of all dependencies of thrift targets, so they could reproduce the -I args passed to thrift/scrooge for each target. This allowed their custom thrift parser to correctly resolve include lines. A task was created in the monorepo to do this, but it relies on hacking the gen() method to record command-line arguments:

pants/contrib/scrooge/src/python/pants/contrib/scrooge/tasks/scrooge_gen.py

Line 146 in 9a38d6f

def gen(self, partial_cmd, target, target_workdir):

Possible Solution

We have informally used the -ldebug argument (or PANTS_LEVEL=debug) to observe process execution command lines, as the command line is printed whenever a workunit is started (the cmd arg to self.context.new_workunit()). It might be useful to abstract this away, and allow pants to write out all of the process executions that it would have invoked (even if the process execution is cached via the process execution cache in the LMDB store).

This could be used for the above use case by having pants write out all of the process executions it performed as lines of json in a @console_rule (similar to #8450), which the external tool could have then parsed for -I arguments provided to thrift.

Result

This would have allowed an external tool with knowledge only of the thrift command line to reproduce the environment outside of pants. It would be nice if pants could wrap this tool, but it would also be nice to avoid that barrier to entry if we could just make pants spit out the right information to stdout!

[Eric-Arellano]

I think that you addressed this through your excellent __run.sh script! Feel free to reopen if that's not the case.