stats: add output format option to save Pants stats as JSON

[AlexTereshenkov]

With the current implementation, when stats output is shown, it's always in plain text which makes parsing the stats really hard. For instance, it may be useful to track the cache hit rate for every Pants invocation to make sure it's being used effectively.

To make parsing the stats easy, it's now possible to export stats as JSON Lines objects. The amount and type of information is identical to what's being produced for plain text output, only the output format differs.

A few implementation details worth sharing:

• when saving the output into a file (that may already have previous executions saved), the write happens in append mode to keep the stats of a previous execution. This makes it possible to have a single place with all the Pants invocations stats (e.g. after a CI build where multiple separate Pants invocations might have happened is complete).

As a follow-up, I'll refactor the code responsible for production of the stats so that the logic is defined in a single place (e.g. getting all the data in JSON format and then dumping it into plain text, if user asks to do so).

[AlexTereshenkov]

Example invocation of multiple commands writing to the same file that is preserved between invocations:

$ PANTS_SOURCE=../pants pants --stats-log --stats-format=jsonlines --stats-output-file=stats.jsonl --stats-memory-summary roots

Output file

{"timestamp": "2024-03-29 11:16:58", "command": "pants --stats-log --stats-format=jsonlines --stats-output-file=stats.jsonl --stats-memory-summary list ::", "counters": [{"name": "backtrack_attempts", "count": 0}, {"name": "docker_execution_errors", "count": 0}, {"name": "docker_execution_requests", "count": 0}, {"name": "docker_execution_successes", "count": 0}, {"name": "local_cache_read_errors", "count": 0}, {"name": "local_cache_requests", "count": 0}, {"name": "local_cache_requests_cached", "count": 0}, {"name": "local_cache_requests_uncached", "count": 0}, {"name": "local_cache_total_time_saved_ms", "count": 0}, {"name": "local_cache_write_errors", "count": 0}, {"name": "local_execution_requests", "count": 0}, {"name": "local_process_total_time_run_ms", "count": 0}, {"name": "remote_cache_read_errors", "count": 0}, {"name": "remote_cache_request_timeouts", "count": 0}, {"name": "remote_cache_requests", "count": 0}, {"name": "remote_cache_requests_cached", "count": 0}, {"name": "remote_cache_requests_uncached", "count": 0}, {"name": "remote_cache_speculation_local_completed_first", "count": 0}, {"name": "remote_cache_speculation_remote_completed_first", "count": 0}, {"name": "remote_cache_total_time_saved_ms", "count": 0}, {"name": "remote_cache_write_attempts", "count": 0}, {"name": "remote_cache_write_errors", "count": 0}, {"name": "remote_cache_write_successes", "count": 0}, {"name": "remote_execution_errors", "count": 0}, {"name": "remote_execution_requests", "count": 0}, {"name": "remote_execution_rpc_errors", "count": 0}, {"name": "remote_execution_rpc_execute", "count": 0}, {"name": "remote_execution_rpc_retries", "count": 0}, {"name": "remote_execution_rpc_wait_execution", "count": 0}, {"name": "remote_execution_success", "count": 0}, {"name": "remote_execution_timeouts", "count": 0}, {"name": "remote_process_total_time_run_ms", "count": 0}, {"name": "remote_store_exists_attempts", "count": 0}, {"name": "remote_store_exists_errors", "count": 0}, {"name": "remote_store_exists_successes", "count": 0}, {"name": "remote_store_missing_digest", "count": 0}, {"name": "remote_store_read_attempts", "count": 0}, {"name": "remote_store_read_cached", "count": 0}, {"name": "remote_store_read_errors", "count": 0}, {"name": "remote_store_read_uncached", "count": 0}, {"name": "remote_store_request_timeouts", "count": 0}, {"name": "remote_store_write_attempts", "count": 0}, {"name": "remote_store_write_errors", "count": 0}, {"name": "remote_store_write_successes", "count": 0}], "memory_summary": [{"bytes": 48, "count": 1, "name": "pants.backend.javascript.package_json.AllPackageJsonNames"}, {"bytes": 48, "count": 1, "name": "pants.backend.project_info.filter_targets.FilterSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.project_info.list_targets.List"}, {"bytes": 48, "count": 1, "name": "pants.backend.project_info.list_targets.ListSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.python.dependency_inference.subsystem.PythonInferSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.python.goals.lockfile.PythonSyntheticLockfileTargetsRequest"}, {"bytes": 48, "count": 1, "name": "pants.backend.python.subsystems.setup.PythonSetup"}, {"bytes": 48, "count": 1, "name": "pants.backend.scala.subsystems.scala.ScalaSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.scala.subsystems.scala_infer.ScalaInferSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.tools.workunit_logger.rules.WorkunitLogger"}, {"bytes": 48, "count": 1, "name": "pants.backend.tools.workunit_logger.rules.WorkunitLoggerCallbackFactoryRequest"}, {"bytes": 48, "count": 1, "name": "pants.backend.visibility.rules.BuildFileVisibilityImplementationRequest"}, {"bytes": 48, "count": 1, "name": "pants.base.build_root.BuildRoot"}, {"bytes": 48, "count": 1, "name": "pants.build_graph.build_configuration.BuildConfiguration"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.environments.AllEnvironmentTargets"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.environments.EnvironmentsSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.search_paths.ValidateSearchPathsRequest"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.search_paths.ValidatedSearchPaths"}, {"bytes": 48, "count": 1, "name": "pants.engine.console.Console"}, {"bytes": 48, "count": 1, "name": "pants.engine.environment.ChosenLocalEnvironmentName"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.build_files.BuildFileOptions"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.dep_rules.BuildFileDependencyRulesImplementation"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.dep_rules.MaybeBuildFileDependencyRulesImplementation"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.graph.ResolveAllTargetGeneratorRequests"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.mapper.SpecsFilter"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.options_parsing._Options"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.parser.BuildFilePreludeSymbols"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.parser.Parser"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.synthetic_targets.AllSyntheticAddressMaps"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.synthetic_targets.SyntheticTargetsSpecPathsRequest"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.FilteredTargets"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.RegisteredTargetTypes"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.TargetTypesToGenerateTargetsRequests"}, {"bytes": 48, "count": 1, "name": "pants.engine.unions.UnionMembership"}, {"bytes": 48, "count": 1, "name": "pants.goal.stats_aggregator.StatsAggregatorCallbackFactoryRequest"}, {"bytes": 48, "count": 1, "name": "pants.goal.stats_aggregator.StatsAggregatorSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.init.bootstrap_scheduler.BootstrapStatus"}, {"bytes": 48, "count": 1, "name": "pants.jvm.subsystems.JvmSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.option.global_options.GlobalOptions"}, {"bytes": 48, "count": 1, "name": "pants.option.global_options.UnmatchedBuildFileGlobs"}, {"bytes": 56, "count": 1, "name": "pants.engine.internals.synthetic_targets.SyntheticTargetsSpecPaths"}, {"bytes": 64, "count": 1, "name": "pants.engine.streaming_workunit_handler.WorkunitsCallbackFactories"}, {"bytes": 96, "count": 1, "name": "pants.backend.javascript.package_json.AllPackageJson"}, {"bytes": 96, "count": 2, "name": "pants.backend.scala.target_types.GenerateJvmArtifactForScalaTargets"}, {"bytes": 96, "count": 2, "name": "pants.backend.scala.target_types.ScalaSettingsRequest"}, {"bytes": 96, "count": 2, "name": "pants.base.specs.RawSpecs"}, {"bytes": 96, "count": 2, "name": "pants.base.specs.Specs"}, {"bytes": 96, "count": 2, "name": "pants.build_graph.address.MaybeAddress"}, {"bytes": 96, "count": 2, "name": "pants.core.util_rules.archive.MaybeExtractArchiveRequest"}, {"bytes": 96, "count": 2, "name": "pants.core.util_rules.environments.EnvironmentTarget"}, {"bytes": 96, "count": 2, "name": "pants.engine.fs.DownloadFile"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.graph.OwnersRequest"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.graph.WrappedTargetForBootstrap"}, {"bytes": 96, "count": 2, "name": "pants.engine.streaming_workunit_handler.WorkunitsCallbackFactory"}, {"bytes": 96, "count": 2, "name": "pants.engine.target.Targets"}, {"bytes": 96, "count": 2, "name": "pants.engine.target.WrappedTargetRequest"}, {"bytes": 96, "count": 4, "name": "pants.base.specs.RawSpecsWithOnlyFileOwners"}, {"bytes": 96, "count": 4, "name": "pants.engine.internals.graph.Owners"}, {"bytes": 136, "count": 1, "name": "(native) session_values"}, {"bytes": 144, "count": 3, "name": "pants.engine.process.Process"}, {"bytes": 144, "count": 3, "name": "pants.engine.process.ProductDescription"}, {"bytes": 144, "count": 3, "name": "pants.engine.target.UnexpandedTargets"}, {"bytes": 144, "count": 5, "name": "pants.engine.env_vars.CompleteEnvironmentVars"}, {"bytes": 144, "count": 1376, "name": "pants.engine.environment.EnvironmentName"}, {"bytes": 160, "count": 4, "name": "builtins.Digest"}, {"bytes": 192, "count": 4, "name": "pants.backend.python.macros.python_requirements.GenerateFromPythonRequirementsRequest"}, {"bytes": 240, "count": 5, "name": "pants.engine.env_vars.EnvironmentVars"}, {"bytes": 240, "count": 5, "name": "pants.engine.env_vars.EnvironmentVarsRequest"}, {"bytes": 288, "count": 2, "name": "builtins.Address"}, {"bytes": 288, "count": 6, "name": "pants.backend.javascript.package_json.GenerateNodePackageTargets"}, {"bytes": 288, "count": 6, "name": "pants.engine.fs.FileContent"}, {"bytes": 288, "count": 12, "name": "pants.backend.javascript.package_json.PackageJson"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_environments_preview"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_filter"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_jvm"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_list"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_python"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_python_infer"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_scala"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_scala_infer"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_stats"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_workunit_logger"}, {"bytes": 297, "count": 1, "name": "(native) pants.backend.javascript.package_json.all_package_json"}, {"bytes": 297, "count": 1, "name": "(native) pants.backend.javascript.package_json.all_package_json_names"}, {"bytes": 297, "count": 1, "name": "(native) pants.core.util_rules.environments.determine_all_environments"}, {"bytes": 297, "count": 1, "name": "(native) pants.core.util_rules.environments.determine_local_environment"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.build_files.evaluate_preludes"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.build_files.extract_build_file_options"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.dep_rules.get_build_file_dependency_rules_implementation"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.graph.extract_unmatched_build_file_globs"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.graph.target_types_to_generate_targets_requests"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.options_parsing.parse_options"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.specs_rules.setup_specs_filter"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.synthetic_targets.all_synthetic_targets"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.bootstrap_status"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_configuration_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_root_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.parser_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.registered_target_types_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.union_membership_singleton"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.python.goals.lockfile.python_lockfile_synthetic_targets"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.tools.workunit_logger.rules.construct_callback"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.visibility.rules.build_file_visibility_implementation"}, {"bytes": 313, "count": 1, "name": "(native) pants.core.util_rules.search_paths.validate_search_paths"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.graph.resolve_all_generator_target_requests"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.specs_rules.filter_targets"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.specs_rules.resolve_addresses_from_specs"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.synthetic_targets.get_synthetic_targets_spec_paths"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.streaming_workunit_handler.construct_workunits_callback_factories"}, {"bytes": 313, "count": 1, "name": "(native) pants.goal.stats_aggregator.construct_callback"}, {"bytes": 329, "count": 1, "name": "(native) pants.backend.project_info.list_targets.list_targets"}, {"bytes": 336, "count": 4, "name": "builtins.AddressInput"}, {"bytes": 336, "count": 7, "name": "pants.engine.fs.PathGlobs"}, {"bytes": 336, "count": 14, "name": "pants.base.specs.RawSpecsWithoutFileOwners"}, {"bytes": 384, "count": 6, "name": "pants.backend.javascript.package_json.PackageJsonSourceField"}, {"bytes": 432, "count": 2, "name": "(native) downloaded_file"}, {"bytes": 432, "count": 7, "name": "pants.backend.javascript.package_json.PackageJsonForGlobs"}, {"bytes": 528, "count": 11, "name": "pants.option.scope.Scope"}, {"bytes": 528, "count": 11, "name": "pants.option.scope.ScopedOptions"}, {"bytes": 626, "count": 2, "name": "(native) pants.backend.scala.target_types.generate_jvm_artifact_targets"}, {"bytes": 626, "count": 2, "name": "(native) pants.backend.scala.target_types.scala_settings_request"}, {"bytes": 626, "count": 2, "name": "(native) pants.core.util_rules.archive.convert_digest_to_MaybeExtractArchiveRequest"}, {"bytes": 626, "count": 2, "name": "(native) pants.core.util_rules.environments.get_target_for_environment_name"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.download_file.download_file"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.build_files.maybe_resolve_address"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.build_files.resolve_address"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.graph.find_owners"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.graph.resolve_target_for_bootstrapping"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.graph.resolve_targets"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.specs_rules.addresses_from_owners"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.specs_rules.addresses_from_raw_specs_with_only_file_owners"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.specs_rules.resolve_addresses_from_raw_specs"}, {"bytes": 935, "count": 4, "name": "(native) read_link"}, {"bytes": 939, "count": 3, "name": "(native) pants.engine.internals.graph.resolve_unexpanded_targets"}, {"bytes": 939, "count": 3, "name": "(native) pants.engine.process.get_multi_platform_request_description"}, {"bytes": 1120, "count": 4, "name": "(native) root"}, {"bytes": 1252, "count": 4, "name": "(native) pants.backend.python.macros.python_requirements.generate_from_python_requirement"}, {"bytes": 1252, "count": 4, "name": "(native) pants.engine.internals.specs_rules.addresses_from_raw_specs_without_file_owners"}, {"bytes": 1565, "count": 5, "name": "(native) pants.engine.internals.specs_rules.address_families_from_raw_specs_without_file_owners"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.resolve.common.ArtifactRequirements"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.resolve.coursier_fetch.CoursierResolveInfo"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.util_rules.ExtractFileDigest"}, {"bytes": 1645, "count": 5, "name": "(native) pants.engine.internals.platform_rules.environment_vars_subset"}, {"bytes": 1878, "count": 6, "name": "(native) pants.backend.javascript.package_json.generate_node_package_targets"}, {"bytes": 1878, "count": 6, "name": "(native) pants.backend.javascript.package_json.package_json_for_source"}, {"bytes": 1878, "count": 6, "name": "(native) pants.backend.javascript.package_json.parse_package_json"}, {"bytes": 2112, "count": 33, "name": "builtins.FileDigest"}, {"bytes": 2191, "count": 7, "name": "(native) pants.backend.javascript.package_json.read_package_jsons"}, {"bytes": 3443, "count": 11, "name": "(native) pants.engine.internals.options_parsing.scope_options"}, {"bytes": 5952, "count": 5, "name": "pants.engine.internals.mapper.AddressFamilies"}, {"bytes": 10329, "count": 33, "name": "(native) pants.jvm.util_rules.digest_to_file_digest"}, {"bytes": 10857, "count": 33, "name": "(native) pants.jvm.resolve.coursier_fetch.prepare_coursier_resolve_info"}, {"bytes": 17136, "count": 357, "name": "pants.engine.internals.mapper.AddressFamily"}, {"bytes": 19824, "count": 413, "name": "pants.engine.internals.build_files.OptionalAddressFamily"}, {"bytes": 19824, "count": 413, "name": "pants.engine.internals.synthetic_targets.SyntheticAddressMapsRequest"}, {"bytes": 19888, "count": 414, "name": "pants.engine.internals.synthetic_targets.SyntheticAddressMaps"}, {"bytes": 20640, "count": 770, "name": "pants.engine.internals.build_files.AddressFamilyDir"}, {"bytes": 24192, "count": 504, "name": "pants.backend.python.target_types.PythonFilesGeneratorSettingsRequest"}, {"bytes": 24288, "count": 506, "name": "pants.engine.target.TargetFilesGeneratorSettings"}, {"bytes": 29616, "count": 617, "name": "pants.engine.internals.graph.GenerateFileTargets"}, {"bytes": 29616, "count": 617, "name": "pants.engine.target.SourcesPaths"}, {"bytes": 29616, "count": 617, "name": "pants.engine.target.SourcesPathsRequest"}, {"bytes": 30192, "count": 629, "name": "pants.engine.internals.graph.ResolveTargetGeneratorRequests"}, {"bytes": 30192, "count": 629, "name": "pants.engine.target.GeneratedTargets"}, {"bytes": 30240, "count": 630, "name": "pants.engine.internals.graph.ResolvedTargetGeneratorRequests"}, {"bytes": 34704, "count": 723, "name": "pants.engine.internals.parametrize._TargetParametrizationsRequest"}, {"bytes": 34704, "count": 723, "name": "pants.engine.internals.target_adaptor.TargetAdaptorRequest"}, {"bytes": 35040, "count": 730, "name": "pants.engine.internals.graph._AdaptorAndType"}, {"bytes": 35040, "count": 730, "name": "pants.engine.internals.graph._RequestAdaptorAndType"}, {"bytes": 40488, "count": 723, "name": "pants.engine.internals.parametrize._TargetParametrizations"}, {"bytes": 46272, "count": 723, "name": "pants.engine.internals.target_adaptor.TargetAdaptor"}, {"bytes": 80928, "count": 10, "name": "pants.engine.addresses.Addresses"}, {"bytes": 97234, "count": 45, "name": "(native) process"}, {"bytes": 111741, "count": 357, "name": "(native) pants.engine.internals.build_files.ensure_address_family"}, {"bytes": 129269, "count": 413, "name": "(native) pants.engine.internals.build_files.parse_address_family"}, {"bytes": 129269, "count": 413, "name": "(native) pants.engine.internals.synthetic_targets.get_synthetic_address_maps"}, {"bytes": 157752, "count": 504, "name": "(native) pants.backend.python.target_types_rules.python_files_generator_settings"}, {"bytes": 193121, "count": 617, "name": "(native) pants.engine.internals.graph.resolve_source_paths"}, {"bytes": 196877, "count": 629, "name": "(native) pants.engine.internals.graph.resolve_generator_target_requests"}, {"bytes": 202993, "count": 617, "name": "(native) pants.engine.internals.graph.generate_file_targets"}, {"bytes": 226299, "count": 723, "name": "(native) pants.engine.internals.build_files.find_target_adaptor"}, {"bytes": 228490, "count": 730, "name": "(native) pants.engine.internals.graph._determine_target_adaptor_and_type"}, {"bytes": 237867, "count": 723, "name": "(native) pants.engine.internals.graph.resolve_target_parametrizations"}, {"bytes": 346479, "count": 1946, "name": "(native) digest_file"}, {"bytes": 380561, "count": 579, "name": "(native) scandir"}, {"bytes": 462289, "count": 432, "name": "(native) snapshot"}], "observation_histograms": [{"name": "local_store_read_blob_time_micros", "min": 62, "max": 56159, "mean": 19625.246, "std_dev": 7174.851, "total_observations": 358, "sum": 7025838, "p25": 14831, "p50": 17375, "p75": 22783, "p90": 29455, "p95": 33055, "p99": 39391}, {"name": "local_store_read_blob_size", "min": 146, "max": 6199, "mean": 344.279, "std_dev": 506.962, "total_observations": 358, "sum": 123252, "p25": 149, "p50": 184, "p75": 341, "p90": 593, "p95": 936, "p99": 2153}]}
{"timestamp": "2024-03-29 11:17:03", "command": "pants --stats-log --stats-format=jsonlines --stats-output-file=stats.jsonl --stats-memory-summary roots ::", "counters": [{"name": "backtrack_attempts", "count": 0}, {"name": "docker_execution_errors", "count": 0}, {"name": "docker_execution_requests", "count": 0}, {"name": "docker_execution_successes", "count": 0}, {"name": "local_cache_read_errors", "count": 0}, {"name": "local_cache_requests", "count": 0}, {"name": "local_cache_requests_cached", "count": 0}, {"name": "local_cache_requests_uncached", "count": 0}, {"name": "local_cache_total_time_saved_ms", "count": 0}, {"name": "local_cache_write_errors", "count": 0}, {"name": "local_execution_requests", "count": 0}, {"name": "local_process_total_time_run_ms", "count": 0}, {"name": "remote_cache_read_errors", "count": 0}, {"name": "remote_cache_request_timeouts", "count": 0}, {"name": "remote_cache_requests", "count": 0}, {"name": "remote_cache_requests_cached", "count": 0}, {"name": "remote_cache_requests_uncached", "count": 0}, {"name": "remote_cache_speculation_local_completed_first", "count": 0}, {"name": "remote_cache_speculation_remote_completed_first", "count": 0}, {"name": "remote_cache_total_time_saved_ms", "count": 0}, {"name": "remote_cache_write_attempts", "count": 0}, {"name": "remote_cache_write_errors", "count": 0}, {"name": "remote_cache_write_successes", "count": 0}, {"name": "remote_execution_errors", "count": 0}, {"name": "remote_execution_requests", "count": 0}, {"name": "remote_execution_rpc_errors", "count": 0}, {"name": "remote_execution_rpc_execute", "count": 0}, {"name": "remote_execution_rpc_retries", "count": 0}, {"name": "remote_execution_rpc_wait_execution", "count": 0}, {"name": "remote_execution_success", "count": 0}, {"name": "remote_execution_timeouts", "count": 0}, {"name": "remote_process_total_time_run_ms", "count": 0}, {"name": "remote_store_exists_attempts", "count": 0}, {"name": "remote_store_exists_errors", "count": 0}, {"name": "remote_store_exists_successes", "count": 0}, {"name": "remote_store_missing_digest", "count": 0}, {"name": "remote_store_read_attempts", "count": 0}, {"name": "remote_store_read_cached", "count": 0}, {"name": "remote_store_read_errors", "count": 0}, {"name": "remote_store_read_uncached", "count": 0}, {"name": "remote_store_request_timeouts", "count": 0}, {"name": "remote_store_write_attempts", "count": 0}, {"name": "remote_store_write_errors", "count": 0}, {"name": "remote_store_write_successes", "count": 0}], "memory_summary": [{"bytes": 48, "count": 1, "name": "pants.backend.project_info.list_roots.Roots"}, {"bytes": 48, "count": 1, "name": "pants.backend.project_info.list_roots.RootsSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.backend.python.goals.lockfile.PythonSyntheticLockfileTargetsRequest"}, {"bytes": 48, "count": 1, "name": "pants.backend.python.subsystems.setup.PythonSetup"}, {"bytes": 48, "count": 1, "name": "pants.backend.tools.workunit_logger.rules.WorkunitLogger"}, {"bytes": 48, "count": 1, "name": "pants.backend.tools.workunit_logger.rules.WorkunitLoggerCallbackFactoryRequest"}, {"bytes": 48, "count": 1, "name": "pants.backend.visibility.rules.BuildFileVisibilityImplementationRequest"}, {"bytes": 48, "count": 1, "name": "pants.base.build_root.BuildRoot"}, {"bytes": 48, "count": 1, "name": "pants.build_graph.build_configuration.BuildConfiguration"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.environments.AllEnvironmentTargets"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.environments.EnvironmentsSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.search_paths.ValidateSearchPathsRequest"}, {"bytes": 48, "count": 1, "name": "pants.core.util_rules.search_paths.ValidatedSearchPaths"}, {"bytes": 48, "count": 1, "name": "pants.engine.addresses.Addresses"}, {"bytes": 48, "count": 1, "name": "pants.engine.console.Console"}, {"bytes": 48, "count": 1, "name": "pants.engine.environment.ChosenLocalEnvironmentName"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.build_files.BuildFileOptions"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.build_files.OptionalAddressFamily"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.dep_rules.BuildFileDependencyRulesImplementation"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.dep_rules.MaybeBuildFileDependencyRulesImplementation"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.mapper.AddressFamily"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.options_parsing._Options"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.parser.BuildFilePreludeSymbols"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.parser.Parser"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.synthetic_targets.AllSyntheticAddressMaps"}, {"bytes": 48, "count": 1, "name": "pants.engine.internals.synthetic_targets.SyntheticAddressMapsRequest"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.RegisteredTargetTypes"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.TargetTypesToGenerateTargetsRequests"}, {"bytes": 48, "count": 1, "name": "pants.engine.target.UnexpandedTargets"}, {"bytes": 48, "count": 1, "name": "pants.engine.unions.UnionMembership"}, {"bytes": 48, "count": 1, "name": "pants.goal.stats_aggregator.StatsAggregatorCallbackFactoryRequest"}, {"bytes": 48, "count": 1, "name": "pants.goal.stats_aggregator.StatsAggregatorSubsystem"}, {"bytes": 48, "count": 1, "name": "pants.init.bootstrap_scheduler.BootstrapStatus"}, {"bytes": 48, "count": 1, "name": "pants.option.global_options.GlobalOptions"}, {"bytes": 48, "count": 1, "name": "pants.source.source_root.AllSourceRoots"}, {"bytes": 48, "count": 1, "name": "pants.source.source_root.SourceRootConfig"}, {"bytes": 48, "count": 2, "name": "pants.engine.internals.build_files.AddressFamilyDir"}, {"bytes": 64, "count": 1, "name": "pants.engine.streaming_workunit_handler.WorkunitsCallbackFactories"}, {"bytes": 96, "count": 2, "name": "pants.build_graph.address.MaybeAddress"}, {"bytes": 96, "count": 2, "name": "pants.core.util_rules.archive.MaybeExtractArchiveRequest"}, {"bytes": 96, "count": 2, "name": "pants.core.util_rules.environments.EnvironmentTarget"}, {"bytes": 96, "count": 2, "name": "pants.engine.fs.DownloadFile"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.graph.WrappedTargetForBootstrap"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.graph._AdaptorAndType"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.graph._RequestAdaptorAndType"}, {"bytes": 96, "count": 2, "name": "pants.engine.internals.target_adaptor.TargetAdaptorRequest"}, {"bytes": 96, "count": 2, "name": "pants.engine.streaming_workunit_handler.WorkunitsCallbackFactory"}, {"bytes": 96, "count": 2, "name": "pants.engine.target.WrappedTargetRequest"}, {"bytes": 104, "count": 2, "name": "pants.engine.internals.synthetic_targets.SyntheticAddressMaps"}, {"bytes": 128, "count": 2, "name": "pants.engine.internals.target_adaptor.TargetAdaptor"}, {"bytes": 136, "count": 1, "name": "(native) session_values"}, {"bytes": 144, "count": 3, "name": "pants.engine.process.Process"}, {"bytes": 144, "count": 3, "name": "pants.engine.process.ProductDescription"}, {"bytes": 144, "count": 36, "name": "pants.engine.environment.EnvironmentName"}, {"bytes": 160, "count": 4, "name": "builtins.Digest"}, {"bytes": 192, "count": 6, "name": "pants.engine.env_vars.CompleteEnvironmentVars"}, {"bytes": 288, "count": 2, "name": "builtins.Address"}, {"bytes": 288, "count": 6, "name": "pants.engine.env_vars.EnvironmentVars"}, {"bytes": 288, "count": 6, "name": "pants.engine.env_vars.EnvironmentVarsRequest"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_environments_preview"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_python"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_roots"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_source"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_stats"}, {"bytes": 297, "count": 1, "name": "(native) construct_scope_workunit_logger"}, {"bytes": 297, "count": 1, "name": "(native) pants.core.util_rules.environments.determine_all_environments"}, {"bytes": 297, "count": 1, "name": "(native) pants.core.util_rules.environments.determine_local_environment"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.build_files.evaluate_preludes"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.build_files.extract_build_file_options"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.dep_rules.get_build_file_dependency_rules_implementation"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.graph.target_types_to_generate_targets_requests"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.options_parsing.parse_options"}, {"bytes": 297, "count": 1, "name": "(native) pants.engine.internals.synthetic_targets.all_synthetic_targets"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.bootstrap_status"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_configuration_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.build_root_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.parser_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.registered_target_types_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.init.engine_initializer.EngineInitializer.setup_graph_extended.union_membership_singleton"}, {"bytes": 297, "count": 1, "name": "(native) pants.source.source_root.all_roots"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.project_info.list_roots.list_roots"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.python.goals.lockfile.python_lockfile_synthetic_targets"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.tools.workunit_logger.rules.construct_callback"}, {"bytes": 313, "count": 1, "name": "(native) pants.backend.visibility.rules.build_file_visibility_implementation"}, {"bytes": 313, "count": 1, "name": "(native) pants.core.util_rules.search_paths.validate_search_paths"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.build_files.ensure_address_family"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.build_files.parse_address_family"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.graph.resolve_unexpanded_targets"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.internals.synthetic_targets.get_synthetic_address_maps"}, {"bytes": 313, "count": 1, "name": "(native) pants.engine.streaming_workunit_handler.construct_workunits_callback_factories"}, {"bytes": 313, "count": 1, "name": "(native) pants.goal.stats_aggregator.construct_callback"}, {"bytes": 336, "count": 4, "name": "builtins.AddressInput"}, {"bytes": 336, "count": 7, "name": "pants.option.scope.Scope"}, {"bytes": 336, "count": 7, "name": "pants.option.scope.ScopedOptions"}, {"bytes": 432, "count": 2, "name": "(native) downloaded_file"}, {"bytes": 626, "count": 2, "name": "(native) pants.core.util_rules.archive.convert_digest_to_MaybeExtractArchiveRequest"}, {"bytes": 626, "count": 2, "name": "(native) pants.core.util_rules.environments.get_target_for_environment_name"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.download_file.download_file"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.build_files.find_target_adaptor"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.build_files.maybe_resolve_address"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.build_files.resolve_address"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.graph._determine_target_adaptor_and_type"}, {"bytes": 626, "count": 2, "name": "(native) pants.engine.internals.graph.resolve_target_for_bootstrapping"}, {"bytes": 935, "count": 4, "name": "(native) read_link"}, {"bytes": 939, "count": 3, "name": "(native) pants.engine.process.get_multi_platform_request_description"}, {"bytes": 1104, "count": 4, "name": "(native) root"}, {"bytes": 1296, "count": 585, "name": "pants.source.source_root.OptionalSourceRoot"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.resolve.common.ArtifactRequirements"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.resolve.coursier_fetch.CoursierResolveInfo"}, {"bytes": 1584, "count": 33, "name": "pants.jvm.util_rules.ExtractFileDigest"}, {"bytes": 1974, "count": 6, "name": "(native) pants.engine.internals.platform_rules.environment_vars_subset"}, {"bytes": 2112, "count": 33, "name": "builtins.FileDigest"}, {"bytes": 2191, "count": 7, "name": "(native) pants.engine.internals.options_parsing.scope_options"}, {"bytes": 10329, "count": 33, "name": "(native) pants.jvm.util_rules.digest_to_file_digest"}, {"bytes": 10857, "count": 33, "name": "(native) pants.jvm.resolve.coursier_fetch.prepare_coursier_resolve_info"}, {"bytes": 28080, "count": 585, "name": "pants.source.source_root.SourceRootRequest"}, {"bytes": 97234, "count": 45, "name": "(native) process"}, {"bytes": 145978, "count": 3, "name": "(native) snapshot"}, {"bytes": 183105, "count": 585, "name": "(native) pants.source.source_root.get_optional_source_root"}, {"bytes": 346479, "count": 1946, "name": "(native) digest_file"}, {"bytes": 380561, "count": 579, "name": "(native) scandir"}], "observation_histograms": [{"name": "local_store_read_blob_time_micros", "min": 64, "max": 64, "mean": 64.0, "std_dev": 0.0, "total_observations": 1, "sum": 64, "p25": 64, "p50": 64, "p75": 64, "p90": 64, "p95": 64, "p99": 64}, {"name": "local_store_read_blob_size", "min": 1634, "max": 1634, "mean": 1634.0, "std_dev": 0.0, "total_observations": 1, "sum": 1634, "p25": 1634, "p50": 1634, "p75": 1634, "p90": 1634, "p95": 1634, "p99": 1634}]}

[huonw]

Thanks for this!

[huonw]

Cool!