[Tune] Custom progress reporter ignored from 2.6.0

[bwohlberg]

What happened + What you expected to happen

Up to and including 2.5.1, the mechanism for specifying a custom progress reported functioned as documented. Since 2.6.0, it seems to be ignored, with the standard progress report being printed despite specification of a custom reported.

Running the test script under 2.5.1 or earlier produces the minimal output from the custom reported as expected. Under 2.6.0 and later, the entire standard reporter output is printed.

Versions / Dependencies

Works in Ray 2.5.1, broken in Ray 2.6.0. Tested under Python 3.9.16 running under Ubuntu 22.04.2.

Reproduction script

import numpy as np
import logging
import ray
from ray import tune


def eval_params(config):
    x, y = config["x"], config["y"]
    cost = x**2 + (y - 0.5) ** 2
    tune.report({"cost": cost})


class _CustomReporter(tune.progress_reporter.TuneReporterBase):
    def report(self, trials, done, *sys_info):
        print("report")
        current_best_trial, metric = self._current_best_trial(trials)
        if current_best_trial is not None:
            val = current_best_trial.last_result["_metric"]["cost"]
            print(f"Current best: {val:.3e}")


ray.init(logging_level=logging.ERROR)


tconfig = tune.TuneConfig(mode="min", metric="_metric/cost", num_samples=5)
rconfig = ray.air.config.RunConfig(verbose=1, progress_reporter=_CustomReporter())

tuner = tune.Tuner(
    eval_params,
    param_space={"x": tune.uniform(-1, 1), "y": tune.uniform(-1, 1)},
    tune_config=tconfig,
    run_config=rconfig
)
results = tuner.fit()


ray.shutdown()

Issue Severity

High: It blocks me from completing my task.

[matthewdeng]

@krfricke could you take a look to see if this is related to the AIR output changes?

[bwohlberg]

Is it possible to estimate when this is likely to be looked at?

[matthewdeng]

Hey @bwohlberg I just took a look, this seems to be because there was a rollout of a new progress reporter.

If you reduce the logging level (in your script it's set to ERROR), you can see the following:

2023-10-04 11:19:47,962 INFO tune.py:654 -- [output] This will use the new output engine with verbosity 1. To disable the new output and use the legacy output engine, set the environment variable RAY_AIR_NEW_OUTPUT=0. For more information, please see https://github.com/ray-project/ray/issues/36949
2023-10-04 11:19:49,540 WARNING tune.py:1000 -- AIR_VERBOSITY is set, ignoring passed-in ProgressReporter for now.

To use your custom progress reporter right now, you can set the RAY_AIR_NEW_OUTPUT=0. In a future release, we'll look into making this configurable without setting this flag.

[bwohlberg]

Thanks @matthewdeng, that seemed to do the trick.

Will you leave this issue open until the new progress reporter also supports customization, so that an update can be added here when this is added?

[matthewdeng]

Yeah I can do that.

[bwohlberg]

Great, thanks for your help with this!