[train v2+tune] Add an environment variable to disable running the TrainController as an actor

[justinvyu]

Why are these changes needed?

Add the RAY_TRAIN_RUN_CONTROLLER_AS_ACTOR environment variable that disables the TrainController running as an actor.

For the Train v2 + Tune integration, the TrainController cannot run as a separate actor, since callbacks would run in a separate process and would not be able to call ray.tune.report to propagate intermediate metrics/checkpoints to Tune. Therefore, Train needs to be able to run in a mode where the TrainController just runs on the process that trainer.fit() was called in. For Tune, this it the function Trainable that acts as the Ray Train driver. This is an internal implementation detail, which is why I introduce this as an environment variable that Ray Tune will set automatically.

[matthewdeng]

Unrelated to this PR, but would this merge with environment variables set by a user-defined runtime env?

[justinvyu]

I don't know actually, I'll test it out

[justinvyu]

This will update a user-defined runtime env, which is as expected.

import ray
import os

ray.init(runtime_env={"env_vars": {"RAY_TRAIN_V2_ENABLED": "0"}})


import ray.train
from ray.train.v2.api.data_parallel_trainer import DataParallelTrainer

os.environ["RAY_TRAIN_V2_ENABLED"] = "1"

trainer = DataParallelTrainer(
    lambda: print(os.getenv("RAY_TRAIN_V2_ENABLED")),
    scaling_config=ray.train.ScalingConfig(num_workers=2)
)
trainer.fit()

(RayTrainWorker pid=69742) 1
(RayTrainWorker pid=69742) 
(RayTrainWorker pid=69743) 1
(RayTrainWorker pid=69743)