Error occurs while training with rllib ray PPO

[Cra2yDavid]

Environment

• Grid2op version: 1.7.2
• System: ubuntu18.04.5
• Using the same environment as Packages available for L2RPN 2022 competition #332

Bug description

When using rllib ray to train a PPO agent for competition L2RPN2022, I met such an error:

AttributeError: type object 'ObservationWCCI2022_l2rpn_wcci_2022' has no attribute 'process_grid2op_shunt_data'

I have no idea about 'process_grid2op_shunt_data' and there's little information about it.

How to reproduce

Code snippet

from grid2op import make
from grid2op.Reward import RedispReward
from grid2op.gym_compat import GymEnv, BoxGymActSpace, BoxGymObsSpace
from ray import tune
import ray
from ray.rllib.agents import ppo
from lightsim2grid import LightSimBackend

class trainenv(GymEnv):
    def __init__(self, env_config):
        self.env = make('l2rpn_wcci_2022', reward_class=RedispReward, backend=LightSimBackend())
        super(trainenv, self).__init__(self.env)
        obs_space_kwargs = {}
        act_space_kwargs = {}
        obs_attr_to_keep = ["month", "day_of_week", "hour_of_day", "minute_of_hour", "gen_p", "load_p", "p_or", "rho",
                            "timestep_overflow", "line_status", "actual_dispatch", "target_dispatch", "storage_charge",
                            "storage_power", "curtailment", "curtailment_limit", "gen_p_before_curtail"]
        act_attr_to_keep = ['curtail', 'set_storage', 'redispatch']
        self.action_space.close()
        self.observation_space.close()
        self.observation_space = BoxGymObsSpace(
            self.env.observation_space,
            attr_to_keep=obs_attr_to_keep,
            **obs_space_kwargs)
        self.action_space = BoxGymActSpace(
            self.env.action_space,
            attr_to_keep=act_attr_to_keep,
            **act_space_kwargs)

    def step(self, gym_action):
        g2op_act = self.action_space.from_gym(gym_action)
        g2op_obs, reward, done, info = self.init_env.step(g2op_act)
        gym_obs = self.observation_space.to_gym(g2op_obs)
        return gym_obs, float(reward), done, info

    def reset(self, seed=None, return_info=False, options=None):
        g2op_obs = self.init_env.reset()
        gym_obs = self.observation_space.to_gym(g2op_obs)
        if return_info:
            return gym_obs, {}
        else:
            return gym_obs

ray.init()

analysis = tune.run(
    ppo.PPOTrainer,
    num_samples=1,
    stop={'timesteps_total': 5000000},
    checkpoint_freq=500000,
    checkpoint_at_end=True,
    local_dir='./results',
    config={
        'env': trainenv,
        'framework': 'torch',
        'num_workers': 2,
        'num_envs_per_worker': 1,
        'rollout_fragment_length': 256,
        "train_batch_size": 512,
        'batch_mode': 'truncate_episodes',
        'horizon': 2018
    },
) 

Current output

(PPOTrainer pid=93468) 2022-08-23 16:21:15,151	ERROR worker.py:449 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PPOTrainer.__init__() (pid=93468, ip=10.214.211.106, repr=PPOTrainer)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1035, in _init
(PPOTrainer pid=93468)     raise NotImplementedError
(PPOTrainer pid=93468) NotImplementedError
(PPOTrainer pid=93468) 
(PPOTrainer pid=93468) During handling of the above exception, another exception occurred:
(PPOTrainer pid=93468) 
(PPOTrainer pid=93468) ray::PPOTrainer.__init__() (pid=93468, ip=10.214.211.106, repr=PPOTrainer)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 830, in __init__
(PPOTrainer pid=93468)     super().__init__(
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/tune/trainable.py", line 149, in __init__
(PPOTrainer pid=93468)     self.setup(copy.deepcopy(self.config))
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 911, in setup
(PPOTrainer pid=93468)     self.workers = WorkerSet(
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 134, in __init__
(PPOTrainer pid=93468)     remote_spaces = ray.get(
(PPOTrainer pid=93468) ray.exceptions.RaySystemError: System error: type object 'ObservationWCCI2022_l2rpn_wcci_2022' has no attribute 'process_grid2op_shunt_data'
(PPOTrainer pid=93468) traceback: Traceback (most recent call last):
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 896, in setup
(PPOTrainer pid=93468)     self._init(self.config, self.env_creator)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1035, in _init
(PPOTrainer pid=93468)     raise NotImplementedError
(PPOTrainer pid=93468) NotImplementedError
(PPOTrainer pid=93468) 
(PPOTrainer pid=93468) During handling of the above exception, another exception occurred:
(PPOTrainer pid=93468) 
(PPOTrainer pid=93468) ray::PPOTrainer.__init__() (pid=93468, ip=10.214.211.106, repr=PPOTrainer)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/serialization.py", line 332, in deserialize_objects
(PPOTrainer pid=93468)     obj = self._deserialize_object(data, metadata, object_ref)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/serialization.py", line 235, in _deserialize_object
(PPOTrainer pid=93468)     return self._deserialize_msgpack_data(data, metadata_fields)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/serialization.py", line 190, in _deserialize_msgpack_data
(PPOTrainer pid=93468)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/ray/serialization.py", line 178, in _deserialize_pickle5_data
(PPOTrainer pid=93468)     obj = pickle.loads(in_band, buffers=buffers)
(PPOTrainer pid=93468)   File "/home/lw/miniconda3/envs/L2/lib/python3.8/site-packages/grid2op/Space/GridObjects.py", line 3742, in init_grid_from_dict_for_pickle
(PPOTrainer pid=93468)     res_cls.process_grid2op_shunt_data()
(PPOTrainer pid=93468) AttributeError: type object 'ObservationWCCI2022_l2rpn_wcci_2022' has no attribute 'process_grid2op_shunt_data'

Expected output

To train agent successfully.

[richardwth]

Same here. The error message is
AttributeError: type object 'LightSimBackend_l2rpn_wcci_2022' has no attribute 'process_grid2op_shunt_data'

Can be solved by changing process_grid2op_shunt_data to process_shunt_data.

[Cra2yDavid]

Same here. The error message is AttributeError: type object 'LightSimBackend_l2rpn_wcci_2022' has no attribute 'process_grid2op_shunt_data'

Can be solved by changing process_grid2op_shunt_data to process_shunt_data.

Thanks for your advice. By the way, can you train and load the agent with rllib ray correctly? I haven't met this kind of error with stablebaselines.

While the code of submission scoring system cannot be modified. So it still need to be solved without changing any package source code.

[richardwth]

can you train and load the agent with rllib ray correctly?

No. I overwrite ray's trainer.save_checkpoint and trainer.load_checkpoint so that they are more user-friendly. Now they output straightforwardly model state_dict in model.pt that can be loaded with or without ray.

While the code of submission scoring system cannot be modified. So it still need to be solved without changing any package source code.

I plan to write a submission agent without ray, so this issue will not appear.

A bit more details on this issue...
Based on the source code, line 3730, of GridObjects.init_grid_from_dict_for_pickle, normally name_res is in globals(), but ray handles globals() in an "abnormally" way in my opinion. So even though I do think that process_grid2op_shunt_data is a bug as it only appears in grid2op once (so definitely not implemented), this issue could have been avoided. In fact, in my case, it could be avoided by simply initializing grid2op env with LightSimBackend everywhere, but unfortunately this may not apply in your case.

[BDonnot]

Hello,

Thanks for noticing this bug. I'll try to fix it to make the training with Ray possible.

Not sure when though :-/

Out of curiosity, which version of lightsim2grid are you using?

[Cra2yDavid]

Out of curiosity, which version of lightsim2grid are you using?

It is lightsim2grid 0.7.0.post1

[Cra2yDavid]

No. I overwrite ray's trainer.save_checkpoint and trainer.load_checkpoint so that they are more user-friendly. Now they output straightforwardly model state_dict in model.pt that can be loaded with or without ray.

Thanks for your geneous reply! I'll have a try on it!

[BDonnot]

Hello,

Sorry for late reply, i could not work on this earlier (parental leave until yesterday)

To avoid any issue what you can do is a 2 steps procedure:

Generate files with the classes definition that you will use during training.

For example call a script "generate_my_classes.py" with:

from grid2op import make
from grid2op.Reward import RedispReward
from lightsim2grid import LightSimBackend

env = make('l2rpn_wcci_2022', reward_class=RedispReward, backend=LightSimBackend())
env.generate_classes()

and then you run this script: python generate_my_classes.py You have to do it once. If you change the backend, you will also need to redo it.

Run you ray script normally

Just run the script in the example and it works

[BDonnot]

Fixed in latest version. Hopefully