[Bug Report] Waterworld Bugs

[jjshoots]

List of Bugs

• action_space currently only has a default range of -0.01 to 0.01, I feel like this is a problem. Of course, usage of a normalizing wrapper can work, but IMHO the default environment should not have an action space this narrow to begin with.
• I think the action magnitude is too small in general, the agents tend to not be able to move out of the way in time. I think increasing it by 100% should help with this.
• Sometimes, the environment instantiates with the agents receiving >10 reward.
• Due to collision logic, the pursuers are trying to match their velocities to the food velocities as much as possible to not get knocked off course, the result is that if the food is travelling towards them but at a totally different velocity vector, the pursuers will actively avoid the food. I propose two fixes to this, remove collision logic between agents and food or reduce food mass.
• This is a small pedantic thing, but I would much rather see the poison as red and the food as green. This intuitively makes more sense without having to go into the docs. But if changing this is too hard then that's fine.

Logs: https://wandb.ai/jjshoots/waterworld_sweep/sweeps/fy9qovbx?workspace=user-jjshoots

[WillDudley]

@jjshoots Are you on this or does this need to be delegated? Is it high-priority?

[jjshoots]

@BolunDai0216 said he'd look at it, it's moderate priority. The env is not broken if this isn't fixed but it's not ideal.