How to replace tileset of selected objects in MineRL environments?

[ndrwmlnk]

Can anyone advise how we can replace tileset of selected objects in MineRL environments? Could you please share helpful links that explain how to do this (e.g. GitHub permalinks, etc.)?

Can we change the tileset in prerecorded episodes from the Imitation Learning (human player) dataset (https://minerl.io/dataset)?

We want to create a binary mask channel for some objects to test our approach. To do this, we plan to replace a set of tiles of some objects with a unique color in order to create masks for these objects using a simple color threshold. Or maybe there is an easier way to get such object related masks?

[Miffyli]

What do you refer to with "tileset"? The textures of different objects (see resource packs).

IIRC doing segmentation like this is not supported by Minecraft/Malmo/MineRL (@brandonhoughton can you confirm?). However if you refer to texture packs, a workaround could be to use texturepack that uses solid/single colors per object type. That should work, apart from shadows/lighting.

[ndrwmlnk]

@Miffyli Thanks for the info!

We have summarized the procedure for changing textures in the MineRL environment (see attached figure). However, this procedure requires manual interaction with the Minecraft simulator. @brandonhoughton @Miffyli Do you have any suggestions on how to automate this?

We want to change textures in the MineRL dataset collected from human players. The dataset contains rendered.npz files with info about actions, observations, and rewards, but there is no seed information. Hence, we cannot reproduce this MineRL dataset with different textures. @brandonhoughton @Miffyli Do you know where to find seeds for episodes of the MineRL dataset?

Thank you in advance!

RedTrees.zip

[Miffyli]

@brandonhoughton Do you think it would be possible to share the original MC replay recordings so people can render their own versions?

@ndrwmlnk I do not have direct answers but I have some pointers:

• This is the file used to render the datasets from replay files: https://github.com/minerllabs/minerl/blob/dev/minerl/data/pipeline/render.py
• This repo contains code to render the 2019 submissions in the new texture pack. It might give you a hint how the resource pack was replaced. You might need to do some dirty modifications to your MineRL installation directory by e.g. directly replacing the resource pack file before launching the instance: https://github.com/MadcowD/minerl_submission_runner

[brandonhoughton]

Are we referring to the rendering code? To use a texture pack we simply edited the template options.txt to point to a texture pack and add that texture pack to the template folder. Then we make n copies of this template dir for each render process.

[brandonhoughton]

We notably forgot to record seeds for the demonstration dateset. We do have evaluation seeds however for runs evaluated on AICrowd

[ndrwmlnk]

@brandonhoughton @Miffyli
Do you mean that the seeds for episodes from the Imitation Learning (human-player) dataset do not exist (have not been collected) or they do exist but are not available to the community?

[Miffyli]

I believe it means the seeds were not recorded in the first place (not included in any dataset at all). To be honest, I am not 100% sure how deterministic Minecraft would be if you tried to replicate the actions, since humans played at their framerates while recording was done at lower framerates and all that.

[ndrwmlnk]

@Miffyli @brandonhoughton
In the final round of the MineRL competition, the texture pack was changed. Correct? However, the Imitation Learning (human-player) dataset was still available for training.
How is it possible?

To be honest, I am not 100% sure how deterministic Minecraft would be if you tried to replicate the actions, since humans played at their framerates while recording was done at lower framerates and all that.

If human players had the original texture pack, then the only way to get the Imitation Learning dataset videos with a new texture pack is to collect seeds and actions and generate videos with a new texture pack, assuming that the Minecraft environment is deterministic. Alternatively, data collection can occur when human players are playing using a different texture pack.
What do you think?

[Miffyli]

Ah sorry if I was misleading. I was referring to the actions in the rendered dataset (the one you can download / the one you linked). Repeating those in MineRL environment with fixed seed might not result in same result. However the creators of the dataset have "Minecraft replay files" (from some mod, not sure which one), which gather more data about the environment to further ensure replaying the file leads to same result. This is what they used to re-render the dataset with new textures for the MineRL competition.