It seems the server is using ObtainDiamondDense env for evaluation instead of the sparse counterpart. Is that a bug?
Thank you
I think this is rather a bug in the environment. MineRLObtainDiamond-v0 seems to give multiple rewards as well. Also MineRLObtainDiamond-v0 rewards seem to be doubled. This is also being discussed here: Can the agent get reward repeatedly? but there is no official response yet.
The agent should not be able to get repeated rewards in ‘MineRLObtainBaimond-v0’ - if you observe this please report it as a bug!
@BrandonHoughton It is a bug indeed. For example, using the given test.py for evaluation, the agent has no “craft” action but it can obtain rewards up to 8 or 10 for some episode, which is impossible.