V2.0 performance drop

Unity Obstacle Tower Challenge
#1

Did anybody else encounter performance drops in v2.0?

I observed on my PPO implementation that one data sampling cycles takes longer.
Before I observed mean duration of one cycle for 53,3 seconds with a deviation of 1,1.

Now I observe a mean of 146 seconds with a deviation of 24,2.

I tested varying parameters, cpu/gpu training and 3 different machines (windows and ubuntu). All came to the conclusion that training takes more time with a strong variation.

v1.3: 56, 54, 53, 53, 53, 53, 54, 53, 52, 54 seconds
v2.0: 91, 128, 133, 136, 144, 160, 145, 166, 156, 165, 182 seconds

#2

I haven’t measured FPS of the v2.0 env yet, but noticed two things:

  • v2.0 is prettier: It has glows and fancier key models/textures. Perhaps this affects the performance little bit?
  • reset() takes much longer now with default config. Set total-floors reset parameter to 25 or lower to speed up reset considerably.
1 Like
#3

Thanks for the hint.

Setting the total-floors to a much smaller number solved this.

However, I cannot pass in a config to reset(). @arthurj
At least it worked for instantiating the game instances.

#4

If it is any help, we do the reset like this:

config = {"total-floors": 25}
env.reset(config)

Granted, we have not tried changing configs mid-run.

On the topic of configs: Some of the variables did not seem to have an effect of game (visual-theme and allowed-rooms), but this is only based on very brief experimentation so might be just a brainfart in our end.

#5

Like I said, I cannot pass the config to reset like how you do it in your example.

It throws a TypeError
TypeError: reset() takes 1 positional argument but 2 were given

#6

Hi @Leckofunny

Are you using the latest version of obstacle tower env? https://github.com/Unity-Technologies/obstacle-tower-env/blob/master/obstacle_tower_env.py#L153

As for increased time, it is correct that in this version by default we generate the definitions for 100 floors on every new seed (i.e. reset, unless you are resetting to the same seed as before). As such it will take longer.

#7

Hi @arthurj

Yes, I’m using the latest obstacle_tower_env.py.