Hi,
Am I the only one who has the issue with the environment provided that it slows down over time? For example, the first 50k steps on my machine took around 4k seconds while the same number of steps from 200k to 250k took almost 10k seconds. Is this a known issue? Would like to hear back from organizers with some ideas on how to fix properly. Right now I have to kill the process simbody and restart it, let’s say, every 100k steps, which is a quite dirty workaround.
Cheers.
Other competitors have noted this in the past (see https://github.com/stanfordnmbl/osim-rl/issues/176). The linked thread has some suggestions for what people in the past have done to deal with this issue.
The specific cause of this is possibly due to a combination of the high accuracy (i.e., low integrator error tolerance) of the simulator and the model being in a state where the step being integrated becomes quite “stiff”. For example, some cases when the integrator will slow down is if there are sudden, large changes in forces (e.g., if the foot is penetrating the ground quickly or going far into the ground, or if joints are hyperextending which puts the muscles in a high, passive state). The integrator tries to slow down these events since these sudden changes in high forces might lead to large errors if the integrator continues to try to take large steps.
Another method competitors have used in the past was to decrease the accuracy of the integrator to speed up simulations when first starting to train, then increase it to the original value to refine the controller. You have to be careful with how “loose” you make this setting, and there can be issues if the value is set to be too large (see some discussion at https://github.com/stanfordnmbl/osim-rl/issues/138).