Evaluation resuts vary even if no changes made to the code

[rsakthivel]

Sometimes even if the submission is made without any changes to the code, the metrics vary drastically after the evaluation. I am not able to figure out why this is happening. Could someone help me understand why this might be happening?

[arav_agarwal]

There’s inherent randomness in the training process; RL exacerbates this randomness.

[siddha_ganju]

we just pushed updates to the eval code and starter kit including how the metrics are calculated.

Recent changes to the StarterKit and Code Documentation

I found a major bug: the bounding box that checks for out-of-bound wheels is rotating in the opposite direction to the actual car. It can be fixed simply with angle *= -1.0 in l2r.envs.utils.py: Geolocation.getCorners(). It changes nothing for corners in the cardinal directions (North, East, etc.), but when going across corners oriented diagonally (South-East, North-West, etc.) the bounding box is completely sideways relative to the car and effectively 0.66 meter too wide on each side. The ya…

[rsakthivel]

Yes, thats right, it might be the case for training… but the submission that I make uses a trained agent and only runs the evaluation. Will that be the case even if we use a trained agent with frozen weights ?