Ideas for better sim2real transfer

Hi Everyone,

After analyzing some of the submissions from the simulator and the real-world track, I get the insight that most of the submissions are struggling on the real track due to the differences we have on the real track vs the simulator. Namely the walls and the extra artifacts at the place we are running the simulations.

I want to share a few ideas that may improve your solutions, do try them out if you think they are helpful.

  1. Train the neural network on a crop of the lower center portion of the image instead of the entire image, such that only most of the road is visible and not the artifacts at the top part of the image.

  2. Train the neural net with random boxes augmented into the images (or any other augmentations) - If using on policy methods like PPO, this generally works well when doing only in the training phase and not during experience collection.

  3. Apply some smoothening on the agent’s actions during inference. So that a sudden change in the action is not incorporated easily.

  4. Try out your models on the video that was shared and the videos on the leaderboard to check where they go wrong. The preprocessing is a simple downscaling by 4x and grayscale conversion. image[50:] might be a good starting point but you can try out different values.

  5. Try some hybrid computer vision methods.

If you have other ideas on what we can share to help with your real track submissions, do let us know.

All the best! Happy Deepracing.


Hi, I did not get anynew results on real track since 20 Nov. When I get my results?

Hi @azam_kamranian

This was your last successful submission on 24th November which is already updated since a few days. Failed submissions are not run on the real track. But seems like the your submission was consistently turning towards the right side so the score is lower than your previous submission.

Please check all your other successful submissions
after 20th November and let me know in case any are missing, we were running the best submission per user per day until recently. We’ll run any older ones in case any are missing.

Please let me know if I can help with anything else, I’ll try my best but might be bit delayed since I’m travelling today.

you are right… sry, my mistake. I was checking the “real track board” only… I thought all results display there.