Get input actions given directly to simulator for creating Imitation Learning data

Learn-to-Race: Autonomous Racing Virtual Challenge
#1

I am trying to record a human-performed demo using the env.record_manually function. I am able to get observations but not the actions. I assume this is because during record_manually the actions are fed via Keyboard to the simulator and not through env.step and thus not captured anywhere in the python code. Can we query the simulator to get last action performed ? Or can we recover action somehow ?

If not record_manually, is there an another way to get actions+observations while driving the car ourselves?

2 Likes
Collecting Data through Arrival Simulator
#2

Yes, the reason why recording transitions from keyboard-based actions is not supported is, indeed, because it bypasses the L2R framework: the python code would then be unable to intercept the commands.

However, we have contacted the developers of the simulator and will continue to look into it!

2 Likes
#3

You can just make your own subclass of BaseAgent that uses the keyboard to send the actions with submit_action(). I have done it for a gamepad.

2 Likes
#4

Also you should not use env.record_manually because the car behavior is different when playing in-game with the keyboard from the actual environment.

In-game, if you keep holding the brake key the car eventually shifts to the Rear gear automatically and goes backward. That is not the behavior in the environment: the car will never move backward.

1 Like
#5

@max333 Agreed on not using record_manually, I came to understand this when I solved my original problem as mentioned below:
Although I did not used BaseAgent class, I wrote a small pygame wrapper over the env, to take input from keyboard and display the screen inside pygame (I can still see the car in simulator, since the pygame window is small) and recorded my observations that way.
Thanks for your inputs.

2 Likes
#6

Hi @shivansh_beohar can you explain the process of recording the driving experience , how do we use pygame wrapper to interact with the simulator while driving manually? Any resource for the same would be appreciated. Thanks!

1 Like
#7

Create a simple pygame program which takes input from keyboard, and displays frames.
Pass the input to the simulator and display the RGB images on pygame window. According to your requirement either save the image,input pair in memory and dump them altogether at the end of the episode or save as it is collected.

1 Like