FAQ: Round 1 evaluations configuration

NeurIPS 2020: Procgen Competition
,
#1

The following values will be set during the evaluations. Any changes that you make to these parameters will be dropped and replaced with the default values during the evaluations.

stop:
  timesteps_total: 8000000
  time_total_s: 7200

checkpoint_freq: 25
checkpoint_at_end: True

env_config:
  env_name: <accordingly>
  num_levels: 0
  start_level: 0
  paint_vel_info: False
  use_generated_assets: False
  distribution_mode: easy
  center_agent: True
  use_sequential_levels: False
  use_backgrounds: True
  restrict_themes: False
  use_monochrome_assets: False

# We use this to generate the videos during training
evaluation_interval: 25
evaluation_num_workers: 1
evaluation_num_episodes: 3
evaluation_config:
  num_envs_per_worker: 1
  env_config:
    render_mode: rgb_array

During the rollouts, we will also pass a rand_seed to the procgen env.

6 Likes
Same marks on the testing video
Performance Difference
#2

Is it still allowed to use any environment configuration during training?

Or when you say “evaluation”, you mean both training and rollouts?

#3

Helo @jurgisp

The above configuration will be used during both training and rollouts.

1 Like
Round 2 is open for submissions 🚀
Round 1 is open for submissions 🚀
#4

Can we pass additional custom parameters in env_config (for example the number of frames to stack together like stack_frames)? I assume you are talking about a update operation where those official parameters above will be replaced with the default values but whatever user-defined additional parameters will be preserved.

#5

Hello @xiaocheng_tang

Yes, you are free to pass additional parameters/flags in env_config. The only requirement is that the base env used by your gym wrapper should be ProcgenEnvWrapper provided in the starter kit.

The discussion from Rllib custom env might be useful to clear things up.

#6
Customized trainer
#7

I found some performance boost when set this to be True. Could you please explain why this is overwritten? Thank you very much!

#8

Hello @jiaxun_cui

The objective of the competition is to measure the sample efficiency and generalization of the RL algorithms. Changing any of the environment config flags is not relevant to the competition.

#9

Where can we find info on the environment where the evaluations are run? (i.e: what’s installed, and how) – thanks!

#10

Hi @maraoz,

The default runtime is as defined in this Dockerfile, which is basically nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 docker image along with dependencies listed in requirements.txt.

2 Likes
#11

@jyotish Can you explain the entry command used for the evaluation?

For example, do you run the run.sh file to start the training process?

How do you replace these above parameters by the default values?

I think this will be helpful for me to debug my code on my machine.

#12

Hello @tesla

For the training phase we run run.sh --train right now. You can specify which experiment config file to use at

You can specify the CPUs/Memory that should be used by rllib for your local runs using

Let me know if you were looking for something else.

#13

If you are okay with rllib taking up all the resources available during your local runs, you can directly run

python train.py -f experiments/<experiment-file>.yaml

PS: You need to update the EXPERIMENT_DEFAULT variable in run.sh for us to pickup the experiment file.

#14

Yes, my question is that if you want to change these parameters such as env name, do you change the experiment config file specified in run.sh or other methods?

#15

So, you will first scan the run.sh file, and then modify the experiment config file specified by EXPERIMENT_DEFAULT variable. Do I understand correctly?

#16

Hello @tesla

Yes. We get the experiment file to use from run.sh and override the necessary values in the experiment yaml file and start the training/rollouts.

You only need to update the experiment file to use in run.sh. Any other change will be in the respective experiment yaml file. For example, for env name, it will be here,

#17

Ok, thank you! I think I already know.

1 Like
#18

If you’re calculating GPU RAM constraints, bear in mind that the evaluation configuration has an additional video rendering worker, not just the trainer and rollout workers! Costed me ~5 submissions and asking @jyotish to realize this ^^

1 Like
#19

Thank you, it confused me some days.

#20

Dear organisers,
Is it possible to remove ray dependency from second round?
From my perspective Ray looks like a too heavy solution for this task.
Do you have any pure TF/Pytorch baselines to start from?