Running the evaluation worker during evaluations is now optional

jyotish · August 28, 2020, 9:08am

Hello!

We are making the evaluation worker that we run during evaluations optional. We use this worker to generate videos during training.

What will happen if I disable the evaluation worker?

Videos will not be generated during training.
You can use one additional rollout worker in its place (by increasing num_workers by 1). This is useful for those who are experiencing low throughput.
rllib's ARS, APEX implementations should work. They need more than one worker to work. A single evaluation worker results in training failures.
The custom random agent code in the starter kit works with no additional modifications.

How can I disable the evaluation worker?

You should set disable_evaluation_worker to True in your experiment YAML file.

For example,

procgen-ppo:
    run: PPO
    env: procgen_env_wrapper
    disable_evaluation_worker: True
    stop:
        timesteps_total: 100000

jyotish · August 28, 2020, 9:42am

joao_schapke · August 28, 2020, 10:45am

Great! Although, is there any way to map our own metrics in the grafana dashboard (e.g. training mean return)?
Edit: Found it, there is an option in the dashboard to plot any metrics your code outputs in each training iteration. Really useful

dipam_chakraborty · August 28, 2020, 9:14pm

Hello @jyotish

I’m getting this error on local machine, is it some ray version issue or something else? Ray version installed is ray[rllib]==0.8.5

File "<...>/python3.7/site-packages/ray/tune/experiment.py", line 170, in from_json
exp = cls(name, run_value, **spec) 
TypeError: __init__() got an unexpected keyword argument 'disable_evaluation_worker'

jyotish · August 28, 2020, 10:01pm

Hello @dipam_chakraborty

This is not a ray specific issue. Infact, there is no such flag in ray. We pop this flag before passing it to run_experiments . You can make this change in your train.py to run it locally.

dipam_chakraborty · September 5, 2020, 1:31pm

Hello @joao_schapke

I’m a complete rllib noob, can you please share some code snippet or link of how to output the custom metrics.

jyotish · September 5, 2020, 1:52pm

Hello @dipam_chakraborty

You can add custom metrics using callbacks

github.com

AIcrowd/neurips2020-procgen-starter-kit/blob/master/callbacks.py


from typing import Dict

import ray
from ray.rllib.env import BaseEnv
from ray.rllib.policy import Policy
from ray.rllib.policy.sample_batch import SampleBatch
from ray.rllib.evaluation import MultiAgentEpisode, RolloutWorker
from ray.rllib.agents.callbacks import DefaultCallbacks

import numpy as np

class CustomCallbacks(DefaultCallbacks):
    """
    Please refer to : 
        https://github.com/ray-project/ray/blob/master/rllib/examples/custom_metrics_and_callbacks.py
        https://docs.ray.io/en/latest/rllib-training.html#callbacks-and-custom-metrics
    for examples on adding your custom metrics and callbacks. 

    This code adapts the documentations of the individual functions from :

This file has been truncated. show original

Visualizing the custom metrics

Once you add your metrics here, we will collect them during evaluation and you can visualize them on the submission dashboard. To visualize your custom metrics,

Open the dashboard.
Hit esc key and you should see a few dropdowns at the top of the window.

Select the metric(s) you want to visualize

the_raven_chaser · September 11, 2020, 2:19am

Hi @jyotish

How should I change train.py to disable the evaluation worker locally?

jyotish · September 11, 2020, 6:47am

Hello @the_raven_chaser

The evaluation worker won’t run locally unless you pass the evaluation config. If you are asking about dealing with disable_evaluation_worker flag, yes, you can pop it from the config in train.py so that it works locally.

the_raven_chaser · September 15, 2020, 3:52am

Thank you @jyotish , I see now.