Hello @jon_grantham
The evaluations run on AWS EC2 instances. The resources available are as follows
GPU enabled flag in aicrowd.json
|
vCPUs | Memory | GPU | AWS instance type |
|---|---|---|---|---|
true |
4 | 16 GB | NVIDIA T4 | g4dn.xlarge |
false |
4 | 16 GB | - | m5.xlarge |
Note:
During the evaluations, you will receive a proxy NetHack env object instead of the actual environment. This proxy object talks to the actual NetHack env over the network and returns the values as needed. We do this to prevent participants from tampering with the env. This also adds an overhead. Based on our benchmarks, a single env should roughly give a throughput of 1500-2000 steps/second. Using something like a batched env increases the throughput.
Hope this helps and please feel free to reach out to us for any help.