Announcement: Debug your submissions

Unity Obstacle Tower Challenge
#1

Hi everyone,

We noticed that in the challenge multiple participants were facing problems to debug their submissions. The debugging was even trickier because logs aren’t accessible to participants directly and they have to wait quite sometime for any admin to share it.

To make this whole process simpler we are introducing a debug flag for your submission. The debug mode has following features:

  • The environment seeds used in debug mode is different than “actual” submissions.
  • The scores for debug mode will be 0.01/0.01 and actual scores will not be reflected back.
  • The debug submission will be counted towards your submission limit/quota, to prevent misuse.
  • Agent/Your code’s logs will be available directly to you by default.

How to submit in debug mode?
You need to add debug: true in your aicrowd.json file (default value is false if not present). The final file will look something like:

{
    "challenge_id" : "unity-obstacle-tower-challenge-2019",
    "grader_id": "unity-obstacle-tower-challenge-2019",
    "authors" : ["aicrowd-user"],
    "description" : "Random Obstacle Tower agent",
    "gpu": false,
    "debug": true
}

What is the difference?

This is how submission status page will look like. Notice a new field “Logs for participant reference” which will be visible for debug submissions.

image
image.png1812×790 103 KB

We hope this to be helpful addition to all the participants. All the best! :aicrowd:

7 Likes
Why I was blocked from uploading new models for 3 days
Submitting in debug mode sets leaderboard score to 0
Evaluation error:Page not found
Submission Failed: Evaluation Error
#2

Thank you for developing this API for us, this is great.

#3

Woohooo !!! Thanks @shivam !!

#4

but Sorry, I seem to test too many times, resulting in a maximum number of submissions.

#5

@ChenKuanSun Yes, the debug submissions will continue to count toward overall quota of submissions.
Given, these are treated exactly the same way as actual submissions internally i.e. utilise prod resources, we would like to stop any possible misuse. As a participant, you would have to choose wisely when & how many submissions need to be done as actual v/s debug.

1 Like
#6

What I want to know is whether there is a basic configuration that provides a similar test environment. I want to do the simulation environment on my own GCP. This way I don’t have to take up resources, and then I can adapt the configuration file and submit it.

It seems that I have tested successful files on the official docker and cannot do it in your environment.

#7

This is a great feature, @shivam! Thanks for taking the time to implement it. Hopefully it will help a lot of participants.

#8

@ChenKuanSun It looks like a good suggestion which we can try to incorporate in AICrowd. /cc @mohanty

#9

Just the GCP has a mechanism for providing direct import into the image. You can consider doing this and you can also make specifications.

#10

My question is, do you have Git LFS when you are doing a clone action to build an evaluation environment? What I am worried about is that once the evaluation environment does not use LFS clone I have a file that uses LFS, it will not be accessed correctly.
@mohanty

#11

@ChenKuanSun : Yes the evaluator can use git lfs, and that shouldnt be a problem.

And not sure what you mean by, “I have tested successful files on the official docker and cannot do it in your environment.”, if its around the software runtime packaging, you can very well drop a Dockerfile at the root of your repository, and the image builder will use that to build your image.

#12

I recently discovered that your evaluation environment uses nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04. For some people (including me) it is possible to use Cuda-10. After several tests, I learned the actual environment of aicrowd, and adjusted it to cuda 9.0 when creating images on GCP.

#13

Also in


A note should be added to the evaluation section: using nvidia-gpu should be added --runtime=nvidia
#14

@ChenKuanSun

Would you be willing to make a quick PR to the repo with this change in the readme to help other users?

#15

If I successfully test the completion environment, I should provide repo to you so that you can publish other methods like Readme.md to help other contestants.
@arthurj

#16

I made a PR just now.

#17

Hi, I used debug mode to test my submission and I tried to set it off to get actual result. But it seems it still runs in debug mode even after I updated aicrowd.json, pushed to the repo, and created a tag.

Does it take time to reflect the debug flag?
What should I do?

#18

delete debug label…

1 Like
#19

@shivam: Nudge Nudge. I see you are checking for the “debug” key and not its actual value :wink: !

2 Likes
#20

Thanks for flagging, fixed the bug and value will be checked now on instead of just key.

@tky Sorry for the inconvenience caused to you.

1 Like