I’m getting this error on all of my submissions recently. Code changes very minor from working submissions. Last week for first submissions I actually had this error initially, turned debug flag on with no changes, it worked, turned debug back off and then it worked.
Again today/last night, I’ve submitted multiple times with the failure. I tried enabling the debug flag in json with no code changes, and everything worked. Turned the debug flag back off, doesn’t work.
Any idea what’s happening? The only info in the issue for the submission is the text below:
The following containers terminated prematurely. : agent
Please contact administrators, or refer to the execution logs.
I’m experiencing the same here, only difference is the debug flag.
So you experience issue with the debug flag both on and off? Was there anything in the logs.
I have only had the issue trying to submit for scoring with the debug flag disabled. Whenever I try to turn debug on to see what the problem is the submission works fine so I’ve got nothing to go on. I @ mentioned the admins in the Gitlab submission issue but received no response from them.
I’ve tried running with/without GPU but there is no consistent ‘working’ state, it seems random which suggests a timing issue for the evaluation mechanics.
My agent eval code is very simple and shouldn’t have variable timing, it’s based on PyTorch so should be a really fast startup, whereas Tensorflow tends to take more time to init and allocate tensor memory.
@ross_wightman: Can you please DM the link to the issue ?
@harperj: Can you please make the necessary changes to the starter kit to include more information about the configurable timeout during the env instantiations ?
For the debug run with debug: true , it ran successfully with 5 episodes being run. The next submission had debug: false (which is the only difference) with the exactly same code — it failed (no logs can be seen unless it’s a debug mode). I ping the admin but I haven’t heard from them yet.
Just out of curiosity, I made another “identical” submission (with debug: false) and this time it worked. So it must not have been my fault.
For the debug run with debug: true , it ran successfully with 5 episodes being run. The next submission had debug: false (which is the only difference) with the exactly same code — it failed (no logs can be seen unless it’s a debug mode). I ping the admin but I haven’t heard from them yet.
Just out of curiosity, I made another “identical” submission (with debug: false ) and this time it worked. So it must not have been my fault.
This is exactly what I see, I direct messaged mohanty some of the issues.