Hello, our submission met an unexpected error after being evaluated for about 19 hours and having received a list of scores. The agent had successfully finished 50+ episodes with rewards, but somehow the evaluation state changed to “error” and the submission became “failed”. We have carefully inspected the logs, but it shows no error during the evaluation, so we really want to know the reasons.
Here is the end of the logs:
2020-09-18T12:25:48.915896553Z 2020-09-18 12:25:48 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:48] [Server thread/INFO]: [STDOUT]: Warning: managed 11/20 ticks this second.
2020-09-18T12:25:49.95209292Z 2020-09-18 12:25:49 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:49] [Server thread/INFO]: [STDOUT]: Warning: managed 12/20 ticks this second.
2020-09-18T12:25:50.163278504Z 2020-09-18 12:25:50 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:50] [Client thread/INFO]: [CHAT] §c§l0…
2020-09-18T12:25:50.173719115Z 2020-09-18 12:25:50 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:50] [Server thread/INFO]: [STDOUT]: SERVER request state: DORMANT
2020-09-18T12:25:50.179293551Z 2020-09-18 12:25:50 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:50] [Server thread/INFO]: [STDOUT]: SERVER enter state: CLEAN_UP
2020-09-18T12:25:50.185303103Z 2020-09-18 12:25:50 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:50] [Server thread/INFO]: [STDOUT]: SERVER request state: CLEAN_UP
2020-09-18T12:25:50.293747746Z 2020-09-18 12:25:50 aicrowd-neurips-2020-minerl-challenge-83001-9136 minerl.env.malmo.instance.1716ad[1348289] DEBUG [12:25:50] [Server thread/INFO]: [STDOUT]: SERVER enter state: DORMANT
2020-09-18T12:25:53.173893457Z 2020-09-18 12:25:53 aicrowd-neurips-2020-minerl-challenge-83001-9136 root[1348289] INFO Inference End…
2020-09-18T12:25:53.174399012Z 2020-09-18 12:25:53 aicrowd-neurips-2020-minerl-challenge-83001-9136 crowdai_api.events[1348289] DEBUG Registering crowdAI API Event : CROWDAI_EVENT_INFO inference_ended {‘event_type’: ‘minerl_challenge:inference_ended’} # with_oracle? : False
2020-09-18T12:25:53.175598194Z None
2020-09-18T12:25:53.175620833Z [0] Starting evaluator.
2020-09-18T12:25:53.175627878Z [1] Starting evaluator.
2020-09-18T12:25:53.175660057Z current total_reward in this episode is (some score)
2020-09-18T12:25:53.175667197Z [0] Episode complete
2020-09-18T12:25:53.175673348Z [0] Episode complete
… (omited many lines) …
2020-09-18T12:25:53.176192674Z [49] Episode complete
2020-09-18T12:25:53.80812693Z Ending testing phase
P.S., we use pytorch 1.6.0 and CPU mode in this submission. May I ask which version of CUDA do you use on the server? or can the server support Pytorch 1.6.0 in GPU mode? Thanks a lot!