On my latest attempts to get things running on the evaluation sever, I ran into a number of issues, some of which I managed to fix, but the current error I get baffles me. Things now get killed without any interpretable error message (at least none that is visible to me in the agent log). Everything works fine on the training server, but the agent log on the evaluation server says:
2019-12-31T23:58:12.255439177Z /home/aicrowd/run.sh: line 9: 11 Killed Rscript predict.R
PS: It would be good to make some official annoucement about the changes to the AICROWD_TRAIN_DATA_PATH environment variable on the evaluation server. Additionally, warning people about the changes to the training data file on the evaluation server would have been good (seems to no longer match the training server - at least read_csv gives different results now versus before the latest changes and different than in the training environment). Those changes took me completely by surprise - I had expected the only changes vs. before to be a scrambled/randomized
row_id column. It’s quite tedious to figure those things out via debug submissions (took me 8 submissions \approx 2 hours due to the slowness of the submission process) to trouble-shoot that.