Hi @amapic, let me get back on this after confirming with organisers.
Meanwhile we can create new questions instead of following up on this thread, it will make QnA search for future simpler. 
How come some of my subs took 14h and didn’t fail if the limit is 8h? Then again, how am I supposed to know that timeout is set to 8h? Where is it written? I also thought for a moment that you keep changing the timeout limit? Can you confirm that this is not true?
inferencing time is way off. Locally my model on 1080ti takes ~10 minutes to execute so obviously it runs on CPU when submitted.
@ValAn No, I can confirm the timeouts haven’t been change b/w your previous and current runs. The only issue has been timeout wasn’t implemented properly in past and it can be reason why your previous (1 week old) submission get missed from timeout.
We can absolutely check why it is taking >8 hours instead of ~10 minutes on local. Can you help me with following:
- The local run is with GPU? I can check if your code is utilising GPU (when allocated) or running only on CPU for whatsoever reason.
- What are the number of images when you are doing locally? The server/test dataset have 32428 images to be exact, which may be causing higher time.
I think specs for online environment would help a bit in case there is significant difference from your local environment: 4 vCPUs, 16 GB memory, K80 GPU (when enabled)
Hello @shivam!
Me as @ValAn I feel this kind of information (resources, time limit, and so on) should be better documented, not just for this challenge but for every challenge you guys host.
Also it seems you are constantly improving your platform, which is something great, but as a user I don’t know when you guys do it, nor what do you update, and when I get some inconsistency with previous experience I become nuts about which point I’m missing.
That being said, I think the timing issue is more related to the time for accessing the images in disk rather than the GPU time, which is kind of… sad.
I’ll definitely write more about that when this round is over. 
1 Like
Hi @ignasimg,
Thanks for the suggestions.
I completely agree that we need to improve our communication & orientation of information for providing seamless experience to participants.
We would be glad to hear back from you after competition and looking forward for the inputs.
I checked all the submissions and unfortunately multiple participants are facing same issue i.e. GPU is being allocated but not used by submissions, due to cuda version mismatch.
For making GPU work out of box, we have introduced force installation as below in our snakes challenge evaluation process:
conda install cudatoolkit=10.0
This should fix the timing issues and we will continue monitoring all the submissions closely.
@ignasimg I have verified disks performance and it was good. Unfortunately on debugging, I found your submission faced same issue i.e. cudatoolkit=10.1 due to which it may have given the impression that disk is the bottleneck (but it was GPU which wasn’t being utilised). The current submission should finish much sooner after condatoolkit version pinning.
@shivam Thanks for your explanation. Do you know which day you force installation of cuda 10.0 ? It could explain some problem I had.
Hi @amapic, I have started force cudatoolkit=10.0 installation at same time above announcement is made i.e. 14 hours ago.
Edit: I remember the conda environment issue you were facing, and it isn’t related to it.
1 Like
Congrats to all. It’s been 3 days since the competition finished and we haven’t got any info how we shall proceed from here. What happens next @devops?
Hi @ValAn, participants,
Congratulations to all for your participation.
There is no update right now. Organisers will be reaching out to the participants shortly with details about their travel grants, etc and post challenge follow-up.
thank you. Looking forward to it.
Best of luck to all and see you on the next challenge
thanks, kudos to you too. See you at the next stage, I assume.