Evaluation time

Flatland Challenge
#1

Hello,

Can you please explain how time limit works.
Do you count the whole time of evaluation? Or you remove time used for environment calculations?

What is time limit? 8 hours?

Thank you.

1 Like
#2

We have per step timeout of 15 minutes while the overall submission timeout is 8 hours.

#3

Dear @shivam,

My question is about overall submission time limit.
I am interested how you measure time for it.

I see that submissions that evaluated more than 8 hours are still scored.
So i wonder do you mesure only our code time.

cc @mlerik :wink:

#4

Can you share ID/link of the submission you are referring above?

The exact timeout from participantsā€™ perspective is 8 hours, while we keep a margin of 2 hours (making it total of 10 hours) as overhead for docker image build, node provisioning, scheduling, etc. If any submission has run for longer time than this, we can look further why timeout wasnā€™t respected.

#5

Here is an example:
https://gitlab.aicrowd.com/vitaly_bondar/flatland/issues/4

It shows: ā€œTotal Execution Time :hourglass: : 52061sā€, what is more than 14 hours.
And anyway it is scored.

I hope that you will review all submissions, not only submissions of our team to keep fairness in the competition :slight_smile:

#6

HI @vitaly_bondar,

Thanks for sharing one example submission. Yes, it will be checked for all the submissions.

#7

So what is the maximum time limit then? (for the ā€œTotal Execution Timeā€ displayed for each submission).

Is it 8 hours = 28800 seconds?
Or is it 10 hours = 36000 seconds ?

My latest submission shows a Total Execution Time of 28845 seconds (so only 45 seconds more than 8 hours). Should I try to optimize its runtime? (or just submit again and hope Iā€™m luckier next time?)

Or, in fact, can I use all the way up to 36000 seconds, in which case, I could really use this extra time :slight_smile:

Also, it would be nice to have the maximum runtime enforced. Disqualifying some submissions later on is quite bad. Anyway, in my case this is the 1st submission which exceeded an 8 hour ā€œTotal Execution Timeā€, so a decision here would be welcome.

2 Likes
#8

@mugurelionut Please use 8 hours as the time limit, we have updated our evaluation phase to strictly enforce it going forward.

NOTE: Your submission having 28845 seconds as total execution time is safe with 8 hours time limit as well. Non-evaluation phase takes roughly 5-10 minutes which is included in timing visible above.

Total Time = Docker Image Building + Orchestration + Execution (8 hours enforced now)