Unfortunately, I have to inform you that we have to switch to the evaluation via the CSV files.
Rules / Guidelines:
Due to the circumstances, the Deadline is postponed to 16.5. 2021 9 pm CET. We hope this will give you some additional time. Anyway, the deadline for submitting the working notes paper have to remain the same (28.5. 2021).
All Participants are allowed to submit up to 10 CSV files.
Hey @picekl, I don’t see many submissions on the respective page. Does this means that there are not much contestants, or is their any private channel where the results are shown for the submissions. As for the new system, are we only allowed to make 10 submissions or is it 10 submissions/day? Also, how will we see the score?
As in other CLEF Challenges, all participants will be allowed to submit up to 10 submissions overall. The result will be available once I will perform the validation manually (through my script). Once it will be done, probably Sunday / Monday, you will find the results on the given URL.
Hi @picekl,
There seems to be a corrupted image in the dataset that was just released ( id 0a51550d-2c21-468c-8256-1fe006ae2031). Should we simply skip it (don’t add it to the csv) or add it with a random prediction?
Also, another question that I have is how the evaluation will be done? Which submission will you use for the final score, the best one or the most recent one?
You are right, there is indeed one corrupted image. Please add a random prediction for this one.
All 10 submissions will be considered as final. Consider submitting it to follow your publication “findings”. For example 1st as a baseline and then as increments based on your experiments,e.g., different architectures, input sizes, tricks etc.
This is a pretty old school technic, but we will order all the submissions from all the teams into the leaderboard. To my best knowledge, We currently have 6 teams, so there is gonna be up to 60 scores on the leaderboard.
Hope this helps. Let me know if you will need anything.
As you might have some technical issues with the submission files, we decided to provide another 5 test submissions for each team. We won’t give you the measured score but just the info that the evaluation went well. This should help you all to prepare good final submission files.
Please send the test submission to lukaspicek@gmail.com. I will evaluate it ASAP and will let you know the “result”. Keep in mind that This is a limited offer, so do not hesitate to send it to me. I won’t be able to evaluate your test files after 15.5. 21:00. All the focus after the deadline will be targeted upon the final evaluation.
