Dear Luka,
I am one of the organisers of the ImageCLEFcoral task.
I can confirm we are the DrawnUI task is using the same evaluation script as the Coral task. We also use the same methodology as past ImageCLEF annotation tasks.
I can also confirm that the evaluation does NOT ignore classes, you can check the line where is written “if predictions[image_key].get(widget_key):”. Therefore is only calculate if the class is correctly identify.
If you wish to calculate only the precision/recall ignoring the box overlapping, you can change “if iou>0.5” to “if iou>=0.0” or remove the conditional.
We appreciate that a system can be evaluate in many ways. Unfortunately, aicrowd only supports two metrics at this moment. The given evaluation gives an overall score over all the images and classes.
Hopefully the resources given by this or other tasks are useful for you and the CV/ML community, they have been created with lot of voluntary effort!!! (from Dimitri and many other people). If you plan to further work on this task and submit the results in a journal, I personally encourage you to provide a further analysis of the results. As you saw in the ImageCLEFcoral 2019 paper, further analysis was also done to identify the accuracy by class (unfortunately this needs to be outside aicrowd as it only support 2 metrics).
Any constructive feedback is always welcome and, indeed, encourage. Therefore, thanks a lot for taking the time to check the script.
Finally, as you are a participant of the DrawnUI task, I would like to encourage you participate in the coral task. We made sure that both tasks shared the same submission format and evaluation in order to facilitate participants to submit results in both tasks. The time is very limited but if you already have your work ready for the DrawnUI task, it would be “easy” to train it on a different image collection (the coral) and indeed very interesting to see how the image collection affects the approach.
Best,
Alba