Evaluation metrics

Indeed, the principal score is RMSE on the 2D distance. There’s just one tweak - we only take the top 90% of results to calculate the score, getting rid of some potentially bad outliers.

Coverage is secondary, it’s simply showing how many of the missing locations have been predicted. Predicting all 100% may lead to worse results (or maybe not for you?). It’s generally a skill by itself to predict those locations that you are not confident in and rather exclude them (give a NaN). However, a minimum of 50% is required and we may also decide to give an award to a team that has a high coverage while maintaining a good RMSE, even if it’s not the very best.

2 Likes