Is the private environment score on public leaderboard

Need a clarification whether the score from the private environment is part of the public leaderboard score. From Kaggle its generally observed that giving away the private score on public leaderboard results in too many submissions doing metric hacking by tuning hyperparameters.

4 Likes

Hi Dipam,

Thanks for pointing this out. And yes we are aware the releasing the private score will lead to metric hacking.

But the metric hacking in this round will not help in the subsequent rounds where we will have different private environments. And finally at the end of the competitions we will have a final exhaustive evaluation where the code submissions will be evaluated on all the available public environments and all the private environments to measure both sample efficiency and generalisation.

Hence any metric hacking attempts in these rounds will not bear fruits.

The private scores in this round are made available to the participants to also prime the community of participants to get used to being evaluated on completely unseen environments.

Hope this clarifies your question.
Cheers,
Mohanty

4 Likes