Do you trust your Leaderboard Score? This is a simple but fundamental question. I personally have not found a good correlation between my local cv score and the public leaderboard one. To be more precise, in my case I see a good correlation for models that scores in the high 0.6xx low 0.7xx, however when considering my best performing models (low 0.6xx, high 0.5xx on local cv) the correlation seems to be completely broken.
My gut feeling is to trust more my local cv and expect a huge shake-up for the final leaderboard, but I’m interested in your experience. It may very well be the case that I have just not found yet a good validation scheme.