Hi All,
Few contestants have raised the issue of leaderboard probing and ADDI wants make its stance clear on this issue. Here is the official comment:
“Per Section 10 of the Challenge Rules, ADDI has determined that leaderboard probing undermines the legitimate operation of the Contest and is considered an unfair practice. Therefore, any Entrants engaging in leaderboard probing will be disqualified, in accordance with the Challenge Rules and at ADDI’s sole discretion.”
Hello everyone, please tag me if you are referring to my submissions, it is hard to keep track of all the forum.
I want to raise a point, that most ML competitions are very far from production-grade code, and are focused on a single goal - optimize scoring metric (if they overfit, do not generalize, etc - this is not a problem for participants if the goal is achieved). The host on their hand gets all the interesting ideas that were generated during the competition.
If an organizer wants to prevent leakage, and make the competition fairer, steps are done in advance by designing the competition itself. You can’t just leave a private target somewhere and say “guys don’t use it, it’s against rules”.
So it is very strange to see a sentence “Entrants engaging in leaderboard probing will be disqualified”, when most of the participants submitted “all 0, all 1 scores” to figure out public class distribution. Leaderboard probing is a part of the competition, people fine-tune models, class balancing, and probabilities post-processing when you use the public score as feedback is a probing. Could someone please give a robust definition of probing?
Why do we need scoring on the 100% of test data, when it creates that many problems?
add: overall I believe that it would be great if clear rules are established (that are reasonable for participants, aicrowd, and the host), and we compete within these rules. As opposed to “everyone interprets rules as they want, the host decides if winners are eligible in the end”.
I find it very interesting that you are rationalizing your behavior and also blurring the line between something that, in my mind, is pretty obvious.
Personally, I see this narrow definition of probing: if one was to identify the true label for specific rows from the test set, and then adjust its predictions accordingly. This has zero merit on replicability.
The competition is set to advance the science behind Alzheimer detection.
Rationalizing that host will get all the interesting ideas from others is not an excuse for one to abuse the flaws of the competition. How can it make sense to award the top prize to a model that has no merit?
Also, blaming the organizers for having designed a competition with a flaw is astonishing.
Not everything is perfect right from the start.
Just see how much debugging has occurred since the launch.
AIcrowd staff is working hard, and readjusting things when needed.
Flaws are uncovered, tweaks are made, clarification statements are posted, fixes are applied.
The previous competition I took part in was continuously getting better as staff made changes throughout the competition, by listening to its participants.
Here, once again, the participants have raised a point, and action was taken.
You request further clarification, that’s fair. We will wait for the response as well.
Congrats @tymur_prorochenko on that significant jump to 0.597 on the public leaderboard.
Quite a leap!
Hopefully, this is a legit submission.
Hi All
Keeping all the queries and concerns raised by the participants in mind, we have concluded that only 40% of the private data, whose score is not visible on the leaderboard, will be used for the final ranking.
Thank you all for raising the concern and helping us improve
Do not worry, I also root for an honest competition 
I’ve sent my thoughts on how the current competition design may be cheated and how it could be improved. I believe that scoring on the private data only is a great step and solves some of the problems. Let’s see if other improvements will come as well.
Many thanks for your clarification. It makes sense to split test data into public set and private set. However I still have few questions.
Is it legitimate to probe the LB to get more ground truth to train models? Could we exploit the leaderboard feedbacks to potentially improve the score - public and/or private ?
Or is it still depending on the ADDI to decide which LB probing actions are accepted after the competition?
Cheers,
Hi,
Our stance on leaderboard probing is the same. We consider it an unfair practice and it can lead to disqualification