Hi @da465f5e,
Thank you for the thoughtful comment, and I personally understand where it comes from.
And I was expecting something along these lines when I wrote my previous comment.
Much of your comment is based around the idea of “releasing the test set”.
In context of DIBRD, the actual test set (with labels) is one google search away. When I included the link, it was also an attempt at being more transparent. Security through obscurity is a myth ! A handful of people who already were abusing the dataset had an unfair advantage which many other had no clue about. Even in a completely competitive setup, I would be a lot more comfortable knowing that everyone had equal access to all information.
Coming to ORIENTME. Great suggestion about actually not releasing the test set. Infact, a lot of the research challenge we run do not actually release the test set at all. But given that those competitions expect code submissions (imagine elaborate code repositories with exotic software runtimes), they happen to have a huge barrier to entry for participants, especially the ones quite new to the field.
And more so, in context of ORIENTME, have you considered that maybe the actual insight on how you can use the test data cleverly was the takeaway we hoped the participants would arrive independently at ? Historically speaking, many hard problems, and many amazing results have had a key moment like that : a simple solution hugely outperforms the obvious more sophisticated solution. As mentioned in this response, one of my most favorite examples of this phenomenon is the results detailed in this paper : https://arxiv.org/pdf/1505.04467.pdf
And thank you for highlighting our faults, we happily acknowledge that this is a huge learning experience for us, with every new challenge that we run. And we will continue to try and improve the experience we bring for the participants. And thank you for all the feedback here, we will weigh them in in the design process for the next iteration of AIcrowd Blitz.
Thanks,
Mohanty