The test dataset that will be used to produce the official ranking is now released and the HTREC evaluation period has started!
The evaluation period ends on 11:59 June 8th, 2022, Anywhere on Earth. Please note that not all of the test instances will be used to produce the official ranking, which might differ from that on the leaderboard. The official ranking will be announced on 11:59 July 1st, 2022.
Hi, I’ve uploaded two new submissions on the new Eval set. However, there’s no change in the results.
It still says the last submission was uploaded on 6th May.
Hi, I see two new submissions, one with a mistake (the first) and one (187047) that was graded successfully (-6.1 CERR). The leaderboard remains unchanged because it still shows the previous scores.
It would be really helpful if we can see the scores for every experiment we perform. We need to know how a particular model is performing on the actual eval set.
It’ll help keep track of the experiments we perform.
Not sure I follow, aren’t all the submissions (5 per day) scored?
They are but I can’t see the scores each submission has .
You should be able to see this if you filter the submissions to your only (an option up on the left on my screen). Also, each submission has a unique ID, returned to you, so you should be able to check.
Let’s test it with the new baseline I uploaded to assist the challenge. Can you please use it to submit and see whether you get the CERR right?
Hi @nikhilbyte,
You can check out the scores of your submissions here. The view button should also work now.
Regards
Ayush
