Note for our final evaluation

pengyue_jia3 · July 11, 2024, 1:04am

Hi @yilun_jin ,

Thank you for organizing the competition. I have two questions about selecting submission IDs.

If one team does not fill out the form, will the top-2 submissions from each track of their team be automatically selected and rerun with different random seeds?
Additionally, for the two selected submissions, will the final score be taken from the best score among all runs or the average score?

Best,
Pengyue

yilun_jin · July 11, 2024, 2:43am

Your case is noted. I am working with aicrowd folks on this.

yilun_jin · July 11, 2024, 2:45am

Yes. If a team does not fill in the sheets, we will take the top-2 submissions by default.
We will take the higher one instead of the average.

yilun_jin · July 11, 2024, 1:02pm

Reply from aicrowd folks.
If your issue is created at roughly 00:00 UTC July 11, it will be counted. We will manually mark them.

tereka · July 11, 2024, 1:20pm

Hi. @yilun_jin
thanks for your correspondence. I will check manually mark issues before selection of submissions

Chris_Deotte · July 11, 2024, 3:22pm

@yilun_jin , Sometimes the exact same code will succeed one time and fail another time. For example we submitted the exact same code [here] and [here]. The first succeeded and the second failed. During re-run what happens if code fails that has previously succeeded? Will the admins run it a second time?

Also, can you tell us why the second link above failed?

When we select our final 2 submissions for each track, should we just select our best scoring submission twice in case it fails the first time it is re-run?

wakawakawakawaka · July 12, 2024, 1:19am

Hi @yilun_jin ,
Thank you for hosting the competition. While selecting my final submission, I have two questions.

Among the submissions marked as “Evaluation Succeeded,” there are some submissions where the score is not displayed. This information is necessary for selecting the final submission. Is it possible to check these scores? Examples are here and here, etc.
Sometimes, even with the same code, there are submissions that failed due to “Evaluation timed out.” In such cases, I believe some of these might succeed if submitted again. Is it possible to select one of these as one of the two final submissions?

yilun_jin · July 12, 2024, 2:28am

@Chris_Deotte It may happen that a valid submission may fail due to network connection issues (which is what happened to the second link). This is a problem we encounter since the beginning, but fail to completely address. We apologize for that.

During re-evaluation, we would make sure that all submissions will run through to get scores, so feel free to select two different submissions.

Hope that helps.

yilun_jin · July 12, 2024, 2:33am

Hi @wakawakawakawaka ,

I manually check the logs to see the final scores. They have been updated on your issue page.
It is possible to submit them, but since you did not see the final scores, you have to do that at some risk. Apologies for the occasional network issue.

wakawakawakawaka · July 12, 2024, 6:44am

Hi @yilun_jin ,

Thank you very much.
Regarding the answer to question 2, I understand well. Thank you for your detailed explanation.

Concerning question 1, I’d like to ask some additional questions.

I can now see the scores. Thank you very much! However, an unbelievable score was displayed. In particular, the Multiple-Choice Score was close to 0.0, which, considering the number of questions, could indicate some error in the evaluation. Or does it mean that none of the answers were correct? I’d like to request a confirmation on this. such case here
If possible, there are three more problems that haven’t been evaluated in the same way, which are here, here, and here that’s all. Would it be possible for you to evaluate these as well?

Best,

fersebasIn · July 12, 2024, 6:58am

Hi @yilun_jin , It is possible to reevaluated this issues?

here
here
here
here
here
here
here
here

just in case, thank so much, Best,
fer

yilun_jin · July 12, 2024, 8:33am

Hello @wakawakawakawaka ,
Regarding your low multiple-choice score, I have replied with a log snippet in the issue. Basically, your output did not conform with our parser, resulting in a very low multiple choice score. We don’t believe that there is any error with our evaluation.
Regarding your second question, I will post the scores in the issues as well.

yilun_jin · July 12, 2024, 8:42am

aicrowd folks are notified about that, but we cannot make sure that all these can be finished.

fersebasIn · July 12, 2024, 9:02am

No problem, just in case, thanks

fersebasIn · July 12, 2024, 10:36am

We have filled out the form but have not received a confirmation email. Could you please confirm that it has been received? @yilun_jin

pp1 · July 12, 2024, 10:47am

boren · July 12, 2024, 1:38pm

Hi @yilun_jin ,Could you please help notify aicrowd folks to reevaluate these issues?

wakawakawakawaka · July 12, 2024, 5:54pm

Hi @yilun_jin ,
Thanks so much for your help! I really appreciate it!

boren · July 13, 2024, 1:51am

[quote=“boren, post:25, topic:10686”]
Hi @yilun_jin ,Could you please help notify aicrowd folks to reevaluate these issues?
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/420
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/418
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/416
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/415
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/386
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/385
https://gitlab.aicrowd.com/boren/amazon-kdd-cup-2024-starter-kit/-/issues/390

pp1 · July 13, 2024, 2:31am

@yilun_jin Thank you for your excellent organization. I have a question could a team be notified which submissions are selected in the final testing stage?