Note for our final evaluation

Amazon KDD Cup 2024: Multi-Task Online Shopping Ch
#4

Hi @yilun_jin ,

Thanks for your notice.

I would like to confirm one question about the deadline. If a submission is made before the deadline but finishes running after the deadline, will it be considered in the final evaluation?

Thanks for your attention,
Best,
Pengyue

3 Likes
#5

I just confirmed from aicrowd that it will be considered.

#6

@yilun_jin

I would like to clarify it more precisely.

The deadline for making an eligible submissions: July 10th 23:59 UTC - is that correct?
If we submit before this deadline and the run finishes afterwards, it is an eligible submission.
If we submit after this deadline, the run will not be eligible

Is that correct?

The deadline of 12th of July is for selecting our top submission, but the submission has to be eligible and has to be submitted before July 10th 23:59 UTC?

1 Like
#7

Deadline for making eligible submissions is 23:55 UTC July 10th, instead of 23:59. All submissions before that would be counted eligible, regardless of when it finished. However, all submissions that are submitted after that would be counted ineligible, regardless of when you fill in the submission selection form.

Hope that helps.

#8

Hi @yilun_jin
I uploaded git tag before 23:55 UTC, but I couldn’t put tag for submission this code before 23:55 UTC.(don’t show issue). but UTC 00:00(July11), submission system automatically start submission.
How about these situation submission? include final submission or not?
ex)

#9

Hi @yilun_jin ,

Thank you for organizing the competition. I have two questions about selecting submission IDs.

  1. If one team does not fill out the form, will the top-2 submissions from each track of their team be automatically selected and rerun with different random seeds?
  2. Additionally, for the two selected submissions, will the final score be taken from the best score among all runs or the average score?

Best,
Pengyue

#10

Your case is noted. I am working with aicrowd folks on this.

1 Like
#11
  1. Yes. If a team does not fill in the sheets, we will take the top-2 submissions by default.
  2. We will take the higher one instead of the average.
#12

Reply from aicrowd folks.
If your issue is created at roughly 00:00 UTC July 11, it will be counted. We will manually mark them.

#13

Hi. @yilun_jin
thanks for your correspondence. I will check manually mark issues before selection of submissions

#14

@yilun_jin , Sometimes the exact same code will succeed one time and fail another time. For example we submitted the exact same code [here] and [here]. The first succeeded and the second failed. During re-run what happens if code fails that has previously succeeded? Will the admins run it a second time?

Also, can you tell us why the second link above failed?

When we select our final 2 submissions for each track, should we just select our best scoring submission twice in case it fails the first time it is re-run?

#15

Hi @yilun_jin ,
Thank you for hosting the competition. While selecting my final submission, I have two questions.

  1. Among the submissions marked as “Evaluation Succeeded,” there are some submissions where the score is not displayed. This information is necessary for selecting the final submission. Is it possible to check these scores? Examples are here and here, etc.

  2. Sometimes, even with the same code, there are submissions that failed due to “Evaluation timed out.” In such cases, I believe some of these might succeed if submitted again. Is it possible to select one of these as one of the two final submissions?

#16

@Chris_Deotte It may happen that a valid submission may fail due to network connection issues (which is what happened to the second link). This is a problem we encounter since the beginning, but fail to completely address. We apologize for that.

During re-evaluation, we would make sure that all submissions will run through to get scores, so feel free to select two different submissions.

Hope that helps.

1 Like
#17

Hi @wakawakawakawaka ,

  1. I manually check the logs to see the final scores. They have been updated on your issue page.

  2. It is possible to submit them, but since you did not see the final scores, you have to do that at some risk. Apologies for the occasional network issue.

#18

Hi @yilun_jin ,

Thank you very much.
Regarding the answer to question 2, I understand well. Thank you for your detailed explanation.

Concerning question 1, I’d like to ask some additional questions.

  • I can now see the scores. Thank you very much! However, an unbelievable score was displayed. In particular, the Multiple-Choice Score was close to 0.0, which, considering the number of questions, could indicate some error in the evaluation. Or does it mean that none of the answers were correct? I’d like to request a confirmation on this. such case here
  • If possible, there are three more problems that haven’t been evaluated in the same way, which are here, here, and here that’s all. Would it be possible for you to evaluate these as well?

Best,

#19

Hi @yilun_jin , It is possible to reevaluated this issues?

here
here
here
here
here
here
here
here

just in case, thank so much, Best,
fer

#20

Hello @wakawakawakawaka ,
Regarding your low multiple-choice score, I have replied with a log snippet in the issue. Basically, your output did not conform with our parser, resulting in a very low multiple choice score. We don’t believe that there is any error with our evaluation.
Regarding your second question, I will post the scores in the issues as well.

#21

aicrowd folks are notified about that, but we cannot make sure that all these can be finished.

#22

No problem, just in case, thanks :slight_smile:

#23

We have filled out the form but have not received a confirmation email. Could you please confirm that it has been received? @yilun_jin