@snehananavati @dipam Hi, could you also help me cancel my operation for creating my team? Thanks.
Anyone who needs their team creation reverted please dm me, mention your registered email in the message, and if the team contains more than one member, all members need to message me consenting to it.
1 Like
@dipam @snehananavati Hello! currently I can achieve about MRR@100=0.14 in the verification set of the custom division in the session_train.csv file, but the result is 0 after submitting the test set in session_test_task1.csv. I would like to know how the official process the submitted parquet file to get the final MRR value ,and this is helpful for me to troubleshoot related issues.
5 Likes
Same question. I am also curious about the official evaluation codes as I could reach MRR@100=0.3 in my self-splitting validation set from the session_train.csv, but I reached almost 0 after submitting the test set.
1 Like
@dipam ,for some complicated reasons, two members of my team (brssqtc) and (wei_zhang44) want to withdraw from my current team, these are accounts that have not submitted results, and guarantee that these two accounts will not participate in this KDD CUP in the future. This operation has been agreed by all teammates. The squad creators and members have sent you an message.Thank you.
@ECNU_Wei_Zhang Please ask all you team members to dm me and confirm that they want to leave the team.
@dipam The two members of the team who need to leave (brssqtc) and (wei_zhang44) have confirmed to you. Thank you
Hi, the last day to team up is 3rd June, 23:55 UTC.
@snehananavati @dipam thanks for providing the team merger deadline.
Does AICrowd / the challenge account for the 5 submissions per task per team limit for team merger?
Yes, every team is limited to 5 submissions per day. After merging teams, submissions limits will apply to the whole team.
Agreed, but this is the system we have, it applies to everyone. If I understand correctly it is also the system on some other platforms.
I understand it’s an outsized advantage for this challenge as it’s not a code based challenge and doesn’t have a private test set.
We’re open to alternative ideas for future challenges, please share your feedback here or in dm.
Certainly not. On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.
You system is an incentive for officially teaming as late as possible. I wish we had known.
1 Like
On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.
Apologies, I wasn’t aware of this.
I’ll mention that we generally do not run competitions without private test sets, which is where this is a much bigger issue. This was an oversight on our part.
I checked the leaderboards and see that there is one team that has managed to make a lot of submissions doing this. After the team merge deadline we’ll check for any more unfair merges, and let the Amazon team decide if they want to provide everyone else with additional submissions to keep things fair.
Thanks.
I also posted a while ago that the 5 submissions per 24 hours is not enforced at team level by your UI. It is only enforced at individual level. We track our submissions as a team to no exceed 5 submissions per day, but this is not convenient and it is error prone.