Same question. I am also curious about the official evaluation codes as I could reach MRR@100=0.3 in my self-splitting validation set from the session_train.csv, but I reached almost 0 after submitting the test set.
1 Like
@dipam ,for some complicated reasons, two members of my team (brssqtc) and (wei_zhang44) want to withdraw from my current team, these are accounts that have not submitted results, and guarantee that these two accounts will not participate in this KDD CUP in the future. This operation has been agreed by all teammates. The squad creators and members have sent you an message.Thank you.
@ECNU_Wei_Zhang Please ask all you team members to dm me and confirm that they want to leave the team.
@dipam The two members of the team who need to leave (brssqtc) and (wei_zhang44) have confirmed to you. Thank you
Hi, the last day to team up is 3rd June, 23:55 UTC.
@snehananavati @dipam thanks for providing the team merger deadline.
Does AICrowd / the challenge account for the 5 submissions per task per team limit for team merger?
Yes, every team is limited to 5 submissions per day. After merging teams, submissions limits will apply to the whole team.
Agreed, but this is the system we have, it applies to everyone. If I understand correctly it is also the system on some other platforms.
I understand it’s an outsized advantage for this challenge as it’s not a code based challenge and doesn’t have a private test set.
We’re open to alternative ideas for future challenges, please share your feedback here or in dm.
Certainly not. On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.
You system is an incentive for officially teaming as late as possible. I wish we had known.
1 Like
On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.
Apologies, I wasn’t aware of this.
I’ll mention that we generally do not run competitions without private test sets, which is where this is a much bigger issue. This was an oversight on our part.
I checked the leaderboards and see that there is one team that has managed to make a lot of submissions doing this. After the team merge deadline we’ll check for any more unfair merges, and let the Amazon team decide if they want to provide everyone else with additional submissions to keep things fair.
Thanks.
I also posted a while ago that the 5 submissions per 24 hours is not enforced at team level by your UI. It is only enforced at individual level. We track our submissions as a team to no exceed 5 submissions per day, but this is not convenient and it is error prone.
@CPMP 5 submissions per team should be enforced by the UI on a per day basis (00:00 to 00:00 UTC fixed window). We have two settings that the organizers can choose from, 24 hour rolling window and 24 hour fixed window. For this challenge its set to fixed window.
Is your team able to make more than 5 submissions in the fixed window?
yes, I have 5 submissions when my team mates have already submitted. We keep track of our submissi0ns to not exceed 5 per tak per day. W euse UTC time.
1 Like
Hi snehananavati, I missed the deadline of team up, Can I still team up now? The total number of submissions for our team is less than 5 times per day. My teammate’s id is RMSprop17.