UPDATE / EXTENSION: DSAI Challenge: Leaderboard & Presentation deadlines

Novartis DSAI Challenge
#1

Dear all,

IMPORTANT UPDATE: We have made some necessary changes to the test data set which will allow for better balance and stability as we transition to the final leaderboard, and may cause the current leaderboard to shuffle. (models will be re-run)

As a consequence of this – in addition to some general requests - we assume it is only fair to allow teams an additional non-holiday week to work on finalizing their submission.

  1. Leaderboard submissions will be frozen on Jan 10th
  2. The organizing team will decide if an additional grace period will be given for presentation submissions.
  3. Machines are accessible from now until Jan 10th

As always, thanks for your continued engagement and support of the organizing team’s efforts around your feedback throughout the initiative.

Thanks and happy holidays

2 Likes
Submission Evaluation - Queued since 2 hours
Submission Evaluation - Queued since 2 hours
#2

Hi Nick,

What do you mean by “changes to test data set”? I guess test data set (in /shared_data/data/test_data_full) stays the same, but for better balance on public vs final leaderboard, the public leaderboard splitting approach will be changed, am I right?

Thanks
Bin

1 Like
#3

Hi @bzhousd,

Yes, the splitting approach is being changed.

#4

Hi,

When Will you update the public leaderboard? Will all the submissions be updated? That would be super helpful.

Thanks!
Carlos

#5

Hi @carlos.cortes,

The submissions are being reevaluated right now. Given we have large amount of submissions i.e. 1000+ successful submissions, it will take few more hours before all the submissions are reevaluated with new dataset.

#6

Hi,

It’s been a few days and the leaderboard is still showing the old scores. Do you have any idea when will the leaderboard be updated with the new data?

Thanks!
Carlos

1 Like
#7

Hi @carlos.cortes, @all,

The scores are now updated for all the submissions and new ranks are available on the leaderboard.

We were following approach to re-evaluate one, if it fails provide feedback/fix for the submission and so on, which turned out to be quite slow. Right now, we have re-evaluated all the submissions, and submissions which have failed are being provided feedback or applied automatic patches asynchronously.

All the best and, Merry Christmas! :smiley: :christmas_tree:

#8

Hi @ kelleni2,

I know the leaderboard deadline is Jan 10th. I wonder if you could tell us the deadline’s time zone. our local time is Jan 7th now, but it is Jan 8th in Basel. Since the last submission mode is the team final model , people might submit a bad model and have no chance to submit the final model.

Thanks
Bin

1 Like
#9

Hi Bin,

I am sending an update now - as suggested in the last update - we are no longer limited by last submission, so do not worry there. We have not observed gaming and have ability to rerun on multiple now - see the coming announcement.

Jan 10th EoB EST is what we had planned - but open to feedback. (will give precise time)