Picking your weekly submission, and other updates

Insurance pricing game
#1

Hi all

It’s now almost been a week since our launch and we’re very happy to see such vigorous engagement with the competition! :muscle:

We’re very excited for our first weekly profit leaderboard on Saturday at 10pm CET!

I have 4 updates for you today:

  1. Choosing the submission you would like to use for the profit leaderboard (form available here)
  2. Added profitability feedback to the RMSE leaderboard
  3. Updated the RMSE-based evaluator
  4. Improved the starter-kit

1 Choosing the submission you would like to use for the profit leaderboard :calling:

You can now pick any of your previous submissions that you would like to enter for the Profit leaderboard. If you don’t make a specific choice, we will use your most recent successful submission for the profit leaderboard.

Please fill up this form to indicate your chosen submission id.
If you are part of a team, please nominate any one person from your team to fill it up.
The form closes at 10 PM CET on Saturday, 26th Dec

2 Profitability feedback on RMSE leaderboard :warning:

Upon every submission, we now include feedback regarding whether your model satisfies the non-negative training profit rule to make sure you are aware which of you submissions are eligible for the profit leaderboard. Note that models that do not have non-negative profit will not be included in the profit leaderboard.

For example, the mean model baseline (the template model) does not satisfy this rule and if I submit it, this is the warning I get (see the blue box):

highlighted_profit_warning
highlighted_profit_warning1640×1165 205 KB

3 Updated RMSE-based evaluator :new:

A very helpful member of the community highlighted a particular exploit in our RMSE leaderboard which allowed the expected claim estimates predicted for past contracts, to be informed by their future.

Therefore, we are altering the RMSE leaderboard evaluator so that predictions for each year are made with access only to data from previous years. The RMSE is then computed by gathering those predictions and using the standard RMSE formula (for details see the overview page).

This way we ensure that, in the market-simulation, your models are able to use past data when available to evaluate the future, without allowing the future to predict the past.

We have now re-run all submissions based on this evaluation metric so please check the leaderboard. Most remain unchanged.

4 Improved starter-kit :rocket:

The starter-kit now includes:

  1. One new baseline model both as a zip and as Colaboratory notebooks(for both R and Python).
  2. New test.sh and test.bat files that check your zip submissions before sending to AIcrowd (these might take a while to update)
  3. New package loading functionality in Colaboratory for R users.

Good luck!
Ali

2 Likes
(low priority) Why does evaluating take so long?
#2

Hi,

thanks for the clarification. It seems that the model chosen for the competitive leaderboard is delayed and not picking the most recent submission. For instance, submission of mine done at 11 am vs the other submissions I sent a couple of hours later (about 5 pm) the same say and before the weekly deadline (and all are marked as eligible).

Is there some delay between submission and the time it is picked for competitive evaluation? If yes how long does it take more or less?

Thanks.

#3

Hi @JeanRodriguez

Since we introduced the option of choosing submissions this week, it’s taking a little longer than expected for the leaderboard to update, apologies for that.

But be sure that the model that goes into the profit leaderboard will be the latest model submitted before 10PM CET on Saturday unless you specify a different submission before that deadline.

I believe the leaderboard will be updated in a few hours, I will make sure to reply here to bring your attention to it once it is done.

From next week onward this should be done within a few hours of the 10PM CET deadline.

1 Like
#4

Ok thanks, no worry. I just wanted to be sure I didn’t miss something for picking the right submission.
(What I understood given the timestamps is that the leaderboard seems to be incrementally updated, taking into account each submission ordered by its timestamp if not chosen explicitely using the form and that there is a delay of a couple of hours before appearing in the board)

1 Like
#5

Yes, just so that it is clear:

  1. The RMSE leaderboard is incrementally updated with your best submission
  2. The profit leaderboard is updated at 10PM CET every Saturday with your most recent submission unless specified otherwise by the form. This week exceptionally we’ve had quite a long delay but it will be updated very shortly :timer_clock:).
#6

Should we use still the same form to choose submission for week 2? I had chosen sumbission last week - if I don’t make a new choice do you still use the previous submission or the newest one?

1 Like
#7

Yes, please use the same form as before. We will be adding the functionality to the submissions page directly.

If you do not choose a submission it will be the most recent eligible submission you’ve made that will be placed in the leaderboard.