The markets have spoken once more !
I have 4 important updates for you.
- Tweaked market metric to boost participation (check the leaderboards!)
- Weekly leaderboard history
- Market feedback for next week
- RMSE leaderboard is now fixed
1 Tweaked market metric
After the first leaderboard we noticed that many models were not passing the participation requirement (i.e. they were having a market share of 0 in more than 95% of our 1000 or so markets).
Upon investigation we tweaked how the markets were being constructed which now allows many more models to participate. All is now updated including the leaderboard. You will notice some changes on week one due to this but I am happy to see our top players are still in the top 10! Most importantly many others have entered the fray
2 Weekly leaderboard history
You will see that we now have a separate leaderboard for each week so you can see how the market rankings have changed over time, week-by-week. For example you can see that 4 of our participants are within the top 11 on either week!.
3 Market feedback
From this week onward you will be receiving the following feedback in your email after each leaderboard:
- Market-share Vs profit. This will be a plot that shows how your model performed in each market it was placed during the leaderboard.
Market summary statistics. These will be the following summary statistics:
- Average revenue
- Average loss
- Average profit-per-policy
- Average loss-ratio
- Average market-share
- Total number of markets that you participated in
We will send out this information for both leaderboards for those that had valid models.These plots are currently being generated and will be sent out shortly.
We are still evaluating the best way to give you dataset related feedback as well if possible. So stay tuned on that next week
4 RMSE leaderboard is now fixed
Lastly there was an issue with the RMSE leaderboard not getting updated in real time due to a display issue. This is now fixed, and we apologies for the brief malfunction.