The first profit leaderboard is out! Some comments

Insurance pricing game
#1

Hi all

(Note that I will edit the third section of this post with more details mid-next week)

You’ll notice the first profit leaderboard is out :rocket:

Before I dive into some details, remember that in real markets, there is a lot of experience and regulation which means things reach a relative equilibrium over time. Here we are attempting to short-circuit that process with the weekly leaderboards and our market rules. But this means that the start may be bumpy until each market player iterates on and crystalises their strategy.

1 Where is my model? :exploding_head:

You may see that your model is not present here despite having passed the non-negative training profit test.

This is because your model will have failed the participation rule which states:

Participation rule. Your model must participate (i.e. win at least 1 policy) in 5% or more of the markets it is placed in

In other words, in more than 95% of the many markets that you were placed in, you won 0 contracts.

2 Why all the negative profits? :chart_with_downwards_trend:

This happens when the prices offered are too low or the risk model is not very accurate, or both. Overtime, we expect the market (i.e. participants) to correct for this effect so that more and more of your models “make money” in competition.

So how can you correct things without feedback? Read on :point_down: :point_down:

3 Can I get more feedback? :information_source:

Over the next week we will be working to create a personalised Market Performance Summary for the submissions that you enter into the profit leaderboads each week. Watch this space for more details.

In the meanwhile, note that the top 2 models in the profit leaderboard, are within the top 5 of the RMSE leaderboard, so play on! :bulb:

4 Likes
Can you clarify what the markets and the participation rates are?
#2

Perhaps can show models that fail the Participation Rule at the end of the leaderboard, with a negative figure of say -9999999, to assure everyone that their models were not inadvertently omitted for evaluation?

#3

Thanks @zhengyh, yes we will be including something similar to this in the coming days and the leaderboard will be updated to include those models.

2 Likes
#4

Hi,

it would be interesting to integrate a profit indicator as the % of the premium received on the leaderboard.

Yours sincerely

Alain

#5

Hi @AlainBugnon

Do you mean for example, the fraction of the written premium from the market? some sort of revenue market share?

#6

I mean the profit in % for the the insurance company

Exemple
earned premium: 10’000
claims: 13’000
profit: -3’000
profit percent: -30%

#7

Ah yes, we will be including overview feedback in the spirit of what you mention.

Once confirmed I’ll make an announcement about it. You should also receive this feedback by Sunday for both week 1 and week 2 markets.

Stay tuned :headphones:

#8

Great :+1: :aicrowd:

#9

The KPIs mentioned above would be interesting but could you share individual data used for results calculation? It might be useful to know who bought my policies and who didn’t buy it.

#10

Hi @pitusg

Ideally we would like to give you the full portfolios that you end up winning! But for two reasons we can’t do this:

  1. There are many thousands of markets that you play in during each round
  2. We don’t want to incentivise maximising marketshare to get more data

However, I believe we may have a happy compromise in the works that will be ready for you, if not for this weeks’ leaderboard, then certainly the one after.

Keep an eye out for announcements relating to this soon :mega:

#11

Hi @alfarzan
Yes having the list of the profiles and, for each profile, the “conversion rate” (% of the markets in which we were the cheapest) and (optionally) the observed claims would have a huge impact on the game !

1 Like
#12

Hi @alfarzan

throwing some ideas around:
re: market share maximisation, if I understand correctly, the profit leaderboard is (sadly) NOT cumulative, so folks would have an interest in charging 0$ for the “weekly leaderboard”. the solution is pretty simple: just show the whole weekly dataset to everyone.

Let’s say we had to score N profiles, it would be great to provide a csv with the characteristics of each of the N (+ claim_amount or not, not important) + the price P offered by each competitors. If not all prices, at least the lowest, second lowest and quartiles would be interesting. Hopefully folks dont use a random price to charge a different price in each market… :slight_smile:

If you can’t show us the profiles for the weekly leaderboard, maybe just show us all the price we charged and for each price what was the winning bid. If we were the winning bid, then show the 2nd place.

1 Like
#13

Hi @guillaume_bs and @simon_coulombe

Both of your suggestions are actually already in the pipeline and will likely show up either from this week onward or from the next!

@simon_coulombe yes, other than research related concerns, the decision to not have the profit leaderboard be cumulative was to allow people to join the competition in the middle and not feel left out over the previous few leaderboards.

I don’t think we will be able to provide the winning bid if you do not win a policy, as that is information that is hard to get in the real world, outside of bespoke deals with brokers and price comparison websites. Also keep in mind that you will have participated in thousands of markets!

But the rest of the suggestions (e.g. offered premium, whether it was won, dataset level statistics) are in the pipeline :slight_smile:

I hope to make an announcement late this week or early next week regarding this.

2 Likes
#14

this all sounds good, thanks :slight_smile: