Now that I have my final submission in and I’m just waiting for the results, my weekend is feeling quite empty. Sooo… I figured I’d start a discussion on some thoughts about the competition while we’re all still pretty engaged. Tbh, in 3 weeks, I will probably forget a lot of these so jotting them down somewhere is probably good .
Thoughts about technical aspects
- For sure I think this was a challenge to a lot of participants. This has been largely mitigated by how responsive and helpful the various admins have been. to the folks at AIcrowd and Imperial College London.
- I think a lot of the technical aspects have now been addressed/discussed, and the admins have a good idea of what should change to make the process easier, so I won’t repeat here (i.e. save model issues, different package versions allowed in the evaluation environment). My main pain point was with the preprocessing function. It doesn’t fit many of the major modeling frameworks to have the preprocessing not include the ydata. For example, @simon_coulombe’s xgboost start kit uses R’s tidymodels and he’s basically saving the preprocessing as an R object (rds) outside of the required functions and loading the object in the required preprocessing function. This separation of xdata and ydata also doesn’t work for steps like target encoding. By the way @simon_coulombe, I finally beat your xgboost model’s RMSE score with a not too complicated model… It really made my day .
Thoughts about data
- I was super impressed with the data anonymization. Motor insurance is one of the insurance segments that you can still do a lot with modeling anonymized data. This would not be true for say Homeowners, where specific address is pretty key to modeling expected loss.
- Predicting the Year 5 data is a cool idea. I wonder if some sort of customer churn idea could be introduced so there is no “data leakage”, since in the current setup, we know that all of the policies we are modeling have been with the insurance company for 5 years.
- Another gripe I have with the preprocessing function is that the way this is setup, when the preprocessing is run Year 5 data, none of my engineered features based on prior years’ history will work. So, my only option is save a copy of the ~60K policy histories with my submission models in order to use the prior year features. I can’t do anything about the 40K policies that were in the RMSE/weekly profit, even though I should have access to their Years 1-4 history to predict Year 5.
- It would be great if in addition to the claim_amount, we were also provided claim_count in the training data. Generally, frequency/severity models in the industry would look at counts of unique claim occurrences, not counts at the policy-year level. For example, it’d be helpful to see how many different claims added up to the claim_amount that we have for each policy-year.
- The reinsurance aspect is interesting, but reinsurance is not typically at the individual policy level for motor insurance (at least not in the US, where I’m based) for personal auto. I understand the reason the capping was done, but no need to call it “reinsurance” . When the capping at 50K was introduced, it was not obvious to me that the training data was also changed. I have been running my models in Google Colab and it took me a full week to figure out that my models were changing because the training data I was downloading was changed to cap losses at 50K. Depending on the model, introducing this cap helps some and hurts others. I’d probably rather have the uncapped losses and let the participants decide if they want to cap to train their models, knowing that the evaluations will cap the losses. In my code, I actually have a cell that “uncaps” the losses back to the original training data that was available for download before the “reinsurance” introduction.
Thoughts about the weekly feedback
- I think this is probably the area that has most room for improvement. The feedback is quite different from what a real insurance company would get from the market. The feedback has improved over the weeks, but this improvement was at the expense of consistency. Since this is a market based game, the consistency from the perspective of what do I think other competitors are reacting to is an important element of the game. Introducing new elements in the feedback makes this really challenging to think through.
All in all, I thought this was a great competition and I’m really glad I participated. I learned so much on modeling and how to consider pricing strategies. I’m very grateful for all the work that the admins put in, and props for coming up with such an intriguing competition!