Just when you thought there would be no more posts about bugs….
I wasn’t expecting to be placed in the top 4, but it was certainly a bit of a surprise (as well as a bit annoying after all the effort) to come last. What happened? Did I get overlooked in the bug checking process? When I extract year 4 data from the training data and make pseudo year 5 data by changing the year from 4 to 5 it seems to produce sensible results for me when I run it in Colab. Did passing through all 5 years of data and throwing away year 1-4 mess up my result?
The final feedback is certainly interesting, as well as my impressive loss of 10M.
I haven’t had any communication from AI crowd about it, so I don’t know what they think. Will it get reviewed? I feel it’s unfair that other people were contacted and given the opportunity to fix issues caused by year 5, but I wasn’t. I am very disappointed.
It definitely seems like something isn’t right with your heat map and profit score… I was wondering what was decided with the years 1-4 (if the training set ended up being passed through with the final evaluation). And, if adjustments were made only if the submitted code errored out. There probably were quite a few people with bugs/errors in their code but their code still ran (like SAS ). @Calico, sorry about your situation. If it means anything, we know your final leaderboard score doesn’t reflect your ability / no-bug model results.
Just as another reference point, here’s the feedback on my submission. Maybe there were 14 large claims >10K, since 1/14 = 7.14%, and my team usually got 2-3 of them? I still don’t think the density plot says too much about number of large losses since the x-axis is profit, not loss, and the average market is pretty big based on the average revenue/losses metrics.
Just to update the thread, unfortunately there was a “production bug” in the submitted code due to (year= 5) that resulted in the code running, yet producing unexpected prices (i.e prices not intended by @Calico) that then resulted in both the profit number and the feedback.
We’ve had a productive conversation about how to remedy this issue and as a result, we’ve re-run a version of the markets in private so that @Calico can see his performance had that bug not existed
Although this was the only model with such an issue, for the future hopefully we’ll have a process that’s immune to this kind of error in place
Thanks @alfarzan for re-running the market and for all your hard work on this. Its appreciated. I would have ranked 33 in the end. It was a bit annoying that an obscure bug triggered by all the data being year 5 ended up resulting in last place, and ironic that something that isn’t even a rating factor could mess things up. Anyway, it’s great to know finally how I would have done if everything had worked as intended. Thanks again.
). 
