(low priority) Why does evaluating take so long?

Insurance pricing game
#1

Running “Rscript predict.R training_data.csv output_claims.csv output_prices.csv” locally requires maybe 5 seconds, while evaluating on the web site takes easily 10 minutes.

I guess it’s spinning up docker instances or something, yet that is surprising. Is it reinstalling all the “tidyverse” everytime ?

#2

Hi @simon_coulombe

There are two reasons for this:

  1. Right now we are generating the weekly leaderboard submissions while also, due to the recent tweak to the RMSE leaderboard, reevaluating all submissions from the previous week. So things might be a little slower than usual. Apologies for that.
  2. Regarding the installation, at the moment, yes. However, over time we keep a record of commonly installed packages and we pre-install them to speed things up. For package requests please post it on the other thread for R packages.
1 Like
#3

No worries for delays (or even exceptional extended delay).

It a light annoyance, I was mostly wondering because that must be driving up processing costs by a factor of 100… :slight_smile:

1 Like
#4

Yes hopefully these annoyances will be over with very soon. Week 1 is always a little bit of a learning curve with these competitions :slight_smile:

No matter how much you test, there will always be novel issues, but I’m confident the submission process will be much easier from next week onward.

1 Like