LightGBM error in subbmission

[matt_simmerson]

Hi

I am trying LightGBM in my solution, which works fine in the Google colab environment. But when I submit the entry, I get the following error:

OSError: libgomp.so.1: cannot open shared object file: No such file or directory

Any help/ideas would be much appreciated. Submission number is # 120699.

Cheers

Matt

[michael_bordeleau]

Interesting timing.
Are you in R or python?
(I’m with R, zip file approach)…

I’m actually playing around with LightGBM for the last few days, only to realize (now) that it doesn’t seem like I can save the model into a RData format.

From what I understand so far, I can load it back, but then if I try to use it for predictions, R crashes.

I’m tired. Will look further into this tomorrow…

[alfarzan]

Hi @matt_simmerson

Yes this is a little unintuitive at the start. I can see you’re using Python on Colab.

Could you follow the advice on this thread?

That should solve it

[alfarzan]

@michael_bordeleau Is this locally? or on the AIcrowd side? If it works locally but crashes on our side let me know and I’ll see what can be done

[MakePredict]

you need to save it with lgb.save and load it with lgb.load

[michael_bordeleau]

Hi,

@alfarzan yep, this is crashing local, so no need to worry… yet

As @MakePredict mentioned, we need to use the special functions.

However, on my end, I decided to try saveRDS.lgb.Booster …

The challenge is combining this with other models… Juggling between RData and RDS… Major overhauling of my code. Today has been 100% debugging… (yikes)

So yeah, anyone reading this using R, tread carefully around lightgbm if you decide on using it. I’ve had other several issues with it. As an example, for some reason, I need FIRST to run a small lightgbm tutorial first before attempting modeling or it will crash …every… …single… time… urrh.

Good luck!

[RHG]

Hi everybody,

I’m running into problems with lightgbm in R too (I’m using google colab notebook for my submissions).

As @michael_bordeleau said, lightgbm is difficult to handle. My problem, as said above, is in the saving /loading step.

I tested lgb.save/lgb.load and saveRDS.lgb.Booster/readRDS.lgb.Booster. With both alternatives my code runs ok in google colab but my submission fails.

I’m saving into save_model function each model into a different .rds file and loading them in the same way. In the review of my submission error I see a error message in the loading step: “Error in lgb.load(file = “trained_model_Freq.rds”) :
lgb.load: file ‘trained_model_Freq.rds’ passed to filename does not exist”.

So, It is mandatory save all models in a model object and save it into trained_model.Rdata? How is possible do that with specific functions of lightgbm?

My submissions ID is 120892.

Thanks in advance.

[alfarzan]

Hi @RHG

I think the fix in your case should be quite simple

So for R notebooks what you can do to use anything other than a .RData file is to just make sure you save the object inside the saved_objects/ directory.

So if your two models are: trained_model_Freq.rds and trained_model_Sev.rds then in your load_model and save_model you should have something like:

save_model <- function(freq_lgb, sev_lgb){
  lgb.save(freq_lgb, 'saved_objects/trained_model_Freq.rds')
  lgb.save(freq_lgb, 'saved_objects/trained_model_Sev.rds')
}

load_model <- function(){
  freq_lgb = lgb.load(file = 'saved_objects/trained_model_Freq.rds')
  sev_lgb = lgb.load(file = 'saved_objects/trained_model_Sev.rds')
  model = c('frequency' = freq_lgb, 'severity' = sev_lgb)
  return(model)
}

That’s basically the gist of it. We then take everything inside saved_objects/ and push it to the server. This directory is created when you setup the notebook and run the AIcrowd cell at the beginning.

Hope that makes it work, if not then we can continue the discussion

[RHG]

Hi @alfarzan

Thank you so much!

I hadn’t used the saved_objects directory before, that was the problem. It works perfectly for me now.

Regards!