Make submissions via google colab copying a whole folder of models rather than a single object

[piercer]

Hi @alfarzan,

I am trying to make a submission via a python colab notebook and in this particular setting I save my set of models in a folder and then load these from the same folder when necessary. However, in the last step of the procedure when I execute %aircrowd_submit the code fails and by having a quick look at the traceback it seems that the problem arise from the function copy_model_file(cfg) in aircrowd_helpers.py which if I am not mistaken has not been designed to handle folders containing models whilst instead seems to work fine with a single model object. So my question is, could you please let me know how can I overcome this issue, so that I can easily lunch %aircrowd_submit and make sure that my model folder is then copied into the submission folder without any error?

Thanks in advance.

[alfarzan]

Hi @piercer

Yes, one workaround would be:

1. Inside your save_model have the function also create a directory and place your objects inside that directory.
2. Inside your save_model then have the function compress that directory into a zip file. One solution to do this would be this.
3. Inside your load_model then you can unzip and take it from there.

In the meanwhile I will look into allowing you to save multiple things in a folder in a bit of a more straightforward way as well

[piercer]

Hi @alfarzan,

Thanks a lot for the advice! That was exactly my backup plan, but before to implement I just wanted to be sure that no other options were available at the moment. But thanks a lot for your support and let me know if you will manage to find an alternative solution.

[alfarzan]

Will do!

The other easy option is to do a zip submission and submit the folder entirely.

But I imagine you wish to stuck to colab

[piercer]

Awesome! Yeah I can also do that but at the time being I simply amended the colab including the zip save option and it works, now submitting is no longer an issue, however I am encountering a new issue when generating predictions and by having a look at the traceback seems now that h2o is rising some errors. It seems that once loading the models, there is no initialisation of the h2o clusters which might be caused by Java:

h2o.exceptions.H2OConnectionError: Could not establish link to the H2O cloud http://localhost:54321 after 5 retries
h2o.exceptions.H2OStartupError: Cannot find Java. Please install the latest JRE from
http://www.oracle.com/technetwork/java/javase/downloads/index.html

@alfarzan Did you hear if anyone else was facing the same issue?

[alfarzan]

Yes, though in my experience the h2o errors are generally a little bit misleading. Can you provide your submission ID and I can look into this further?

[piercer]

I know dealing with h2o it is always tricky…yes sure! Thanks a lot for the support!
Please find it here: #118276

[alfarzan]

Hi @piercer

We’ve looked a bit more into seeing how to allow you to do this and indeed the most straightforward way to do it would be through a zip submission. You will have to follow the following steps:

1. Create an apt.txt file. Inside it in one line include the text openjdk-11-jdk. You can include any other apt packages as well.
2. Include a file with the include below, with the name: Dockerfile

That should do it!

Dockerfile text

FROM python:3.8.3-slim

​

# Create user home directory

ENV USER_NAME aicrowd

ENV HOME_DIR /home/${USER_NAME}

​

# Replace HOST_UID/HOST_GUID with your user / group id

ENV HOST_UID 1001

ENV HOST_GID 1001

​

# Use bash as default shell, rather than sh

ENV SHELL /bin/bash

​

# Set up user

RUN adduser --disabled-password \

    --gecos "Default user" \

    --uid ${HOST_UID} \

    ${USER_NAME}

​

# Install base dependencies

RUN apt -qq update && apt -qq install -y --no-install-recommends \

 man \

 build-essential \

 && mkdir -p /usr/share/man/man1

RUN mkdir -p /usr/share/man/man1

​

# Install apt packages from apt.txt

COPY apt.txt apt.txt

RUN xargs -a apt.txt apt -qq install -y --no-install-recommends

​

# Install pypi packages

COPY --chown=1001:1001 requirements.txt requirements.txt

RUN pip install -r requirements.txt --no-cache-dir

​

USER ${USER}

WORKDIR ${HOME_DIR}

COPY --chown=1001:1001 . ${HOME_DIR}

[piercer]

Hi @alfarzan,

hope you are doing fine and enjoying the weekend.
I am reaching out cause finally, thanks to your support and by following the indications above, I got the Java issue sorted and everything seems working fine now. Thanks again!

[alfarzan]

Perfect! Glad to hear