Java and Docker don't like each other

NeurIPS 2019 : MineRL Competition
#1

Hi,

Has anybody tried to create a docker with the environment?

My problem comes when I install minerl, I get the following error:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000076a82d000, 454598656, 0) failed; error='Cannot allocate memory' (errno=12)
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 454598656 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /tmp/pip-install-bpk6xbfz/minerl/minerl/env/Malmo/Minecraft/hs_err_pid44.log
    :decompileMc FAILED

Apparently the problem comes as a result that docker and java are not good friends, as you can read here: https://developers.redhat.com/blog/2017/03/14/java-inside-docker/

I’ve tried to implement the proposed solution without success. Any idea?

The Dockerfile I am using follows…

Any help would be appreciated! Thanks.

FROM nvidia/cuda:10.0-base-ubuntu18.04

# Install some basic utilities
RUN apt-get update && apt-get install -y \
    curl \
    ca-certificates \
    sudo \
    git \
    bzip2 \
    libx11-6 \
    tmux \
    htop \
 && rm -rf /var/lib/apt/lists/*

# Create a working directory
RUN mkdir /code
WORKDIR /code

# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /code
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user

# All users can use /home/user as their home directory
ENV HOME=/home/user
RUN chmod 777 /home/user

# Install Miniconda
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh \
 && chmod +x ~/miniconda.sh \
 && ~/miniconda.sh -b -p ~/miniconda \
 && rm ~/miniconda.sh
ENV PATH=/home/user/miniconda/bin:$PATH
ENV CONDA_AUTO_UPDATE_CONDA=false

# Create a Python 3.7 environment
RUN /home/user/miniconda/bin/conda install conda-build \
 && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 \
 && /home/user/miniconda/bin/conda clean -ya
ENV CONDA_DEFAULT_ENV=py37
ENV CONDA_PREFIX=/home/user/miniconda/envs/$CONDA_DEFAULT_ENV
ENV PATH=$CONDA_PREFIX/bin:$PATH

# CUDA 10.0-specific steps
RUN conda install -y -c pytorch \
    cuda100=1.0 \
    magma-cuda100=2.4.0 \
    "pytorch=1.1.0=py3.7_cuda10.0.130_cudnn7.5.1_0" \
    torchvision=0.3.0 \
 && conda clean -ya

# Install jupyter notebook

RUN pip install jupyter

# Install Minecraft needed libraries
RUN sudo apt-get update
RUN sudo apt-get install openjdk-8-jdk -y
CMD java -XX:+PrintFlagsFinal -Xmx900m -jar java-container.jar
RUN pip install --upgrade minerl

# Set the default command to bash
CMD ["/bin/bash"]
#2

What operating system are you running docker on? It looks like you are out of memory! (I suspect that if you’re on Windows or Mac you’ll need to increase the allocated memory to the docker VM).

#3

Thanks William, that actually worked. I increased the memory, and now don’t have anymore that error. But I have another one, and will open a new thread…

#4

Out of curiosity what was this other error?

#5

Increasing the heap size is a bad solution, 100% temporary, because you will hit the same issue if you get several parallel requests or when you try to process a bigger file. Normally this is a runtime error in Java which occurs when you allocate a new object in your application over a period of time continuously and the Garbage Collector (GC) cannot make space available to accommodate a new object, and the heap cannot be expanded further, which resulted this error.

Therefore you pretty much have the following options:

  • Find the root cause of memory leaks with help of profiling tools like MAT, Visual VM , jconsole etc. Once you find the root cause, You can fix this memory leaks.
  • Optimize your code so that it needs less memory, using less big data structures and getting rid of objects that are not any more used at some point in your program.

To avoid OutOfMemoryError, write high performance code:

  • Use local variables wherever possible.
  • Release those objects which you think shall not be needed further.
  • Avoid creation of objects in your loop each time.