How to use conda-forge or CRAN for packages in evaluation?

bjoern.holzhauer · November 23, 2019, 9:47pm

I’ve spent a huge amount of time trying to make the recommended solution submission approach using conda & environment.yml work. Naturally, we developed our solution using the latest tidyverse from CRAN, as anyone would.

I keep running into the problem that the main conda channel is outdated and that there’s breaking changes between versions (e.g. in the unnest function). While conda-forge would have the up-to-date packages, it seems that anything I’ve tried like adding conda-forge in the channels at the top of environment.yml and/or adding “conda-forge::” before the relevant packages (and various permutations like specifying individual packages with version numbers etc.) fails.

There seems to be zero documention on how to do this - or even better simply how to use CRAN (just using intall.packages in the code appears to fail due to some compiler issues that are presumably to do with how the evaluation server is configured). Surely, this question has come up before?

bjoern.holzhauer · November 25, 2019, 2:09pm

Here’s the requested environment.yml (I just added two packages - tidyverse and glmnet plus the conda-forge channel compared to the starter kit environment.yml). The below works fine, if I change tidyverse=1.3.0 to tidyverse=1.2.1 and remove conda-forge from the channels. When I try to do “library(tidyverse)” an image is build “successfully” (not really) and I get “2019-11-25T14:00:07.235091205Z Error: package or namespace load failed for ‘tidyverse’ in dyn.load(file, DLLpath = DLLpath, …): 2019-11-25T14:00:07.235120506Z unable to load shared object ‘/srv/conda/envs/notebook/lib/R/library/stringi/libs/stringi.so’:
2019-11-25T14:00:07.235125507Z libicui18n.so.64: cannot open shared object file: No such file or directory 2019-11-25T14:00:07.235128907Z Execution halted”.

name: novartis-dsai
channels:
  - conda-forge
  - r
  - defaults
dependencies:
  - _r-mutex=1.0.0
  - bwidget=1.9.11
  - bzip2=1.0.8
  - ca-certificates=2019.10.16
  - cairo=1.14.12
  - certifi=2019.9.11
  - curl=7.65.3
  - fontconfig=2.13.0
  - freetype=2.9.1
  - fribidi=1.0.5
  - gettext=0.19.8.1
  - glib=2.56.2
  - graphite2=1.3.13
  - gsl=2.4
  - harfbuzz=1.8.8
  - icu=58.2
  - jpeg=9b
  - krb5=1.16.1
  - libcurl=7.65.3
  - libedit=3.1.20181209
  - libffi=3.2.1
  - libiconv=1.15
  - libpng=1.6.37
  - libssh2=1.8.2
  - libtiff=4.0.10
  - libxml2=2.9.9
  - make=4.2.1
  - ncurses=6.1
  - openssl=1.1.1d
  - pango=1.42.4
  - pcre=8.43
  - pip=19.3.1
  - pixman=0.38.0
  - python=3.6.9
  - r=3.6.0
  - r-base=3.6.1
  - r-boot=1.3_20
  - r-class=7.3_15
  - r-cluster=2.0.8
  - r-codetools=0.2_16
  - r-foreign=0.8_71
  - r-kernsmooth=2.23_15
  - r-lattice=0.20_38
  - r-mass=7.3_51.3
  - r-matrix=1.2_17
  - r-mgcv=1.8_28
  - r-nlme=3.1_139
  - r-nnet=7.3_12
  - r-recommended=3.6.0
  - r-rpart=4.1_15
  - r-spatial=7.3_11
  - r-survival=2.44_1.1
  - r-glmnet 2.0_16
  - r-tidyverse=1.3.0
  - readline=7.0
  - setuptools=41.6.0
  - sqlite=3.30.1
  - tk=8.6.8
  - tktable=2.10
  - wheel=0.33.6
  - xz=5.2.4
  - zlib=1.2.11
  - zstd=1.3.7
  - pip:
    - numpy==1.17.3
    - pandas==0.25.2
    - python-dateutil==2.8.0
    - pytz==2019.3
    - six==1.12.0

bjoern.holzhauer · November 25, 2019, 6:50pm

Creating an apt.txt (and adding it to the commit) that contains
sudo apt-get install libicu-dev
(as per https://stackoverflow.com/questions/13887432/how-to-get-libicui18n-so-36) fails to build the image, similarly, adding pyicu=2.4.2 to the environment.yml (the other suggestion from the stackoverflow thread) results in a version conflict between python packages.

bjoern.holzhauer · November 26, 2019, 5:53pm

No details on what was discussed in the call seen in the FAQ. I’d really rather not start to learn about dockerfiles and how to configure one for this particular platform, surely there is a simple solution for something that most R users would want to do?

I’m sorry for posting again, but it seems like I am still spending 80% of the time I can spare for this competition on figuring out by trial-and-error how to install software/work around the particular undocumented limitations on this platform (and yes, I actually have the same Ubuntu version at home). This is pretty frustrating and not really sustainable.

kelleni2 · November 27, 2019, 9:07am

hi @shivam - could you see if you could help Bjoern? And adding the learnings to the FAQ or forums would be great.

@bjoern.holzhauer - you need details on the docker etc? I can connect the two of you via email if it helps.

bjoern.holzhauer · November 27, 2019, 10:33am

@ kelleni2 I’d love to simply avoid having to do a docker - all I’m trying to do is using the tidyverse (but not in the outdated version on the main conda channel - but e.g. instead the version from conda-forge or even better the one from CRAN*). I have the hope that this is not such a crazy thing to want…

In the last call, I think Mohanty mentioned an option using an apt.txt (?) file, but I could not get that to work, and I don’t think the recording of the call is available, yet (so I could not double check what he showed). If it has to be a docker, then any additional explanations/help that could be available would be great.

* Mohanty seemed to say they do not enable that, if you have a mixed R+Python pipeline (which happens all the time, if you primarily work from R, because many R packages use Python in the background)

mohanty · November 27, 2019, 2:15pm

@bjoern.holzhauer: We are on it. We will send you a working version with tidyverse soon. Thanks for your patience.

shivam · November 28, 2019, 3:55am

Hi @bjoern.holzhauer,

The issue you are facing was multi-fold due to which it took some time on our side as well to figure best solution for you.

The procedure you were following was correct except apt.txt file. This file excepts just the packages name you want to install (although it isn’t the fix/cause of the error). So it should be something like:

~❯ cat apt.txt
libicu-dev

But the error still continues as:

> library('tidyverse')
Error: package or namespace load failed for ‘tidyverse’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/srv/conda/envs/notebook/lib/R/library/stringi/libs/stringi.so':
  libicui18n.so.64: cannot open shared object file: No such file or directory

We found the issue is due to the dependency r-tidyverse -> r-stringr -> r-stringi, and r-stringi package in conda-forge channel is broken.

We checked with the recommended channel i.e. r and it had working packages by default, so this should have worked at first place.

    - r::r-tidyverse
    - r::r-stringi

[Solution; tldr] But I remember from our call, you needed 1.3.0 version specifically. So, this is the environment.yml entry you need for 1.3.0. Basically getting r-stringr from r channel instead of conda-forge one.

  - conda-forge::r-tidyverse==1.3.0
  - r::r-stringr
  - r::r-stringi

Sorry that you went through long debug cycle.

[Update] I found issue on Github in conda-forge repository as well, and have added the comment about this issue there. https://github.com/conda-forge/r-stringi-feedstock/issues/20

bjoern.holzhauer · November 28, 2019, 3:36pm

Thank you, great news! This seems to have fully solved this problem, I’m still testing more of what we had in mind, but at least in debug mode it seems to work now.

I guess this illustrates why for R CRAN may be a better choice (given their extensive testing suite for packages).

wangbot · December 19, 2019, 2:48pm

@bjoern.holzhauer Hi Bjoern were you successful in using glmnet during evaluation by adding - r-glmnet 2.0_16 to the environment.yml file? Thx.

bjoern.holzhauer · December 19, 2019, 5:10pm

Yes, adding r-glmnet=2.0_16 (and conda-forge in the channels) works for me.