Update python3 to 3.7.6 #108
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,14 @@ | ||
| #! /bin/bash | ||
|
|
||
| #COOKIE=$(mcookie); grep -v V_GREP_ME $0 > /dev/shm/runme-$COOKIE.sh ; sleep 1; exec bash /dev/shm/runme-$COOKIE.sh $1 | ||
|
|
||
|
|
||
| PKG=python | ||
| VERSION=3.7.6 | ||
| BUILD=0 | ||
| PYTHONVER=${VERSION%.*} | ||
|
|
||
| CUDADIR=/pkg/cuda-10.2.89-1 | ||
|
|
||
| PREFIX=/pkg/$PKG-$VERSION-$BUILD | ||
| if [ -n "$TESTING" ]; then PREFIX=/dev/shm/$PKG-$VERSION-$BUILD ; fi | ||
|
|
@@ -18,7 +18,7 @@ PATH=/bin:/usr/bin:/usr/sbin:/usr/local/bin | |
| set -xe | ||
| umask 022 | ||
|
|
||
| BUILD_TMPDIR=/scratch/local2/$PKG-$VERSION-$BUILD.build.tmp | ||
| test -d $BUILD_TMPDIR && ( chmod -R u+rwx $BUILD_TMPDIR || true ; rm -rf $BUILD_TMPDIR ) | ||
| mkdir -p $BUILD_TMPDIR/home | ||
| export TMPDIR=$BUILD_TMPDIR | ||
|
|
@@ -175,7 +175,7 @@ make install | |
| python3 -m ensurepip | ||
| pip3 install --prefix=$PREFIX -I pip | ||
|
|
||
| pip3 install --upgrade --prefix=$PREFIX ipython[all] | ||
|
There was a problem hiding this comment. Oh, I have add this for rebuilding to find the downgrader |
||
|
|
||
| pip3 install --prefix=$PREFIX SciPy NumPy Matplotlib pandas SymPy nose statsmodels | ||
| # clean up some dirt that statsmodels placed also into $PREFIX | ||
|
|
@@ -217,7 +217,6 @@ pip3 install --prefix=$PREFIX louvain # monocle3 prep for Helene | |
| pip3 install --prefix=$PREFIX selenium # 19.7.19 Kreitler -> https://webxray.org/ | ||
| pip3 install --prefix=$PREFIX cooltools # Maria Valieva 17.7.2019 | ||
| pip3 install --prefix=$PREFIX gmpy # From viper context (Helpdesk Laura Glaser 9.8.2019) | ||
| pip3 install --prefix=$PREFIX torch | ||
| pip3 install --prefix=$PREFIX torchvision | ||
| pip3 install --prefix=$PREFIX plotly | ||
|
|
@@ -231,18 +230,19 @@ pip3 install --prefix=$PREFIX threadpoolctl | |
|
|
||
| pip3 install --prefix=$PREFIX pyopengl | ||
| rm -rf ${PREFIX}/lib/python$PYTHONVER/site-packages/OpenGL/Tk | ||
| # creates error, so remove it | ||
|
|
||
| # pip3 install --prefix=$PREFIX HiCExplorer # Would be a nice to have, but it likes to downgrade too much, pick some cherries | ||
| pip3 install --prefix=$PREFIX HiCMatrix | ||
| pip3 install --prefix=$PREFIX pyGenomeTracks #downgrader cooler 0.8.7-> 0.8.5 | ||
|
There was a problem hiding this comment. Isn’t that also the version pinned in the hic2cooler requirements? There was a problem hiding this comment. by hic2cool is the problem numpy and scipy and not cooler |
||
|
|
||
| pip3 install --prefix=$PREFIX python-bioformats #downgrader matplotlib 3.2.0->3.1.1 docutils 0.16 -> 0.15.2 | ||
|
|
||
| pip3 install --prefix=$PREFIX jupyterlab # request Peter Arndt | ||
| pip3 install --prefix=$PREFIX jupyterlab-git | ||
| pip3 install --prefix=$PREFIX libxml2-python3 # required by several bee packages after swich to python3 | ||
| pip3 install --prefix=$PREFIX scour # https://github.molgen.mpg.de/mariux64/pkg-scripts/issues/98 | ||
| pip3 install --prefix=$PREFIX multiqc | ||
|
|
||
| #boost for pycuda | ||
| ( | ||
|
|
@@ -265,24 +265,6 @@ pip3 install --prefix=$PREFIX scour # https://github.molgen.mpg.de/mar | |
| --with-python=$PREFIX/bin/python3 \ | ||
| --with-libraries=python,thread || exit 1 | ||
|
|
||
| ./b2 \ | ||
| $NUMJOBS \ | ||
| --layout=system \ | ||
|
|
@@ -304,25 +286,135 @@ pip3 install --prefix=$PREFIX scour # https://github.molgen.mpg.de/mar | |
|
|
||
| #pycuda git version to get GL support | ||
| ( | ||
| PYCUDAVERS=2019.1.2 | ||
|
|
||
| . ${PREFIX}/profile | ||
| cd $BUILDDIR | ||
| test -e pycuda-${PYCUDAVERS}.tar.gz || wget https://github.com/inducer/pycuda/archive/v${PYCUDAVERS}/pycuda-${PYCUDAVERS}.tar.gz | ||
| test -d pycuda-${PYCUDAVERS} || tar -xf pycuda-${PYCUDAVERS}.tar.gz | ||
| #test -d pycuda || git clone --recursive http://git.tiker.net/trees/pycuda.git | ||
|
|
||
| cd pycuda-${PYCUDAVERS} | ||
| test -e siteconf.py && rm siteconf.py | ||
| python3 configure.py \ | ||
| --cuda-enable-gl \ | ||
|
There was a problem hiding this comment. Please use tabs to indent this. There was a problem hiding this comment. next time I promise :-) |
||
| --cuda-root=$CUDADIR \ | ||
| --no-use-shipped-boost \ | ||
| --boost-python-libname=boost_python37 \ | ||
| --boost-inc-dir=$PREFIX/include \ | ||
| --boost-lib-dir=$PREFIX/lib \ | ||
| --cudadrv-lib-dir='${CUDA_ROOT}/lib,${CUDA_ROOT}/lib/stubs' \ | ||
| --cudart-lib-dir='${CUDA_ROOT}/lib,${CUDA_ROOT}/lib/stubs' \ | ||
| --curand-lib-dir='${CUDA_ROOT}/lib,${CUDA_ROOT}/lib/stubs'\ | ||
| --ldflags=-L$PREFIX/lib | ||
|
|
||
| python3 setup.py install --prefix $PREFIX | ||
| ) | ||
|
|
||
| # hic2cool from source to stop downgrading python packages | ||
|
There was a problem hiding this comment. Which requirements are a problem? |
||
| ( | ||
| HICVERS=0.8.0 | ||
|
|
||
| . ${PREFIX}/profile | ||
| cd $BUILDDIR | ||
| test -e hic2cool-${HICVERS}.tar.gz || wget https://github.com/4dn-dcic/hic2cool/archive/${HICVERS}/hic2cool-${HICVERS}.tar.gz | ||
| test -d hic2cool-${HICVERS} || tar -xf hic2cool-${HICVERS}.tar.gz | ||
|
|
||
| cd hic2cool-${HICVERS} | ||
|
|
||
| if $( grep '==' requirements.txt >/dev/null ); then | ||
| patch -p1 <<'EOFHIC' | ||
| diff --git a/requirements.txt b/requirements.txt | ||
| index d7e701b..9d3a01c 100644 | ||
| --- a/requirements.txt | ||
| +++ b/requirements.txt | ||
| @@ -1,5 +1,5 @@ | ||
| -h5py==2.8.0 | ||
| -pandas==0.24.2 | ||
| -numpy==1.16.3 | ||
| -scipy==1.2.1 | ||
| -cooler==0.8.5 | ||
| +h5py | ||
| +pandas | ||
| +numpy | ||
| +scipy | ||
| +cooler | ||
| EOFHIC | ||
| fi | ||
| pip3 install -r requirements.txt --prefix ${PREFIX} | ||
| python3 setup.py install --prefix ${PREFIX} | ||
|
There was a problem hiding this comment. https://github.com/4dn-dcic/hic2cool/pull/44#issuecomment-605348278 claims, |
||
| ) | ||
|
|
||
| #tensorflow | ||
| ( | ||
| TFVERSION=2.1.0 | ||
|
There was a problem hiding this comment. Why do we need to install Tensorflow from source? Why is the package not good enough? There was a problem hiding this comment. f.e. the half of the cluster dont work with the pip package |
||
| TFSRC=https://github.com/tensorflow/tensorflow/archive/v${TFVERSION}/tensorflow-${TFVERSION}.tar.gz | ||
|
|
||
|
|
||
| . ${PREFIX}/profile | ||
| . /pkg/bazel-0.28.1-0/profile | ||
| cd $BUILDDIR | ||
| test -e tensorflow-${TFVERSION}.tar.gz || wget $TFSRC | ||
| test -d tensorflow-${TFVERSION} || tar -xf tensorflow-${TFVERSION}.tar.gz | ||
|
|
||
| cd tensorflow-${TFVERSION} | ||
| #vars for config | ||
| export PYTHON_BIN_PATH=${PREFIX}/bin/python3 | ||
| export USE_DEFAULT_PYTHON_LIB_PATH=1 | ||
| export TF_NEED_JEMALLOC=1 | ||
| export TF_NEED_KAFKA=0 | ||
| export TF_NEED_OPENCL_SYCL=0 | ||
| export TF_NEED_AWS=0 | ||
| export TF_NEED_GCP=0 | ||
| export TF_NEED_HDFS=0 | ||
| export TF_NEED_S3=0 | ||
| export TF_ENABLE_XLA=1 | ||
| export TF_NEED_GDR=0 | ||
| export TF_NEED_VERBS=0 | ||
| export TF_NEED_OPENCL=0 | ||
| export TF_NEED_MPI=0 | ||
| export TF_NEED_TENSORRT=1 | ||
| export TF_NEED_NGRAPH=0 | ||
| export TF_NEED_IGNITE=0 | ||
| export TF_NEED_ROCM=0 | ||
| export TF_SET_ANDROID_WORKSPACE=0 | ||
| export TF_DOWNLOAD_CLANG=0 | ||
| export TF_NCCL_VERSION=2.5 | ||
| export TF_IGNORE_MAX_BAZEL_VERSION=1 | ||
| export NCCL_INSTALL_PATH=${CUDADIR} | ||
| export GCC_HOST_COMPILER_PATH=/usr/bin/gcc | ||
| export TF_CUDA_CLANG=0 | ||
| export CLANG_CUDA_COMPILER_PATH=/usr/bin/clang | ||
| export TF_CUDA_PATHS=${CUDADIR},${CUDADIR}/extras/CUPTI,${CUDADIR}/nvvm | ||
| export TF_CUDA_VERSION=$(${CUDADIR}/bin/nvcc --version | sed -n 's/^.*release \(.*\),.*/\1/p') | ||
| export TF_CUDNN_VERSION=$(sed -n 's/^#define CUDNN_MAJOR\s*\(.*\).*/\1/p' ${CUDADIR}/include/cudnn.h) | ||
| export TF_CUDA_COMPUTE_CAPABILITIES=3.5,3.7,5.0,5.2,5.3,6.0,6.1,6.2,7.0,7.2,7.5 | ||
| export CC_OPT_FLAGS="-march=nehalem" | ||
| export TF_NEED_CUDA=1 | ||
|
|
||
| ./configure | ||
| bazel build \ | ||
| --cxxopt="-march=nehalem" \ | ||
| --copt="-march=nehalem" \ | ||
|
There was a problem hiding this comment. Do all our servers support this? There was a problem hiding this comment. No, but the important do it. This activate SSE4.1 There was a problem hiding this comment. if you want I can send you the full list |
||
| //tensorflow:libtensorflow.so \ | ||
| //tensorflow:libtensorflow_cc.so \ | ||
| //tensorflow:install_headers \ | ||
| //tensorflow/tools/pip_package:build_pip_package | ||
|
|
||
| test -d tensorflow-pip && rm -rf tensorflow-pip | ||
|
There was a problem hiding this comment. The check is not needed, as There was a problem hiding this comment. I have changed it d0c427b |
||
| mkdir tensorflow-pip | ||
| ./bazel-bin/tensorflow/tools/pip_package/build_pip_package --src ${BUILDDIR}/tensorflow-pip | ||
| cd ${BUILDDIR} | ||
| #remove symlinks into TMPDIR | ||
| test -e tensorflow-pip.tar && rm tensorflow-pip.tar | ||
| tar -chf tensorflow-pip.tar tensorflow-pip | ||
| test -d tensorflow-pip && rm -rf tensorflow-pip | ||
|
There was a problem hiding this comment. Ditto. The check is not needed. |
||
|
|
||
| tar -xf tensorflow-pip.tar | ||
| cd ${BUILDDIR}/tensorflow-pip | ||
| python3 setup.py build | ||
| python3 setup.py install --prefix $PREFIX | ||
|
|
||
| ) | ||
| # try to load all packages, maybe we can see installation errors | ||
| python3 -c 'help("modules")' > /dev/null | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is that not needed anymore? It’d be great if you mentioned that in the commit message (and also prefix it with
python-3.7.6:). Maybe even delete that line.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is a dev line
copy the script to /dev/shm and run there
so you can edit while building