Update python3 to 3.7.7 donald #109
Conversation
…ariux64/pkg-scripts/pull/104/files" This reverts commit 10307f6. The commit has been added to the master branch by accident. Remove it.
Create bee file with
cp python-3.7.4-2.build.sh python-3.7.6-0.build.sh
and update version and build number.
If pip finds the system installation path not writable, it falls back to --user and installs into ~/.local. Of course, when this script runs, its system installation path should be writable. However, other errors in our environment, which prevent $PREFIX from being written to, might go undetected because of the fallback. Specify --prefix to abort if $PREFIX is not writable.
Add
python3 -c 'help("modules")'
at the end of build script. The idea is, that this command requires all
available modules to be loaded and would emmit error messages and
non-zero exist status in case of any errors in the installation.
We are going to install pycuda into this package, which requires cuda at compile- and runtime, so add the cuda package to this packages environment.
We are going to install Tensorflow into this package.
Tensorflow loads implementation libraries at runtime which need to be
found. Add LD_LIBRARY_PATH to avoid runtime errors like
Traceback (most recent call last):
File "/dev/shm/python-3.7.6-0/lib/python3.7/site-packages/tensorflow-2.1.0-py3.7-linux-x86_64.egg/tensorflow_core/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/dev/shm/python-3.7.6-0/lib/python3.7/site-packages/tensorflow-2.1.0-py3.7-linux-x86_64.egg/tensorflow_core/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/dev/shm/python-3.7.6-0/lib/python3.7/site-packages/tensorflow-2.1.0-py3.7-linux-x86_64.egg/tensorflow_core/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/dev/shm/python-3.7.6-0/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/dev/shm/python-3.7.6-0/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: libnccl.so.2: cannot open shared object file: No such file or directory
We are going to install Tensorflow into this package. The compilation of Tendowflow produces huge amounts of data (335612 files, 35 GB) in ~/.cache/bazel. Change BUILD_TMPDIR (which implies $HOME during the build) away from /dev/shm to avoid running out of memory.
The requireiments of hic2cool would downgrade a lot of packages, so build it from source.
The latest version of pyGenomeTracks on PyPi (3.3) has a dependency on hicmatrix>=12 which is not yet published. Request previous version.
Store the variants of BOOST_VERSION in an array instead of in a space seperated string to avoid parsing it over and over.
The distribuition server seems to be broken at the moment:
+ wget https://dl.bintray.com/boostorg/release/1.72.0/source/boost_1_72_0.tar.gz
--2020-03-28 13:02:47-- https://dl.bintray.com/boostorg/release/1.72.0/source/boost_1_72_0.tar.gz
Resolving dl.bintray.com... 52.28.187.224, 52.29.129.139
Connecting to dl.bintray.com|52.28.187.224|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-03-28 13:02:47 ERROR 403: Forbidden.
Use cached copy.
This reverts commit 5e93e04. The requirements of hic2cool have been updated and we can fetch it with pip again [1] [2] [1]: https://github.com/4dn-dcic/hic2cool/issues/46 [2]: https://github.com/4dn-dcic/hic2cool/releases/tag/0.8.2
|
|
So this means that the rebuild succeeded? Cool! (Mega-Cool!!!) |
Well as long as it isn't merged, we still have the option to replace it, don't we?
Mild irony(tm)? I was about to attempt Python 3.8.2 . Or 3.7.7 ? Would fix the build number question, too :-) |
|
Well, let's call it an 'interim' package (and merge it asap). The rationale is, that this build provides a functional environment for the openGL/CUDA stuff/issues that still sit in our back. (There are reasons why instagraal (cough...) isn't part of this package, albeit it was helpfull to try: ) |
Create bee file with
mv python-3.7.6.build.sh python-3.7.7.build.sh
and update version.
|
Rebuilt with python-3.7.7. Tested with tensorflow on CPU and GPU. |
|
LGTM :) |
|
Building into package/pkg/python-3.7.7-0 currently. When the build is done in a few hours and some testing has been done, this is ready to rumble. |
|
Tröööt! -- Well, I opt to keep the 'older' 3.7.6 build lying around (at least for some time). |
|
Warum? Benutzt den schon wer? |
|
Naja, es wurde in der build phase benutzt, und es existieren hier und da scripte die das profile sourcen. Zwar meist nur für tests, aber dennoch. Und auch wenn es unwahrscheinlich ist, wenn da sowas wie 'regression in 3.7.7' aufpoppt, dann gucken wir ein bischen doof aus der wäsche. |
|
Build finished. Ready to be tested. Ohhhhh..... :-( Ahhh. :-) Ahhhhhh :-) |
|
Okay, the systems have pulled this to local disk over night. I make it the default for python-3. Objections? |
|
Yes (Call it Minority-Report:), |
|
no objections from me
Am 4/2/2020 um 5:24 PM schrieb Donald Buczek:
…
Okay, the systems have pulled this to local disk over night. I make it
the default for python-3. Objections?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub Enterprise
<#109?email_source=notifications&email_token=AAAABEBCJ6KVRTZP3XAT5G3RKSU4HA5CNFSM2HUIVRRW63LNMVXHIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSVGG33NNVSW45C7NFSM2JB7#issuecomment-9279>,
or unsubscribe
<https://github.molgen.mpg.de/notifications/unsubscribe-auth/AAAABEEVXZJGFFS6QYMXYATRKSU4HANCNFSM2HUI>.
--
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus
|
Ich kann von zu Hause besser arbeiten als im Institut. Und da ich durchaus bereit bin, mich auch mit eventuellen Problemen zu beschäftigen, zählt das Argument hier nicht :-) |
|
-- no risk, no fun :) -- N.B. kannst du dir einen reim drauf machen, warum die keras samples je nach prozessor (gpu, cpu-arch1, cpu-arch2) unerschiedliche model-parameter liefern, und ob die so 'trainierten' modelle (wufff) dann doch die gleichen resultate liefern? |
|
Two features, I've read about, had in my mind, wanted to use and learned, that they are in python 3.8 only:
Time of 3.8, hhehe |
|
Wow, syntactic sugar - when will there be a real printf in (% (python)) (it took ages in java) :) |
Alternative to #108
Now that @david did all the main work in #107, #108 , I didn't want to ask him for less important cosmetic changes and instead help a bit with that myself. I've shuffled and amended Davids commits to clean the history and made some less important changes. David, okay with you?