Comments (9)
limits.h
should be in /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed
Try the following command:
ls -l /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/limits.h
from deepvariant.
@pgrosu I am actually reckoning some sort of include order issue related to clif, which I am not quite familiar with. To my understanding, it is quite normal when defining new types, one would like to extend the defs in limits.h (really bad...). In my case I think it is telling me at the end of my include path list there is a include_next which leads to no where, I am not sure if what I need is a modified derivative for headers.
/root/opt/clif/clang/lib/clang/5.0.0/include/limits.h
/root/clif_backend/llvm/tools/clang/lib/Headers/limits.h
/root/clif_backend/build_matcher/lib/clang/5.0.0/include/limits.h
/usr/include/c++/v1/support/ibm/limits.h
/usr/include/c++/4.8/tr1/limits.h
/usr/include/c++/5/tr1/limits.h
/usr/include/limits.h
/usr/include/linux/limits.h
/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/limits.h
/usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h
/usr/src/linux-headers-4.4.0-103/arch/arm/include/asm/limits.h
/usr/src/linux-headers-4.4.0-103/include/uapi/linux/limits.h
includes = [
include_htslib,
".",
"/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/",
]
I actually modified the chunk in deepvariant/third_party/htslib.BUILD, to specifically include a particular location which contains the type python needs and without the include_next loophole, but still doesnot work, maybe the syntax was wrong, the build_and_test actually issued a warning saying
WARNING: /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/external/htslib/BUILD.bazel:216:16: in includes attribute of cc_library rule @htslib//:htslib: ignoring invalid absolute path '/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed'
from deepvariant.
Having personally fought through all sorts of similar errors when we were preparing the OSS release, I know how painful this is. Before diving into this, maybe you can tell me what you are trying to do here. Are you saying that you can't run build_and_tesh.sh without modification, and you are trying to overcome some issue that's not itemized here? Or are you trying to do something else?
from deepvariant.
I do have one recommendation for you though; try building CLIF on your machine before running ./build-prereqs.sh. That script only uses our prebuilt CLIF binary if CLIF isn't already installed on your machine. It's possible that your system has local upgrades to ubuntu 16 that are confusing our pre-built binary (we are waiting for an official CLIF binary distribution, so in the meantime we are stuck with this less-than-ideal solution). The installation is pretty painless (https://github.com/google/clif#installation)
from deepvariant.
@depristo Thanks a lot for the clarification.
In my case, I am trying to have a standalone version to test with, without GCP at this stage.
In fact, I did manually pull clif from github (commit 6c6d894a112d978bd5abfcab1052c60c5ee365a9
) before building, bypassing the chunk note_build_stage Install CLIF binary
in build-prereq.sh
,
since I encountered the complaint of that OSS specific part with gsutil and clif, with this modified piece instead:
export DV_PLATFORM="ubuntu-16"
cd ..
git clone https://github.com/google/clif
cd clif
./INSTALL.sh
python setup.py install
sudo ldconfig # Reload shared libraries.
To my understanding, the manually built clif should be of the same structure as the setup script intended. However I didnot check if this pre-built binary is actually functionally similar to my manually built one. Any chance to put this oss-prebuilt in the git repo, at least for testing ?
OSS_CLIF_PKG="oss_clif.${DV_PLATFORM}.latest.tgz"
from deepvariant.
Does build_and_test.sh run properly without this modification? There's really nothing special about our pre-built CLIF binary. Here's our exact build commands:
# Install prereqs.
sudo -H apt-get -y install ninja-build subversion
sudo -H apt-get -y install virtualenv python-pip pkg-config
sudo -H pip install 'pyparsing>=2.2.0'
sudo -H pip install 'protobuf>=3.4'
echo === building protobufs
sudo -H apt-get install -y autoconf automake libtool curl make g++ unzip
wget https://github.com/google/protobuf/releases/download/v3.4.1/protobuf-cpp-3.4.1.tar.gz
tar xvzf protobuf-cpp-3.4.1.tar.gz
(cd protobuf-3.4.1 &&
./autogen.sh &&
./configure &&
make -j 32 &&
make -j 32 check &&
sudo make -j 32 install &&
sudo ldconfig)
echo === building CLIF
git clone https://github.com/google/clif.git
sed -i 's/\$HOME\/opt/\/usr\/local/g' clif/INSTALL.sh
sed -i 's/-j 2//g' clif/INSTALL.sh
(cd clif && sudo ./INSTALL.sh)
echo === creating package tgz
sudo find ${CLIF_DIR} -type d -exec chmod a+rx {} \;
sudo find ${CLIF_DIR} -type f -exec chmod a+r {} \;
tar czf "${CLIF_PACKAGE}" /usr/local/lib/libproto* "${CLIF_DIR}"
echo === SUCCESS: package is "${CLIF_PACKAGE}"
which is similar to your script but not identical. In fact you may be getting burned by sed -i 's/\$HOME\/opt/\/usr\/local/g' clif/INSTALL.sh
which changes the install dir from ~/opt to /usr/local/.
Is there a reason not to use this pre-built binary? If you are able to do git clone you should be able to reach GCS to get the binary. If you don't want to do that inside the script, you can always do it outside the script once, install it manually, and then build_and_test.sh won't try to refetch. Or you can follow the exact instructions above and it should create the actual clif binaries we distribute. We are looking forward to an official binary version of CLIF from that team...
from deepvariant.
@depristo In fact, without touching any setup sh scripts, that same build_and_test attempt failed with a missing header of this prepend:
--prepend clif/python/types.h
That is why I ended up modifying clif.bzl
in third_party
to include the absolute path of this include, not sure if this is the culprit.
I will try to move the clif installation to /usr/local to give it a shot, also with this new 0.4.1 release.
from deepvariant.
Yes, you can't do that with bazel - it doesn't allow you to do absolute path operations like that in general due to their approach to sandboxing / hermetic builds. I suspect moving CLIF to the expected location may fix it.
from deepvariant.
@depristo The porting to /usr/local works, thanks a lot.
With 0.4.0 build_and_test did complain about the htslib_gcp_oauth.init issue, which is now fixed in 0.4.1
(18:20:33) INFO: Elapsed time: 2.092s, Critical Path: 0.03s
(18:20:33) INFO: Build completed successfully, 2 total actions
Case closed, cheers.
oops, just to append a comment, by exporting an include variable with a non /usr/local
installation of clif
doesnot help, pyclif
still cant find it. This was tested before touching clif.bzl
. This clif
thing must have its own include logic.
from deepvariant.
Related Issues (20)
- Shuffle script not compatible with versions of tensorflow packages? HOT 4
- Fatal Python error: Segmentation fault HOT 12
- Singularity import issue and possible solution HOT 2
- How to merge Pacbio gvcfs? HOT 2
- Issue testing custom model HOT 10
- Error about expected cur_seq.size() < Max_READ_LEN HOT 2
- Missing --gpus flag in deeptrio HOT 1
- How to set threshold to filter vcf files HOT 2
- I am confusion about your deep learning architecture. HOT 3
- error in training DeepVariant HOT 13
- Restarting from post process variants step of deeptrio HOT 4
- BrokenPipeErro during postprocess_variants HOT 7
- Question about training HOT 1
- unable to run deepvariant using conda HOT 6
- Fatal Python error: Segmentation fault HOT 3
- How to get list of variants after make_examples step? HOT 1
- Highest mapping quality = 42 in bowtie2 HOT 3
- Output files are missing after running deepvariant. HOT 9
- Merging gvcf with GLnexus introduces non-zero heterozygous PL in hemizygous PAR HOT 1
- Dynamic cast failed HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepvariant.