Installation-from-source issues on OSX about vg

[ekg]

Very interesting. I don't think anyone has compiled it in a Mac yet. I have
some to test on so let me see if I can reproduce these problems.

It looks like you'll need the openmp header libraries. If you have them
let's make sure that you have openmp directives enabled in gcc.

htobe64 is defined in endian.h. Not sure what it should be on OSX. This one
could be resolved by porting just the few functions I'm using for reversing
byte order.
On Mar 10, 2015 8:20 PM, "Ryan Williams" [email protected] wrote:

Hi there, I've hit a few snags trying to install from source on OSX
10.10.2. All of my cmds / output are in this gist
https://gist.github.com/ryan-williams/715a1ac797633617aa3f but I'll
link to specific files / sections within it below:

• git clone --recursive, cd vg, make:
  • option -s is obsolete and being ignored warning and error
    https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-clone-and-make-L199-L200
  • this is freebayes#83 freebayes/freebayes#83
    • fix by commenting out the ,-s in vcfutils/smithwaterman/Makefile
      https://github.com/ekg/smithwaterman/blob/203218b47d45ac56ef234716f1bd4c741b289be1/Makefile#L18
• make again
  • Error: ar: smithwaterman/sw.o: No such file or directory
    https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-make-need-to-clean-L19
  • turns out the previously interrupted make has left some
    inconsistent state
    • make clean
  • Mostly works, but fails in snappy module and exits with an error
    code
    https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-make-clean-L22-L23
    • make again
  • Undefined symbols for architecture x86_64:
    https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-make-pb2json-L70-L85
    related to google::protobuf things in pb2json

That's as far as I can get right now.

I've installed "real" gcc/g++ via brew and they are the only ones on my
path
https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-clone-and-make-L140-L151.
I did this due to earlier errors like:

In file included from vg.cpp:1:
./vg.hpp:9:10: fatal error: 'omp.h' file not found
#include <omp.h>
^
1 error generated.

Some googling led me to believe that this was the result of using clang,
cf. this SO answer http://stackoverflow.com/a/25990538/544236.

When running in to the pb2json linker errors above, I thought they might
be due to my having run brew install protobuf-c when I was still using
clang, so I brew remove'd and re-brew install'd it, and that actually got
me further, to errors in index.cpp:

Tue 19:41:36 ryan@mbp: vg:master$ make
g++ -std=c++11 -fopenmp -g -O3 -c -o vg.o vg.cpp -I./ -Ipb2json -Icpp -Ivcflib/src -Ivcflib -Ifastahack -Igssw/src -Irocksdb/include -Iprogress_bar -Isparsehash/build/include -Ilru_cache
g++ -std=c++11 -fopenmp -g -O3 -c -o cpp/vg.pb.o cpp/vg.pb.cc -I./ -Ipb2json -Icpp -Ivcflib/src -Ivcflib -Ifastahack -Igssw/src -Irocksdb/include -Iprogress_bar -Isparsehash/build/include -Ilru_cache
g++ -std=c++11 -fopenmp -g -O3 -c -o main.o main.cpp -I./ -Ipb2json -Icpp -Ivcflib/src -Ivcflib -Ifastahack -Igssw/src -Irocksdb/include -Iprogress_bar -Isparsehash/build/include -Ilru_cache
g++ -std=c++11 -fopenmp -g -O3 -c -o index.o index.cpp -I./ -Ipb2json -Icpp -Ivcflib/src -Ivcflib -Ifastahack -Igssw/src -Irocksdb/include -Iprogress_bar -Isparsehash/build/include -Ilru_cache
index.cpp: In member function 'const string vg::Index::key_for_node(int64_t)':
index.cpp:119:20: error: 'htobe64' was not declared in this scope
id = htobe64(id);
^
index.cpp: In member function 'const string vg::Index::key_for_edge_from_to(int64_t, int64_t)':
index.cpp:133:20: error: 'htobe64' was not declared in this scope
to = htobe64(to);
^
index.cpp: In member function 'const string vg::Index::key_for_edge_to_from(int64_t, int64_t)':
index.cpp:151:20: error: 'htobe64' was not declared in this scope
to = htobe64(to);
^
index.cpp: In member function 'const string vg::Index::key_for_kmer(const string&, int64_t)':
index.cpp:168:20: error: 'htobe64' was not declared in this scope
id = htobe64(id);
^
index.cpp: In member function 'const string vg::Index::key_for_node_path(int64_t, int64_t, int64_t)':
index.cpp:182:30: error: 'htobe64' was not declared in this scope
node_id = htobe64(node_id);
^
index.cpp: In member function 'const string vg::Index::key_for_path_position(int64_t, int64_t, int64_t)':
index.cpp:202:30: error: 'htobe64' was not declared in this scope

For unknown reasons, I can't even get back to this point after having
removed everything and started over; I'm currently stuck at the
aforementioned pb2json linker errors.

Just putting all of this here in case people have ideas / anyone else hits
the same issues.

—
Reply to this email directly or view it on GitHub
#3.

[ryan-williams]

thanks. re: openmp, you mean for the "omp.h" error I referenced? That went away when I switched from clang to actual gcc/g++

[ekg]

Oh! Nice.

The endian functions could be totally imported and the dependence on
endian.h dropped. Is that all that's blocking you now?
On Mar 10, 2015 8:37 PM, "Ryan Williams" [email protected] wrote:

thanks. re: openmp, you mean for the "omp.h" error I referenced? That went
away when I switched from clang to actual gcc/g++

—
Reply to this email directly or view it on GitHub
#3 (comment).

[ryan-williams]

unfortunately I rm'd the version where I'd gotten to the htobe64 errors.

atm I have the pb2json linker errors

[ekg]

I suspect you'll need the protocol buffers compiler and development
libraries.
On Mar 10, 2015 8:41 PM, "Ryan Williams" [email protected] wrote:

unfortunately I rm'd the version where I'd gotten to the htobe64 errors.

atm I have the pb2json linker errors
https://gist.github.com/ryan-williams/715a1ac797633617aa3f#file-make-pb2json-L70-L85

—
Reply to this email directly or view it on GitHub
#3 (comment).

[ryan-williams]

not sure I follow. I have the protobuf compiler installed: protoc --cpp_out=./ person.proto in pb2json/test works.

something really is fishy is happening, actually. After the protobuf linker errors when running make from the vg root, I cd'd into pb2json and ran make, got the same errors, went back to vg root dir, ran make again, and got to the htobe64 errors again!

[ekg]

It might be in need of the development libraries themselves. Are you sure
they are installed with protoc? Can you see all the headers?
On Mar 10, 2015 8:55 PM, "Ryan Williams" [email protected] wrote:

not sure I follow. I have the protobuf compiler installed: protoc
--cpp_out=./ person.proto in pb2json/test works.

something really is fishy is happening, actually. After the protobuf
linker errors when running make from the vg root, I cd'd into pb2json and
ran make, got the same errors, went back to vg root dir, ran make again,
and got to the htobe64 errors again!

—
Reply to this email directly or view it on GitHub
#3 (comment).

[ryan-williams]

yea, sorry, I got protoc via brew install protobuf-c which includes brew install protobuf as a dependency.

[ryan-williams]

I think that running make again after the pb2json linker errors is just incorrectly moving on from pb2json after e.g. seeing some evidence that compilation happened there.

It's similar to this failure from my gist where, after the freebayes#83 issue, a second make fails in a spurious way that a clean fixes.

In any case, having tricked make into moving past pb2json, it went for a while and hit errors like this when building rocksdb:

cd rocksdb && /Applications/Xcode.app/Contents/Developer/usr/bin/make static_lib
g++  -g -Wall -Werror -Wsign-compare -Wshadow -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_MACOSX -DZLIB -DBZIP2 -march=native   -DHAVE_JEMALLOC -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -c db/builder.cc -o db/builder.o
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1502:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1588:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1683:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1767:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1840:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:1914:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:3350:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:3397:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:3429:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7418:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7521:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7540:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7558:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7580:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7600:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7605:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7611:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7624:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7633:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7648:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7679:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7685:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7701:no such instruction: `vzeroupper'
/var/folders/m0/mj2x82p1527349z6mn8btgtr0000gn/T//ccPuhXU9.s:7709:no such instruction: `vzeroupper'
make[1]: *** [db/builder.o] Error 1
make: *** [rocksdb/librocksdb.a] Error 2

[ryan-williams]

I should add that I also added these lines to index.cpp by way of a port of <endian.h>, which got me past the htobe64 errors

[ekg]

I'll pull that patch! Can you send a PR?
On Mar 10, 2015 9:45 PM, "Ryan Williams" [email protected] wrote:

I should add that I also added these lines
https://github.com/zbackup/zbackup/blob/f4ff7bd8ec63b924a49acbf3a4f9cf194148ce18/endian.hh#L10-L28
to index.cpp by way of a port of <endian.h>, which got me past the htobe64
errors

—
Reply to this email directly or view it on GitHub
#3 (comment).

[ryan-williams]

sure, sent #4

[ryan-williams]

update: using PORTABLE=1 make seems to unstick the rocksdb submodule, per this link (the -march=native in my comment above about that, and in all rocksdb compiler invocations by default, was the issue)

[ryan-williams]

update, rocksdb seemed to finish installing, snappy failed with:

./autogen.sh: line 5: libtoolize: command not found
make: *** [snappy/libsnappy.a] Error 127

I had glibtoolize installed (via brew), so I made a symlink to it on my $PATH called libtoolize. That fixed that.

Later on, snappy failed with a bunch of linker errors.

[david4096]

I followed the homebrew instructions on the README. cmake wasn't included so I installed it using homebrew.

This is the error I'm at now.

protoc --proto_path=src --cpp_out=cpp src/example.proto 
[libprotobuf WARNING google/protobuf/compiler/parser.cc:492] No syntax specified for the proto file. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
g++ -O3 -std=c++11 -fopenmp -g -c -o obj/main.o src/main.cpp -I./ -Isrc -Iobj -Icpp -L./ -L/usr/local/lib/ -lprotobuf -lz
g++ -O3 -std=c++11 -fopenmp -g -c -o cpp/example.pb.o cpp/example.pb.cc -I./ -Isrc -Iobj -Icpp
g++ -O3 -std=c++11 -fopenmp -g -o bin/example cpp/example.pb.o obj/main.o -I./ -Isrc -Iobj -Icpp -L./ -L/usr/local/lib/ -lprotobuf -lz && cp src/*.h* include/
Undefined symbols for architecture x86_64:
  "google::protobuf::MessageLite::ParseFromString(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      bool stream::for_each<Person>(std::basic_istream<char, std::char_traits<char> >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
  "google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))", referenced from:
      protobuf_AddDesc_example_2eproto()     in example.pb.o
      (anonymous namespace)::protobuf_RegisterTypes(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in example.pb.o
      protobuf_AssignDesc_example_2eproto()     in example.pb.o
      Person::default_instance()      in example.pb.o
  "google::protobuf::io::CodedInputStream::ReadString(std::basic_string<char, std::char_traits<char>, std::allocator<char> >*, int)", referenced from:
      bool stream::for_each<Person>(std::basic_istream<char, std::char_traits<char> >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
  "google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned char*)", referenced from:
      Person::SerializeWithCachedSizesToArray(unsigned char*) const in example.pb.o
  "google::protobuf::io::IstreamInputStream::IstreamInputStream(std::basic_istream<char, std::char_traits<char> >*, int)", referenced from:
      bool stream::for_each<Person>(std::basic_istream<char, std::char_traits<char> >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
  "google::protobuf::io::OstreamOutputStream::OstreamOutputStream(std::basic_ostream<char, std::char_traits<char> >*, int)", referenced from:
      bool stream::write<Person>(std::basic_ostream<char, std::char_traits<char> >&, unsigned long long, std::function<Person (unsigned long long)>&) in main.o
  "google::protobuf::internal::ArenaStringPtr::AssignWithDefault(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, google::protobuf::internal::ArenaStringPtr)", referenced from:
      Person::MergeFrom(google::protobuf::Message const&) in example.pb.o
      Person::Person(Person const&) in example.pb.o
      Person::Person(Person const&) in example.pb.o
      Person::MergeFrom(Person const&) in example.pb.o
      Person::CopyFrom(Person const&) in example.pb.o
  "google::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)", referenced from:
      Person::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in example.pb.o
  "google::protobuf::internal::WireFormatLite::ReadBytes(google::protobuf::io::CodedInputStream*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)", referenced from:
      Person::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*)      in example.pb.o
  "google::protobuf::MessageLite::SerializeToString(std::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const", referenced from:
      bool stream::write<Person>(std::basic_ostream<char, std::char_traits<char> >&, unsigned long long, std::function<Person (unsigned long long)>&) in main.o
  "google::protobuf::DescriptorPool::FindFileByName(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const", referenced from:
      protobuf_AssignDesc_example_2eproto()     in example.pb.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [bin/example] Error 1
make[1]: *** [include/stream.h] Error 2
make: *** [lib/libxg.a] Error 2

By the way, which protoc:

➜  vg git:(master) ✗ which protoc
/Users/david/vg/bin/protoc
➜  vg git:(master) ✗ protoc --version
libprotoc 3.0.0

[ekg]

Try removing the compilation of example.proto. That suggested, I do not
understand the error.
On Feb 25, 2016 7:01 PM, "David Steinberg" [email protected] wrote:

I followed the homebrew instructions on the README. cmake wasn't included
so I installed it using homebrew.

This is the error I'm at now.

protoc --proto_path=src --cpp_out=cpp src/example.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:492] No syntax specified for the proto file. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
g++ -O3 -std=c++11 -fopenmp -g -c -o obj/main.o src/main.cpp -I./ -Isrc -Iobj -Icpp -L./ -L/usr/local/lib/ -lprotobuf -lz
g++ -O3 -std=c++11 -fopenmp -g -c -o cpp/example.pb.o cpp/example.pb.cc -I./ -Isrc -Iobj -Icpp
g++ -O3 -std=c++11 -fopenmp -g -o bin/example cpp/example.pb.o obj/main.o -I./ -Isrc -Iobj -Icpp -L./ -L/usr/local/lib/ -lprotobuf -lz && cp src/.h include/
Undefined symbols for architecture x86_64:
"google::protobuf::MessageLite::ParseFromString(std::basic_string<char, std::char_traits, std::allocator > const&)", referenced from:
bool stream::for_each(std::basic_istream<char, std::char_traits >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
"google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const_, void ()(std::basic_string<char, std::char_traits, std::allocator > const&))", referenced from:
protobuf_AddDesc_example_2eproto() in example.pb.o
(anonymous namespace)::protobuf_RegisterTypes(std::basic_string<char, std::char_traits, std::allocator > const&) in example.pb.o
protobuf_AssignDesc_example_2eproto() in example.pb.o
Person::default_instance() in example.pb.o
"google::protobuf::io::CodedInputStream::ReadString(std::basic_string<char, std::char_traits, std::allocator >, int)", referenced from:
bool stream::for_each(std::basic_istream<char, std::char_traits >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
"google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::basic_string<char, std::char_traits, std::allocator > const&, unsigned char_)", referenced from:
Person::SerializeWithCachedSizesToArray(unsigned char_) const in example.pb.o
"google::protobuf::io::IstreamInputStream::IstreamInputStream(std::basic_istream<char, std::char_traits >, int)", referenced from:
bool stream::for_each(std::basic_istream<char, std::char_traits >&, std::function<void (Person&)>&, std::function<void (unsigned long long)>&) in main.o
"google::protobuf::io::OstreamOutputStream::OstreamOutputStream(std::basic_ostream<char, std::char_traits >, int)", referenced from:
bool stream::write(std::basic_ostream<char, std::char_traits >&, unsigned long long, std::function<Person (unsigned long long)>&) in main.o
"google::protobuf::internal::ArenaStringPtr::AssignWithDefault(std::basic_string<char, std::char_traits, std::allocator > const_, google::protobuf::internal::ArenaStringPtr)", referenced from:
Person::MergeFrom(google::protobuf::Message const&) in example.pb.o
Person::Person(Person const&) in example.pb.o
Person::Person(Person const&) in example.pb.o
Person::MergeFrom(Person const&) in example.pb.o
Person::CopyFrom(Person const&) in example.pb.o
"google::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::basic_string<char, std::char_traits, std::allocator > const&, google::protobuf::io::CodedOutputStream_)", referenced from:
Person::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream_) const in example.pb.o
"google::protobuf::internal::WireFormatLite::ReadBytes(google::protobuf::io::CodedInputStream_, std::basic_string<char, std::char_traits, std::allocator >)", referenced from:
Person::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream) in example.pb.o
"google::protobuf::MessageLite::SerializeToString(std::basic_string<char, std::char_traits, std::allocator >_) const", referenced from:
bool stream::write(std::basic_ostream<char, std::char_traits >&, unsigned long long, std::function<Person (unsigned long long)>&) in main.o
"google::protobuf::DescriptorPool::FindFileByName(std::basic_string<char, std::char_traits, std::allocator > const&) const", referenced from:
protobuf_AssignDesc_example_2eproto() in example.pb.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [bin/example] Error 1
make[1]: *** [include/stream.h] Error 2
make: *** [lib/libxg.a] Error 2

By the way, which protoc:

➜ vg git:(master) ✗ which protoc
/Users/david/vg/bin/protoc
➜ vg git:(master) ✗ protoc --version
libprotoc 3.0.0

—
Reply to this email directly or view it on GitHub
#3 (comment).

[adamnovak]

We've updated Protobuf a few times since February, and Travis isn't showing any problems on OS X. I'm going to close this; it it's still broken please re-open.