Comments (21)
hehe, I see what happened. The Markdown syntax ate the underscores; it should read char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};
.
from highwayhash.
@diegocrzt, you need to edit the file external/highwayhash/highwayhash/state_helpers.h
to make the changes to line 31 that @jan-wassenberg listed above. It should read
char __attribute__((aligned(32))) final_packet[State::kPacketSize] = {0};
Then try re-compiling.
from highwayhash.
@diegocrzt no problem. That is an unrelated issue. According to this thread, your version of gcc is too old. If you are using a scientific computing environment, try typing module load gcc
and tab-complete that command before running it to see which versions you have available. Try loading the latest version and re-compiling. For example, module load gcc/4.7.1
or module load gcc/4.9.1
depending on what versions you have.
from highwayhash.
Hi, thanks for reporting. We require GCC 4.8 for "alignas". Is it an option for you to compile with a more recent GCC or clang?
Alternatively, you can replace alignas(32) with attribute((aligned(32))).
from highwayhash.
Hi @jan-wassenberg, thanks for the reply. Here's my diff:
- alignas(32) char final_packet[State::kPacketSize] = {0};
+ attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};
Unfortunately, compiling with gcc 4.9.1 leads with a whole set of other errors (though not with highwayhash), and the next-closest version I have is 4.7.2. I'm using a compute cluster so the gcc versions available to me are limited.
I'm now getting the same error:
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:28: error: expected ';' before 'char'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
Here's the block of code. Could it be a syntax issue?
#ifndef HIGHWAYHASH_HIGHWAYHASH_STATE_H_
#define HIGHWAYHASH_HIGHWAYHASH_STATE_H_
// Helper functions to split inputs into packets and call State::Update on each.
#include <cstddef>
#include <cstring>
#include <memory>
#include "highwayhash/code_annotation.h"
#include "highwayhash/types.h"
namespace highwayhash {
// Copies the remaining bytes to a zero-padded buffer, sets the upper byte to
// size % 256 (always possible because this should only be called if the
// total size is not a multiple of the packet size) and updates hash state.
//
// The padding scheme is essentially from SipHash, but permuted for the
// convenience of AVX-2 masked loads. This function must use the same layout so
// that the vector and scalar HighwayTreeHash have the same result.
//
// "remaining_size" is the number of accessible/remaining bytes
// (size % kPacketSize).
//
// Primary template; the specialization for AVX-2 is faster. Intended as an
// implementation detail, do not call directly.
template <class State>
INLINE void PaddedUpdate(const uint64 size, const char* remaining_bytes,
const uint64 remaining_size, State* state) {
attribute((aligned(32))) char final_packet[State::kPacketSize] = {0};
// Unusual layout matches the AVX-2 specialization in highway_tree_hash.h.
const size_t remainder_mod4 = remaining_size & 3;
uint32 packet4 = static_cast<uint32>(size) << 24;
const char* final_bytes = remaining_bytes + remaining_size - remainder_mod4;
for (size_t i = 0; i < remainder_mod4; ++i) {
const uint32 byte = static_cast<unsigned char>(final_bytes[i]);
packet4 += byte << (i * 8);
}
memcpy(final_packet, remaining_bytes, remaining_size - remainder_mod4);
memcpy(final_packet + State::kPacketSize - 4, &packet4, sizeof(packet4));
state->Update(final_packet);
}
// Updates hash state for every whole packet, and once more for the final
// padded packet.
template <class State>
INLINE void UpdateState(const char* bytes, const uint64 size, State* state) {
// Feed entire packets.
const int kPacketSize = State::kPacketSize;
static_assert((kPacketSize & (kPacketSize - 1)) == 0, "Size must be 2^i.");
const size_t remainder = size & (kPacketSize - 1);
const size_t truncated_size = size - remainder;
for (size_t i = 0; i < truncated_size; i += kPacketSize) {
state->Update(bytes + i);
}
PaddedUpdate(size, bytes + truncated_size, remainder, state);
}
// Convenience function for updating with the bytes of a string.
template <class String, class State>
INLINE void UpdateState(const String& s, State* state) {
const char* bytes = reinterpret_cast<const char*>(s.data());
const size_t size = s.length() * sizeof(typename String::value_type);
UpdateState(bytes, size, state);
}
// Computes a hash of a byte array using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 }; char data[4];
// ComputeHash<SipHashState>(key, data, sizeof(data));
//
// This function avoids duplicating Update/Finalize in every call site.
// Callers wanting to combine multiple hashes should repeatedly UpdateState()
// and only call State::Finalize once.
template <class State>
uint64 ComputeHash(const typename State::Key& key, const char* bytes,
const uint64 size) {
State state(key);
UpdateState(bytes, size, &state);
return state.Finalize();
}
// Computes a hash of a string's bytes using the given hash State class.
//
// Example: const SipHashState::Key key = { 1, 2 };
// StringHasher<SipHashState>()(key, std::u16string(u"abc"));
//
// A struct with nested function template enables deduction of the String type.
template <class State>
struct StringHasher {
template <class String>
uint64 operator()(const typename State::Key& key, const String& s) {
State state(key);
UpdateState(s, &state);
return state.Finalize();
}
};
} // namespace highwayhash
#endif // HIGHWAYHASH_HIGHWAYHASH_STATE_H_
from highwayhash.
Hi, good news, it will compile with just a minor change; GCC wants the type first and attribute second.
attribute((aligned(32))) char -> char attribute((aligned(32))).
from highwayhash.
@jan-wassenberg still getting a similar error. Sorry for the silly fixes:
In file included from external/highwayhash/highwayhash/sip_hash.h:24:0,
from external/highwayhash/highwayhash/sip_hash.cc:15:
external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)':
external/highwayhash/highwayhash/state_helpers.h:31:33: error: expected ',' or ';' before 'final_packet'
external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope
from highwayhash.
@jan-wassenberg Thank you so much! That worked!
from highwayhash.
I've run into this issue (while compiling tensorflow) as well.
@jan-wassenberg: if the change you suggested makes the code compatible with more gcc versions, why not to make it permanent?
from highwayhash.
Hi @mnicky, it's a tradeoff - one disadvantage of __attribute__
is that it's not supported by MSVC. I had hoped that C++11 is widely available by now. Can you share which GCC version you have, and why it's infeasible to use a more recent version? Perhaps we might switch to a macro if necessary.
from highwayhash.
@mnicky, if you are in a scientific computing Red Hat / CentOS environment, you might have more recent versions of gcc available. Try module load gcc<tab complete>
to check. The default on my system was much older than the other available versions. Disregard if this isn't your situation...
from highwayhash.
I have gcc 4.7.2 which is included in Debian Wheezy that runs on our cluster. In this case I'm able to modify the source file to get rid of the compilation error. I was just curious why the workaround/fix didn't get it into the master. I think there are still quite a few people running older versions of gcc without possibility of upgrading. And as highwayhash is needed to build tensorflow, I'd assume there are people affected by this.
from highwayhash.
Ah, thanks for mentioning your GCC version. That's still fairly recent; it is unfortunate that alignas didn't make it in until 4.8. We will soon add a HH_ALIGNAS macro that resolves to alignas or falls back to attribute.
from highwayhash.
The fix is in - please reopen if there's still an issue :)
from highwayhash.
I am using GCC5.3.0, and getting a similar error:
C++ compilation of rule '@highwayhash//:sip_hash' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command
My set up is:
Bazel 0.4.3 (compiled from source)
Tensorflow 0.12.1 (from zip file)
GCC 5.3.0
Cuda 8.0
CuDNN 5.1
from highwayhash.
Can you include more output to show the actual compile error and in which file/header it occurs?
Should be a few lines below the "error executing command" you mentioned.
from highwayhash.
Sorry for the late reply. I started from scratch (including removing ~/.cache/bazel...), and it seems TF now compiles fine. Should I delete my previous message?
from highwayhash.
Glad to hear it! No worries, we can leave this issue unchanged/closed.
from highwayhash.
Hi guys/girls, I am having the same problem with gcc 4.7.4
ERROR: /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/external/highwayhash/BUILD:125:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 32 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. In file included from external/highwayhash/highwayhash/sip_hash.h:24:0, from external/highwayhash/highwayhash/sip_hash.cc:15: external/highwayhash/highwayhash/state_helpers.h: In function 'void highwayhash::PaddedUpdate(highwayhash::uint64, const char*, highwayhash::uint64, State*)': external/highwayhash/highwayhash/state_helpers.h:31:13: error: there are no arguments to 'alignas' that depend on a template parameter, so a declaration of 'alignas' must be available [-fpermissive] external/highwayhash/highwayhash/state_helpers.h:31:13: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) external/highwayhash/highwayhash/state_helpers.h:31:15: error: expected ';' before 'char' external/highwayhash/highwayhash/state_helpers.h:42:10: error: 'final_packet' was not declared in this scope Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 239.020s, Critical Path: 225.21s
But after reading this I am not sure what to do. And I can't upgrade my version of GCC
from highwayhash.
@rasmi, thank you for you quick answer and sorry for not understanding at first time, I was lost on the first markdown issues 😆, highwayhash step is done but right now I have other issue, with the core tensorflow, I think this is probably related no more to this thread, but just to be sure, what do you think, it could be related yet ?
ERROR: /home/diegocrzt/codice/dpi/tensorflow/tensorflow-git/tensorflow/contrib/layers/BUILD:19:1: C++ compilation of rule '//tensorflow/contrib/layers:python/ops/_bucketization_op.so' failed: gcc failed: error executing command
(cd /home/diegocrzt/.cache/bazel/_bazel_diegocrzt/8df0525fbbd1e101024ee0b817c7eaef/execroot/tensorflow-git && \
exec env - \
PATH=/usr/local/bin:/bin:/usr/bin:/opt/texlive/2016/bin/x86_64-linux:/opt/java/jdk/bin \
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++0x' -MD -MF bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.d '-frandom-seed=bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY -iquote . -iquote bazel-out/local-py3-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local-py3-opt/genfiles/external/bazel_tools -iquote external/eigen_archive -iquote bazel-out/local-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local-py3-opt/genfiles/external/local_config_sycl -iquote external/protobuf -iquote bazel-out/local-py3-opt/genfiles/external/protobuf -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/eigen_archive -isystem bazel-out/local-py3-opt/genfiles/external/eigen_archive -isystem external/protobuf/src -isystem bazel-out/local-py3-opt/genfiles/external/protobuf/src -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions -pthread -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/contrib/layers/ops/bucketization_op.cc -o bazel-out/local-py3-opt/bin/tensorflow/contrib/layers/_objs/python/ops/_bucketization_op.so/tensorflow/contrib/layers/ops/bucketization_op.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./tensorflow/core/lib/gtl/array_slice.h:101:0,
from ./tensorflow/core/lib/strings/str_util.h:23,
from ./tensorflow/core/framework/op.h:29,
from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/array_slice_internal.h:232:38: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<const T>::ArraySliceImplBase' names constructor
./tensorflow/core/lib/gtl/array_slice_internal.h:252:32: error: 'tensorflow::gtl::array_slice_internal::ArraySliceImplBase<T>::ArraySliceImplBase' names constructor
In file included from ./tensorflow/core/lib/gtl/array_slice.h:102:0,
from ./tensorflow/core/lib/strings/str_util.h:23,
from ./tensorflow/core/framework/op.h:29,
from tensorflow/contrib/layers/ops/bucketization_op.cc:16:
./tensorflow/core/lib/gtl/inlined_vector.h: In member function 'void tensorflow::gtl::InlinedVector<T, N>::Destroy(T*, int)':
./tensorflow/core/lib/gtl/inlined_vector.h:396:10: error: 'is_trivially_destructible' is not a member of 'std'
./tensorflow/core/lib/gtl/inlined_vector.h:396:42: error: expected primary-expression before '>' token
./tensorflow/core/lib/gtl/inlined_vector.h:396:43: error: '::value' has not been declared
Target //tensorflow/tools/pip_package:build_pip_package failed to build
from highwayhash.
Thank you @rasmi, is clear for me now. I will see what can I do with my environment
from highwayhash.
Related Issues (20)
- Unexpected low speeds with the generic C?! HOT 2
- New Nodejs bindings package HOT 2
- highwayhash-wasm - WASM JS binding for the browser and nodejs HOT 3
- Alignment attributes are in the wrong place for arrays HOT 1
- Build fails (gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0) HOT 6
- Small bug in documentation (?) HOT 1
- Question: When the README says "an expected 2^32 guesses of m per the birthday paradox", does it refers to 128 and 256-bits versions too? HOT 3
- Alignment warning when compiling with GCC7 on aarch64 HOT 2
- Issues with Core 2 Duo HOT 1
- [Partial implementation] NEON version HOT 1
- Question: SSSE3 HOT 3
- doc: wikipedia article has been deleted HOT 1
- Failed to build on arm64 HOT 1
- Please strip the installed library HOT 1
- SipHash performs badly on short msgs (< 100 bytes). And missing le64toh swap on key? HOT 4
- Installs an unstripped binary lib/libhighwayhash.so HOT 3
- Builds static library with relocatable code (-fPIC) HOT 3
- Problem with Python 3 bindings HOT 5
- Test fails with gcc 10.2.0 HOT 3
- The tricky __SSE4_1__ macro under Visual Studio HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from highwayhash.