Boost Endian library
The Endian library provides facilities for dealing with endianness. It's part of Boost since release 1.58.0. See the documentation for more information.
C++11 is required since release 1.84.
Tested on Github Actions and Appveyor.
Distributed under the Boost Software License, Version 1.0.
One often needs to read or write values from/to a buffer while having a pointer. Could you support this?
I guess one could use *reinterpret_cast<big_int64_buf_t&>(*p) but I think functions would be more convenient.
The functions are probably implemented already but aren't exposed.
The docs clearly state there is no support for floating point type endian conversion. Nevertheless the following code compiles fine on my x86_64 (ie. little endian) computer:
float f = 3.1415f;
double d = 2.7182;
boost::endian::native_to_little(f);
boost::endian::native_to_little(d);Since no conversion needs to be done everything is fine. When this code is changed to native_to_big() (or I assume compiled on a big endian system) the compiler starts complaining about ambiguous overloads:
error: call of overloaded ‘endian_reverse(float&)’ is ambiguous
I understand this is documented but as a C++ programmer I unfortunately have developed the habit that everything is fine as long as the compiler does not bark. So I would prefer a static_assert that always checks if the conversion is supported, regardless of the conversion being actually necessary on the current platform.
I'd be willing to try writing this check but I wanted to ask if such a check would be of interest first?
While the native types are supported by numeric_limits<>, the non-native types are not (at least, as far as I can tell). It would be a really nice addition to the library to have all boost::endian types provide numeric_limits specialisations.
Incidentally, if this is not the correct place to report bugs/feature requests, please let me know and I'll move it to wherever is more appropriate.
Thanks,
Roger
Hi,
Are floating point buffers (ie float and double) going to make their way back?
I took a look at the 1.59 version and I don't see it. Maybe there's a workaround I missed?
And by the way, my compliments, I haven't seen anything nearly as clean to handle endianness.
Thanks,
Rene
Hi,
I am a bit confused by what the documentation about Boost.Endian reads. I read that:
Why is there no floating point support?
An attempt was made to support four-byte floats and eight-byte doubles, limited to IEEE 754 (also known as ISO/IEC/IEEE 60559) floating point and further limited to systems where floating point endianness does not differ from integer endianness. Even with those limitations, support for floating point types was not reliable and was removed. For example, simply reversing the endianness of a floating point number can result in a signaling-NAN. For all practical purposes, binary serialization and endianness for integers are one and the same problem. That is not true for floating point numbers, so binary serialization interfaces and formats for floating point does not fit well in an endian-based library.
However in the change log we can see:
Changes in 1.71.0
- Added support for float and double
Let's say I am only interested in serializing a float/double from one machine to another, over the wire, which may use different architecture. Is it safe to do like for ints (ie convert it to big endian and then use endian_buffer to read/write it on the socket on both sides), or is it indeed doomed to fail in the general case as explained in the first paragraph ?
Cheers,
Romain
We are in the process of making B2 build changes to all of the B2 build files
to support "modular" consumption of the Boost Libraries by users. See this list
post for some details: https://lists.boost.org/Archives/boost/2024/01/255704.php
The process requires making a variety of changes to make each Boost library
independent of the super-project structure. But the changes do not remove the
super-project structure or the comprehensive Boost release. The changes make
solely make it possible, optionally, for users, like package manages, to easily
consume libraries individually.
Generally the changes include:
Some examples of such changes:
We are asking how you would like us to handle the changes. We would prefer if
you allow the owners of the Boost.org GitHub project to make changes to B2
build files, as needed, to accomplish the changes. But understand
that you may want to manage the proposed changes yourself.
We previously sent emails to all known maintainers to fill out a form with their
preference. We are contacting you in this issue as we have not gotten a response
to that email. You can see the ongoing responses for that form and the responses
to these issues here https://github.com/users/grafikrobot/projects/1/views/6
We are now asking if you can reply directly to this issue to indicate your
preference of handling the changes. Please supply a response to this question
and close the issue (so that we can verify you are a maintainer).
How would you like the build changes to be processed?
Also please indicate any special instructions you want us to consider. Or other
information you want us to be aware of.
Thanks you, René
Use byte swap intrinsics in arithmetic types when they are available.
1.71.0 revision history states that support for float and double is added. But why no {little,big,native}_float32_t and {little,big,native}_float64_t typedefs provided?
As far as I understand it, the library now supports float and double. So the FAQ entry 'Why is there no floating point support?' is wrong.
I would love to be able to use the endian swapping routines in contexts where I need to use a constexpr, and it seems like these routines all fit the constexpr premise perfectly (as they are pure math based on a static property of the architecture). FWIW, I tried marking these functions as constexpr, and it worked great ;P.
I might be missing something, but why detect_order() in test/endian_test.cpp does this:
View v = { 0x0102030405060708LL };
and then
memcmp( v.c, "\10\7\6\5\4\3\2\1", 8)
Shouldn't the string be "\8\7\6\5\4\3\2\1"?
Screenshot from Firefox 116 on macOS:
Note that unsigned char const* p is getting rendered as unsigned char const p and the asterisk is getting eaten by some markup generator that's using it to vary the boldness.
conversion.hpp should support char, wchar_t, char16_t, and char32_t.
The constraints for type T in this case are not ducumented.
From the errors at compile time I have deduced them to
// Needed by unaligned
Int() = default;
Int(int i) : value (i) {}
explicit operator char() { return value; }
int operator >>(int n) { return value >> n; }
int operator <<(int n) { return value << n; }
http://melpon.org/wandbox/permlink/erLEWpc1oaR64iMW
While
Int() = default;
explicit Int(int i) : value (i) {}
are natural on a wrapped opaque int, the implicit conversion is less desirable as well as the conversions to char and the shift operations.
However, I believe these constraints are not justified. I believe that the conversion could be explicit and that the implementation could use reinterpret_cast instead of static_cast, as we are reinterpreting the bits. IMHO, the shift shouldn't be required. It should be an implementation detail.
It is not clear from the documentation of endian_buffer that the user needs to overload endian_reverse for UDT.
float_typedef_test.cpp fails in 32-bit linux platform. Line 24 in float_typedef_test.cpp works only for 64-bit target when class big_float64_buf_at or little_float64_buf_at are passed to function test_buffer.
BOOST_TEST_EQ( sizeof(x), aligned? 2 * sizeof(U): 1 + sizeof(U) );
In 32-bit target, this code should be updated for the size of char+double is 12 bytes, not 2*8=16 bytes.
Constexpr std::copy could possibly be used instead of std::memcpy, which I believe was the only thing preventing full constexpr types last time I tried.
Wiven that the conversion from T is explicit, the assignment shouldn't be allowed.
explicit endian_buffer(T v) noexcept;
endian_buffer& operator=(T v) noexcept;Either the conversion is implicit and the assignment is an optimization, or the conversion is explicit and the assignment should be removed.
Even when we define
#define BOOST_ENDIAN_FORCE_PODNESS
#define BOOST_ENDIAN_NO_CTORSendian_buffer is not a POD, because it uses a protected data member.
See https://wandbox.org/permlink/PSYwP0oCoJ9cNtkV
Defining it as public in C++03 solves the issue.
std::is_pod works in C++11 as the definition was relaxed.
The problem is that boost::is_pod is based on C++03 definition of POD.
Formal definition from the C++ standard (C++03 8.5.1 §1):
An aggregate is an array or a class (clause 9) with no user-declared constructors (12.1), no private or protected non-static data members (clause 11), no base classes (clause 10), and no virtual functions (10.3).
Formal definition from the C++ standard (C++03 9 §4):
A POD-struct is an aggregate class that has no non-static data members of type non-POD-struct, non-POD-union (or array of such types) or reference, and has no user-defined copy assignment operator and no user-defined destructor. Similarly, a POD-union is an aggregate union that has no non-static data members of type non-POD-struct, non-POD-union (or array of such types) or reference, and has no user-defined copy assignment operator and no user-defined destructor. A POD class is a class that is either a POD-struct or a POD-union.
while validating my code in UBsan (-fsanitize=undefined) I learned that the endian buffers perform unaligned loads and stores on x86-family of machines.
There are preprocessor checks here and a few other places.
In my mind they are problematic because:
Furthermore, I would expect the compiler to be able to identify a 4-byte or 8-byte memcpy() as a potentially unaligned store/load, and generate a mov (which the compiler knows allows unaligned access), but not to generate SSE instructions.
I'm proposing that these special cases be removed
[1]. my x86 compiler is free to generate SSE instructions that do require aligned access
[2]. Imagine storing a value through a pointer (that the standard requires be aligned) and later check the pointer to see whether or not its last few bits are zero. The compiler may remove the check since it must be aligned, otherwise you invoked UB anyway.
In http://www.boost.org/doc/libs/1_60_0/libs/endian/doc/arithmetic.html is said
In C++03, endian_arithmetic does not meet the requirements for POD types because it has constructors, private data members, and a base class.
and
// if BOOST_ENDIAN_FORCE_PODNESS is defined && C++11 PODs are not
// available then these two constructors will not be present
endian_arithmetic() noexcept = default;
endian_arithmetic(T v) noexcept;
As the class has a base class the constructor doesn't help to make the type a C++03 POD.
While I don't like to repeat the code it is a pity that endian_arithmetic type is not a POD, as it is used in a POD context, to transport data between machines with different endianness.
All the code in endian_buffer could be included inside endian_arithmetic via a common macro, and make this type a POD.
If you don't want to do that, there is no major reason to remove the constructors in C++03 when BOOST_ENDIAN_FORCE_PODNESS is defined.
Relying in UB is not something I would advice to use in Boost.
Recent commit in buffers.hpp omitted to fully qualify namespace resulting in ambiguous symbol error when including boost::spirit headers too. Suggested fix: use boost::endian::* everywhere instead of endian::*. Environment: Windows 10, msvc15, boost 1.7.0 beta1 RC3.
How to reproduce:
#include <boost/spirit/include/qi.hpp>
#include <boost/endian/arithmetic.hpp>
struct record
{
boost::endian::big_int16_t type;
record(boost::int16_t t)
{
type = t;
}
};
The 1.71 documentation is supposed to have floating point conversion. Yet I encounter BOOST_STATIC_ASSERT failures when I try to convert a double.
I have boost version 1.71.0-2 loaded from the arch repositories.
I had a code like this to automatically call endian_reverse_inplace for my custom classes:
template <typename T, typename = std::enable_if_t<!std::is_integral_v<T>>>
static T endian_reverse(T t) noexcept
{
endian_reverse_inplace(t);
return t;
}(I know, that enable_if might be a bit too broad, but I only call endian_reverse on integral types and my own structs of them, so it's probably okay).
As far as I can see from the documentation, EndianReversible requires that endian_reverse(x) is well-formed, it doesn't mention anywhere it can't be a template. With boost 1.70.0 this worked fine, but after updating to 1.72.0 compilation fails with call to 'endian_reverse' is ambiguous. If I remove my endian_reverse function I get a static assert instead from here:
Hi,
While this is purely cosmetic, it would be nice to have native_float32_t and native_float64_t types to match the other types. Unless there's a good reason not to, of course.
Kind regards,
Roger
Documentation doesn't state this requirement clearly. Furthermore, these functions are part of the header conversion.hpp for which documentation states that alignment is required. I think it's correct to assume that T is aligned, but not p (otherwise I'd have to create an extra buffer and perform an extra memcpy for no good reason). What is the requirement for these functions?
1>C:\Users\vinnie\src\boost\boost/endian/buffers.hpp(282): warning C4127: conditional expression is constant (compiling source file C:\Users\vinnie\src\boost\libs\beast\test\beast\websocket\stream.cpp)
1>C:\Users\vinnie\src\boost\boost/endian/buffers.hpp(282): note: consider using 'if constexpr' statement instead (compiling source file C:\Users\vinnie\src\boost\libs\beast\test\beast\websocket\stream.cpp)
1>C:\Users\vinnie\src\boost\boost/endian/buffers.hpp(409): note: see reference to function template instantiation 'T boost::endian::detail::load_little_endian<T,4>(const void *) noexcept' being compiled
inline void big_endian(Reversible& x) BOOST_NOEXCEPT
inline void little_endian(Reversible& x) BOOST_NOEXCEPT
These trigger unused variable compiler warnings when used on the native architecture i.e. little_endian on a little endian system:
/path/to/endian/conversion.hpp:398:41: warning: unused parameter 'x' [-Wunused-parameter]
Could the argument name be #ifdefed out? Or a token operation like (x) be done to prevent the warning?
http://boostorg.github.io/endian
It lets me browse and read the documentation without having to download the library.
In 1.73.0 I get the following error:
fatal error: boost/ detail/endian.hpp: No such file or directory | #include <boost/detail/endian.hpp>
The behavior is different in 1.72.0
When using a c++17 compiler, we may take advantage of the strongly typed std::byte class instead of relying on the arithmetic type char, whether signed (prior to 1.72) or not (1.72 and above).
Since that would probably break client code, it could be an opt-in feature.
Please excuse the paths; this is embedded in our sources until this goes into boost proper. It was embedded verbatim with a tweak to the include guards.
/home/rleigh/code/bioformats/cpp/lib/ome/compat/endian/types.hpp:209:64: warning: conversion to ‘short unsigned int’ from ‘int’ may alter its value [-Wconversion]
template <typename T, std::size_t n_bytes,
bool sign=boost::is_signed<T>::value >
struct unrolled_byte_loops
{
typedef unrolled_byte_loops<T, n_bytes - 1, sign> next;
static T load_big(const unsigned char* bytes) BOOST_NOEXCEPT
{ return *(bytes - 1) | (next::load_big(bytes - 1) << 8); }
^ here
This is when comparing endian types inside a boost.variant, but this is likely an unimportant detail. It looks like GCC 4.9 with -Wconversion triggers this warning.
It looks like the load_ and store_ functions here might be operating on types other than intended. For example maybe this needs to use temporaries of defined type for the bitwise operations and/or a cast to the return type.
Kind regards,
Roger
I'm just checking if I can use your library in my project and for my understanding of the standard it is very unportable.
For signed T, unrolled_byte_loops<T,...>::load_big/little use bitwise or on a signed value. The behaviour of this is well defined (it operates on the bit representation) but since the bit representation of negative values can differ on different architectures, the resulting is unpredictable (more or less). Are you sure that your code works on platforms with one-complement or sign-magnitude representation? (Just a question, as I don't intend to use your code with signed types.)
What I worry more about is that unrolled_byte_loops::store_big/little uses a static_cast to char from a potentially larger type (signed, or in my case unsigned). As far as I can tell, only the cast to an unsigned char is well defined but the cast to an signed type is implementation defined. I know that char, signed char and unsigned char are three distinct types and I think I once red, that the cast to char would also work, but I couldn't find anything in the standard about it. Are you sure the cast to char is really well defined and not just implementation defined? If not, would it hurt to use an unsigend char here? I'd feel better with that.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.