Comments (18)
Hi,
I have created this simple example that replicates the issue. When I run it in x86 mode the limit is 512Mb, whereas in x64 mode the limit is 2 GB. LargeAddressAware option does not affect any mode. The result is the same.
I am using stringstream because GLB is using stringstream as found here:
Here is my example
#include <iostream>
#include <sstream>
int main() {
// Create a chunk of data that is 128 Mbs
int szDataChunk = pow(2, 27);
char* data = new char[szDataChunk];
//The number of times to add this chunk to a stream.
// So, 128 Mbs * 32 = 2^(27+5) = 2^32 = 4 GBs
int ntimes = pow(2, 5);
std::stringstream* m_stream = new std::stringstream();
for (int i = 0; i < ntimes; i++)
{
// Write the data to stream
m_stream->write(data, szDataChunk);
// See the size onscreen
std::cout << m_stream->tellp() << " bytes | "
<< m_stream->tellp()/ (1024*1024) << " Mb \n";
}
return 0;
}
from gltf-sdk.
It seems that it is OS / Compiler related.
-
Windows 10 with MSVC 142, can go up to 2GB.
-
CentOS 6.5 with gcc 4.9.4, can go up to 8GBs:
https://developercommunity.visualstudio.com/t/limited-maximum-length-of-stdstringstream-on-x64/102668 -
Ubunt 22.04 with gcc 11.3.0 can go up to 4GBs. Live example (https://onlinegdb.com/N0znB-g-x)
from gltf-sdk.
See here: KhronosGroup/glTF#2114
from gltf-sdk.
I don't think it is related to #2114.
It is related to the limit of std::ostream to 2GB because we are flushing data in the end of all processes:
I think we should flush more frequently to the file in order to avoid the 2GB limit of std::ostream.
It has also to do that we have only
1 Buffer
and
1 BufferView
from gltf-sdk.
I don't think it is related to #2114.
The code issue is not related, but the GLB file format itself cannot be bigger than 2GB because the header uses a uint32 for the length. Even if we fix the code issue, it still won't work.
EDIT: Or are you writing to a glTF and not a GLB?
from gltf-sdk.
Hi,
I am trying to export a 2.8GB CAE model in a single GLB file. So far, I have managed to export models of maximum size of int32, namely 2GB.
The total file length limit and the chunck1 (binary part) length limit are both uint32. See below :
The uint32 maximum is 4GB: 0xFFFFFFFF (or 4294967295 in decimal)
So I assume that writing a 3.8 GB binary buffer with a 0.2 GB json is possible as regards GLB standards.
The crash in GLTF-SDK happens during addAccessor - more specifically in StreamUtils::WriteBinary(std::ostream& stream, const T* data, size_t size). It has the following callstack:
I have made a function that calculates the number of bytes (N) needed for my model a priori - without any bufferBuilder - by iterating all my models parts and summing up the nbytes for indices, nbytes for vertices, nbytes for morphing, and nbytes for vertex color for the last simulation frame.
Somehow, I should find a way to pre-allocate N bytes in ostream, so that StreamUtils::WriteBinary(std::ostream& stream, const T* data, size_t size) does not crash when the size of stream exceeds 2GBs.
Best,
Dimitrios
from gltf-sdk.
In my PC the streamsize does not seem to suffer the 2GB limit
from gltf-sdk.
Ahh, sorry, I'm dumb and I'm not thinking about unsigned. Well, that seems like a bug then. Do you have an easy repro I can use to test the code?
from gltf-sdk.
Unfortunately, I can not share the CAE models, and it is difficult to reproduce without a very big model. I am copying some screenshots with information below. It is seen that if the limit of 2GB is reached while adding accessors through bufferBuilder, then the size of the ostream (in this case it is stream in ram) becomes -1, and failbit and badbit become true.
See information for failbit and badbit in here: https://cplusplus.com/reference/ios/ios/rdstate/
from gltf-sdk.
Hi, I have found this below but not tested yet as my project is compiled with make and I do not know how to insert this option.
In order in C++ to use objects more than 2GB you should set /LARGEADDRESSAWARE in Linker properties in
Configuration Properties > Linker > System property page. See these resource for more details:
[1] https://stackoverflow.com/questions/37413998/why-my-program-does-not-take-more-than-2-gb-ram-on-64-gb-ram-system
[2] https://learn.microsoft.com/en-us/cpp/build/reference/largeaddressaware-handle-large-addresses?view=msvc-170
[3] https://stackoverflow.com/questions/3109543/what-to-do-to-make-application-large-address-aware
from gltf-sdk.
On the other side, there is no strict limit if we use std::string instead of std:stringstream. The example below in Windows demonstrates that std::string can easily reach 16GBs. Moreover std::string seems faster and we can also preallocate the memory needed by calculating a priori the number of bytes per indice and vertex (Stringstream does not allow preallocation). However, many things should change in the GLTF-SDK code ...
from gltf-sdk.
It looks like Microsoft's STL implementation limits string buffers to INT_MAX.
from gltf-sdk.
Looks like there were similar issues before:
microsoft/STL#578
microsoft/STL#388
@jimver04 Do you mind filing an issue on Microsoft's STL for your test case?
from gltf-sdk.
@jimver04 I don't know what your code looks like, but GLBResourceWriter
has a constructor with a second argument std::unique_ptr<std::iostream> tempBufferStream
. If you pass in your own for this (maybe using a local file instead), it will avoid the usage of std::stringstream
. Hopefully this is good enough until the STL is fixed.
from gltf-sdk.
@bghgary It seems that it is an old issue that never got priority in MSVC. In Gcc the limits seem to be based on __string_type::size_type
https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/std/sstream which is not so explicitly provided. So I guess it goes up to memory available. I have started modifying GLTF-SDK to replace std:stringstream with std::string wherever necessary. Thanks for the information. My changes are send to branch: https://github.com/jimver04/glTF-SDK
from gltf-sdk.
I have also mentioned the issue of 4GB limitation of GLTF to Khronos group:
KhronosGroup/glTF#1051 (comment)
from gltf-sdk.
@bghgary It seems that it is an old issue that never got priority in MSVC. In Gcc the limits seem to be based on __string_type::size_type https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/std/sstream which is not so explicitly provided. So I guess it goes up to memory available. I have started modifying GLTF-SDK to replace std:stringstream with std::string wherever necessary. Thanks for the information. My changes are send to branch: https://github.com/jimver04/glTF-SDK
I don't see this as an old issue. As I pointed to earlier, the code is checking against INT_MAX and fails to allocate bigger. The two old issues I pointed to are fixed as far as I can tell. I don't see a reason why std::stringstream should be limited to INT_MAX.
from gltf-sdk.
I am trying to use std::string instead of std::stringstream. Why do you have caches ? StreamCache, IStreamCache, StreamCacheLRU ? What do you cache ? The Buffer stream or the File output path ? and why ?
from gltf-sdk.
Related Issues (20)
- Is it possible to write not interleaved buffers? HOT 1
- Exceptions necessary ? HOT 2
- Can it export quads into gltf ? HOT 2
- How to export data as ascii text in GLTF? HOT 1
- ByteOffset should be a multiple of 4 in Javascript HOT 3
- Changes - suggestion for the Serialization example HOT 4
- Procedure to upgrade for the support many Vertex Colors ? HOT 2
- Example for animation? HOT 3
- about draco HOT 2
- clang/LLVM for Windows build failures HOT 3
- compile error, SchemaValidation HOT 1
- This repo is missing important files HOT 1
- GenerateSchemaJsonHeader.ps1 fails because D:\source\GLTFSDK does not exist
- Point size and Line width HOT 1
- googletest, POWERSHELL_PATH-NOTFOUND and other errors on Linux HOT 1
- Request for GLTF-SDK Support on Visual Studio 2022 HOT 2
- Add `ResourceWriter::Write` overload with a buffer id parameter instead of a buffer view
- Can a single vertex have multiple UV coordinates depending on how many faces it belongs to? HOT 1
- Add support for punctual lights extension
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gltf-sdk.