Comments (2)
If I were guessing from your description it sounds like you are stressing BP5's ability to gather metadata. Some of the characteristics of BP5 that make it more efficient at scale, unfortunately put pressure on metadata size in the first timestep. BP5Writer takes steps to minimize this problem, but it can't be completely eliminated. That said, let me ask a quick question about your application. Are you writing attributes on all ranks? Or only on rank 0? (Attributes from all ranks are merged after being gathered, so writing duplicate attributes on non-zero ranks adds unnecessary pressure to metadata.).
from adios2.
Thanks for the quick pointer! This fixed the issue.
For future reference (in case other people stumble across this):
We're using ADIOS2 through OpenPMD. Once I updated to current develop (from 0.15.2) there was an option available to specify the ranks that write attributes (adios2.attribute_writing_ranks = 0
), which successfully solved the issue.
from adios2.
Related Issues (20)
- begin_step gets blocked when meshes become bigger. HOT 4
- bpls fails to read HDF5 files HOT 1
- error loading adios 2.10.1 into python3.12.4 HOT 1
- Fortran BP5 read finds no variables HOT 6
- f951: Fatal Error: Reading module โmpivarsโ at line 1525 column 47: Unexpected EOF HOT 7
- Begin Step slower on large meshes (Benchmarking Adios) HOT 2
- Remote.GetRemote SST tests unstable in Frontier HOT 3
- BP5 aggregating attributes
- Conda for dependendies in CI in Win/Mac HOT 1
- regression test failure with mpi4py 4.0.0 HOT 1
- Failure in Install.Make.* tests in macos
- Failure in some dataman tests in macos HOT 2
- Failure in test Api.Python.FileReader in windows builds
- Failure of Api.Python.BPWriteTypesHighLevelAPI.MPI in windows msmpi builds
- Failure of Api.Python.BPWriteTypesHighLevelAPI_HDF5 in windows builds
- Design a new memory allocation scheme for derived variables
- Build failure: `INTERFACE_LIBRARY targets may only have whitelisted properties.` HOT 4
- Reading numpy-array attributes broken with numpy>2.0.0 with RockyLinux 9
- CMake scripts for the MPI dataplane of SST don't work
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adios2.