Comments (1)
Fixing the main issue of the different memory layouts of collections that are newly created and the ones that have been read from file is actually fairly trivial by introducing a boolean flag to the collection that tracks this:
diff --git a/python/templates/Collection.cc.jinja2 b/python/templates/Collection.cc.jinja2
index 282d45a..01d3834 100644
--- a/python/templates/Collection.cc.jinja2
+++ b/python/templates/Collection.cc.jinja2
@@ -16,7 +16,7 @@
{% with collection_type = class.bare_type + 'Collection' %}
{{ collection_type }}::{{ collection_type }}() :
- m_isValid(false), m_collectionID(0), m_entries()
+ m_isValid(false), m_isReadFromFile(false), m_collectionID(0), m_entries()
{%- for relation in OneToManyRelations + OneToOneRelations %},
m_rel_{{ relation.name }}(new std::vector<{{ relation.namespace }}::Const{{ relation.bare_type }}>())
{%- endfor %},
@@ -109,6 +109,11 @@ void {{ collection_type }}::prepareForWrite() {
const auto size = m_entries.size();
m_data->reserve(size);
for (auto& obj : m_entries) { m_data->push_back(obj->data); }
+
+ // if the collection has been read from a file the rest of the information is
+ // already in the correct format and we have to skip it, since the temporary
+ // buffers are invalid
+ if (m_isReadFromFile) return;
for (auto& pointer : m_refCollections) { pointer->clear(); }
{% for relation in OneToManyRelations %}
@@ -155,6 +160,7 @@ void {{ collection_type }}::prepareAfterRead() {
// have a redundant (but now useless) copy of the data
m_data->clear();
m_isValid = true;
+ m_isReadFromFile = true;
}
bool {{ collection_type }}::setReferences(const podio::ICollectionProvider* collectionProvider) {
diff --git a/python/templates/Collection.h.jinja2 b/python/templates/Collection.h.jinja2
index 03749bb..8ea3641 100644
--- a/python/templates/Collection.h.jinja2
+++ b/python/templates/Collection.h.jinja2
@@ -135,6 +135,7 @@ public:
private:
bool m_isValid;
+ bool m_isReadFromFile{false};
int m_collectionID;
{{ class.bare_type }}ObjPointerContainer m_entries;
The main idea is that all that prepareForWrite
does is to bring all the temporary data into contiguous buffers that can then be written. However, since we have read the data into contiguous buffers we can skip this (almost completely). The one thing that we have to do is to copy all the Data
objects into one buffer again. Before #129 not even this would have been necessary and prepareForWrite
would have been a no-op.
Things get more tricky at the interplay between the EventStore
and the ROOTReader
and ROOTWriter
. When we are writing events then we are currently re-using the same collection objects internally and simply clear
them via EventStore::clearCollections
before we start filling them again in the next event. However, when we read events we get entirely new collection objects for every event and we have to dispose them via EventStore::clear
. The problem now is that the ROOTWriter
will never realize that his initially valid collection objects have become invalid after we have processed the first event. To fix this we would have to re-register all the collections that should be written again for each event to make sure that the ROOTWriter
has access to the correct (and valid) collection objects.
Also tagging #140 here to make sure that this is considered in the discussion there.
from podio.
Related Issues (20)
- Issue with clang format in the tests HOT 1
- Handle writing of datamodel definitions of dependent models properly
- Problems in the IO code HOT 2
- Generated cc files are deleted on cmake --target clear HOT 2
- Invalid files cause irrecoverable crashes in `ROOTReader::openFile(...)` HOT 2
- PODIO implicit plug-in mechanism is not robust enough
- type inconsistency between `Collection::size()` and index for const object accessors HOT 4
- Frames API to write collection metadata? HOT 2
- Add emplace_back to UserDataCollection
- ROOTFrameReader crash when reading multiple edm4hep files HOT 6
- New podio Tag without c++20 prohibition HOT 4
- Removal of deprecated EventStore and related components
- Mutability conversion fails on `push_back` to collection via Python bindings HOT 2
- Improve python interface when adding collections to a frame HOT 1
- In PODIO 00-16-05, `ROOTLegacyReader` can't read files written by 00-16-02 HOT 1
- `PODIO_JSON_OUTPUT` not set downstream, but json support still compiled into libraries HOT 5
- Misleading Error: Invalid Member Definition HOT 10
- Generated `id` method for user facing types has become useless
- Codify state machines of readers and writers
- Cleanup double `void*` in CollectionBuffers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from podio.