How to handle heterogenous documents that lack expected fields about bson-numpy

[ajdavis]

Right now we're not handling varied documents. Each document must have all the fields you list in your dtype. If some of your documents have a "geo" subdocument and some don't, then pass this query string to "find":

find({"geo": {"$exists": True}})

But there might be other scenarios in which you want all the documents, but some documents are missing some fields. I'm curious, what behavior should BSON-NumPy have for this scenario? Should it return zeros for the missing values, or return a masked ndarray?

[alysivji]

Thanks for the tip. I will split up my queries for now.

Just read thru the above link and I like the idea of returning masked ndarrys for missing fields.

[prashantmital]

Hi @alysivji . I am closing this as we have discontinued development of BSON-NumPy. PyMongoArrow is now the recommended way to materialize MongoDB query results as NumPy ndarrays as well as tabular formats like Pandas' DataFrames and PyArrow Tables.

With PyMongoArrow you are no longer required to split your queries due to a difference in document structure. Given a specified schema, PyMongoArrow will attempt to load all documents from the query result set and will replace missing fields or fields containing a value of unexpected type with a Null/NaN which can be accounted for/handled by your application logic. See the documentation for an example - https://mongo-arrow.readthedocs.io/en/pymongoarrow-0.1.1/quickstart.html.