Corrupted NP Array after many inserts into DB about pgvector-python

[ankane]

Hi @vade, I'm not sure what would cause this. Can you try getting the full statement from the Postgres server log?

[vade]

Hi @ankane - thanks for the prompt response. I'll poke around, forgive me, is there a specific method for fetching the logs that you prefer?

As an aside, I think I resolved the issue - however I don't thinks a 'fix', I'm just avoiding perhaps an edge case in some of the underlying code.

In our SQLAlchemy benchmarking code, we have some nested loops which create Python class objects associated with a SQLAlchemy class/schema definition - some of which have a PGVector associated.

We had a DB commit on one of the outer loops. By making the commit sooner in the inner loop, it seems that the issue has been resolved. My intuition is that we aren't creating lots of overhead in the SQLAlchemy code.

This leads me to believe some sort of memory compression or optimization, or underlying issue was occurring with LOTS of objects and LOTs of NP Arrays

Whats interesting is that if I adjust the loop count, I can re-create the issue even with the commit on the inner loops.

This leads me to believe I am unintentionally cross some working memory threshold on the SQLAclhemy ORM side of things.

Does this seem reasonable?

[ankane]

Re logs: Postgres should log errors (and the statements that cause them) in the server log by default. The relevant lines will look like:

ERROR:  invalid input syntax for type vector: ...
STATEMENT:  SELECT ...

Re memory threshold: I'm not aware of something like that. If you can create a single file script to reproduce, happy to look into it more.

[ankane]

Closing due to no response