Understanding HNSW and IVFFLAT index creation and storage about pgvector

[jkatz]

First, I'd recommend running with v0.6.2 (or v0.7.0) as v0.6.2 has changes that impact index build performance that will be noticeable in what you're testing.

The index build occurs in the portion of maintenance_work_mem that's allocated -- at the point of the index build shared_buffers is used for pages already persisted the disk (i.e. heap pages).

If the data used for building the index exceeds maintenance_work_mem, the "excess" data is written to temporary storage, which is wherever you've defined where your PostgreSQL temporary storage goes.

Once the index is built, pgvector allocates the pages for flushing to disk and handles that portion in bulk. This is when the new index pages make their way into the shared buffer pool.

My hunch is that the second case makes more sense since there's a possibility to lose index if database restarts.

I'm not following this -- assuming this is the "builk build" scenario, if your database restarts while your index is still building, you have to start the index build again as the transaction never finished.

[ankane]

Hi @taimur1991, someone could come up with a formula for estimating the index size based on the # of rows, # of dimensions, data type, index type, and index options (see #545 for the layout / storage of HNSW index tuples). You'd need to take into account the Postgres block size / page layout as well.

[taimur1991]

Thanks @jkatz, that clears the concept. I'd really appreciate your comment on the size of index with more recent version of pgvector as I have gone through your study here with pgvector 0.5.0. Your study shows that index size depends on the configuration of hnsw and type of dataset used. Similar analysis was concluded by this study as well although this study does not provide the pgvector version. Could it be possible to estimate the size of index from the table or from any relation? For instance, this answer shows the estimation of index memory with the help of a relation proposed by FAISS. Can similar relation be made for index size as well?