Comments (6)
Hey @yjhjstz, existing pages with free space are reused (pages aren't deleted) - see InsertTuple
function.
from pgvector.
yes, but if marked insertPage have N pages, How to manage N free pages ?
from pgvector.
insertPage
is only a single page. InsertTuple
will start at insertPage
and follow nextblkno
until it finds a page with free space.
from pgvector.
I see, but to find the free page to insert must seq scan the list , not efficient.
refer to contrib/bloom
/* Metadata of bloom index */
typedef struct BloomMetaPageData
{
uint32 magickNumber;
uint16 nStart;
uint16 nEnd;
BloomOptions opts;
FreeBlockNumberArray notFullPage;
} BloomMetaPageData;
is that ok ?
from pgvector.
There are more efficient ways, but I'd like to keep it simple for now. I don't think it'll be an issue for typical usage patterns (in the worst case scenario, it'll end up scanning the list once after a vacuum). Tables and some index types use a free space map, but pgvector needs one for each list (rather than one for the entire index, which isn't supported). Using an array like Bloom indexes would probably work, but it'll add more complexity.
from pgvector.
Hey @yjhjstz, just fyi, found a bug in the vacuum code that caused indexes to not reuse space at all. There's a fix in 0.2.4 (and a test case to prevent regressions - it still uses a single insertPage
, but there was a logic error).
You can shrink existing indexes with:
-- Postgres 12+
REINDEX INDEX CONCURRENTLY index_name;
-- Postgres < 12
CREATE INDEX CONCURRENTLY temp_name ON table USING ivfflat (column opclass);
DROP INDEX CONCURRENTLY index_name;
ALTER INDEX temp_name RENAME TO index_name;
from pgvector.
Related Issues (20)
- Follow up #425 - pgvector issues with postgresql 16.1 on macOS Sonoma HOT 1
- Limit HNSW build's shared memory size for small tables HOT 7
- crtdefs.h not found issue while running command nmake /F Makefile.win HOT 1
- text-embedding-3-small not working HOT 2
- Uninitialized entryPoint->level HOT 2
- Valgrind complains about uninitialized HnswCandidate.closer HOT 5
- OpenAI's text-embedding-3-large model not compatible with pgvector 5.1 HOT 8
- How can Use Max vCPUs at HNSW? HOT 1
- pgvector doesn't get installed if the required LIBS are not in same directory as in PGROOT argument HOT 1
- Clarify build instructions for Windows (the missing postgres.h header) HOT 2
- Publish v0.6.0 to Postgres apt repo HOT 3
- Vacuums extremely slow for HNSW indices? HOT 13
- vector_eq raises ERROR instead of returning FALSE. Is that intended behavior? HOT 7
- Install in windows not found vcvars64.bat HOT 2
- Problem creating HNSW index after upgrading to 0.6.0 HOT 2
- Does the vector query with distance return result in sorted order? HOT 2
- Performance Issue with Large Tables and HNSW Indexes HOT 9
- Mac M1/M2 arm Arch issue installation HOT 2
- Normalize vectors HOT 1
- Issue with Installing pgvector Extension on PostgreSQL 14 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgvector.