Panic `findNewEntrypoint called on empty hnsw graph` (HNSW delete cleanup logic) about weaviate

[glesperance]

@etiennedi Seeing this in one of my helm-release deployed cluster -- can you share more details about the recovery process?

{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24343) without a tombstone, tombstone was added","node_id":24343,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24346) without a tombstone, tombstone was added","node_id":24346,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24347) without a tombstone, tombstone was added","node_id":24347,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24353) without a tombstone, tombstone was added","node_id":24353,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24357) without a tombstone, tombstone was added","node_id":24357,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24358) without a tombstone, tombstone was added","node_id":24358,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24359) without a tombstone, tombstone was added","node_id":24359,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24362) without a tombstone, tombstone was added","node_id":24362,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24378) without a tombstone, tombstone was added","node_id":24378,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24379) without a tombstone, tombstone was added","node_id":24379,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24382) without a tombstone, tombstone was added","node_id":24382,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24385) without a tombstone, tombstone was added","node_id":24385,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24386) without a tombstone, tombstone was added","node_id":24386,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24387) without a tombstone, tombstone was added","node_id":24387,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24388) without a tombstone, tombstone was added","node_id":24388,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24389) without a tombstone, tombstone was added","node_id":24389,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24390) without a tombstone, tombstone was added","node_id":24390,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24391) without a tombstone, tombstone was added","node_id":24391,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24392) without a tombstone, tombstone was added","node_id":24392,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24394) without a tombstone, tombstone was added","node_id":24394,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24395) without a tombstone, tombstone was added","node_id":24395,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24398) without a tombstone, tombstone was added","node_id":24398,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24401) without a tombstone, tombstone was added","node_id":24401,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24402) without a tombstone, tombstone was added","node_id":24402,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24403) without a tombstone, tombstone was added","node_id":24403,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24404) without a tombstone, tombstone was added","node_id":24404,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (24405) without a tombstone, tombstone was added","node_id":24405,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (25775) without a tombstone, tombstone was added","node_id":25775,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (25779) without a tombstone, tombstone was added","node_id":25779,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (28241) without a tombstone, tombstone was added","node_id":28241,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (30758) without a tombstone, tombstone was added","node_id":30758,"time":"2024-04-11T12:48:41Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (31120) without a tombstone, tombstone was added","node_id":31120,"time":"2024-04-11T12:48:41Z"}
panic: class Chunk: shard zrbqtsoucgsj0c8u8qw172oi: findNewLocalEntrypoint called on an empty hnsw graph

goroutine 59 [running]:
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).findNewLocalEntrypoint(0xc0026e4000, {0x1db10f8, 0xc036dc8038}, 0x1, 0x5f1e)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/delete.go:569 +0x2de
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).pickEntrypoint(0xc002917aa8)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:277 +0xe9
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).doAtLevel(0xc002917aa8, 0x6?)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:73 +0x54
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*neighborFinderConnector).Do(0xc002917aa8)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:62 +0x47
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).findAndConnectNeighbors(0x0?, 0x0?, 0xc02949ac00?, {0xc02949ac00?, 0x300?, 0x427185?}, 0xe9586c?, 0x5e64a660?, {0x1db10f8, 0xc036dc8030})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/neighbor_connections.go:30 +0x99
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).addOne(0xc0026e4000, {0xc02949ac00?, 0x300, 0x300}, 0xc01688f180)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/insert.go:189 +0x81a
github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw.(*hnsw).AddBatch(0xc0026e4000, {0x1da6a18, 0xc02dda0230}, {0xc04b572800, 0x3e8, 0x0?}, {0xc028d28000, 0x3e8, 0x555})
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/insert.go:97 +0x585
github.com/weaviate/weaviate/adapters/repos/db.asyncWorker(0x0?, {0x1dbc990, 0xc0037cc380}, 0x0?)
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/repo.go:350 +0x58d
github.com/weaviate/weaviate/adapters/repos/db.New.func1()
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/repo.go:170 +0x6b
created by github.com/weaviate/weaviate/adapters/repos/db.New in goroutine 1
	/go/src/github.com/weaviate/weaviate/adapters/repos/db/repo.go:167 +0x79e

Update 2024-04-11: I updated to 1.24.8 and the node managed to recover — previously was at 1.23.7.
😅

[rthiiyer82]

Closing this issue as fixed as part of #4701