Should we use Bleve for our search? about photoprism

[lastzero]

Thank you for the hint! The only issue I see is concurrent requests and backups. If you use standard SQL technology, both is not an issue. How would you handle read/write requests from concurrent processes with an embedded store? Probably locking and only run a single instance? For example when the Web frontend is running and the user wants to index new photos on the command-line.

[xeoncross]

Yes, depending on which key/value store, large online-backup runs could lock the table for writes - but it generally isn't a problem. Don't let the "key/value" store throw you. Boltdb / LevelDB are vary powerful and both take different approaches to table locking / concurrent use.

LevelDB is a LSM tree so better for loads >10,000 writes/sec, while Bolt uses a B+tree so is optimized for reads. Either of them could easily handle millions of images.

https://github.com/boltdb/bolt#comparison-with-other-databases

http://blevesearch.com/bleve-bench/ can index about 1MB of text/attributes a second and search about 120 queries a second (both only using a single-core)

[lastzero]

Nice, we'll give it a try :) Although I can imagine some users/admins would like to access their data in MySQL, e.g. from a CMS.

[lastzero]

We've recently switched to TiDB as our default database, see #60. Is this enough to implement our search or do we need something else like Bleve? While Bleve looks nice, it seems to be based on the unmaintained BoltDB.

• Does Bleve also properly support LevelDB?
• How to keep indices of TiDB and Bleve in sync without much effort?
• Does TiDB have more features for search than MySQL?
• What would be the specific disadvantage of just using TiDB for now?

All of this needs to be investigated. Help is most welcome! 👍

[lastzero]

Implemented full-text search with SQL only, even without support for FULLTEXT. So this is not needed anymore.