Comments (13)
Tomorrow I'll try to optimize ndx-serializable data structures. I'll replace trie data structure for inverted index with two arrays: keys and values. gzip should be able to compress common prefixes for keys.
from ndx.
Published: ndx, ndx-index, ndx-query, ndx-compat and ndx-serializable.
from ndx.
I will not be able to try ndx with email-securely-app earlier than in a week or few, doing some refactoring and preparations for full-text feature enabling. Besides the initial implementation is not going to use serialization, but there is going to be such need.
from ndx.
This release https://github.com/vladimiry/email-securely-app/releases/tag/v2.2.0 uses recently published ndx
beta version for search. The app doesn't use the serialization yet though. Search has been working great so far. Will provide feedback in a few weeks and then close this issue.
from ndx.
Do you think it would be better if I move all code from ndx-index
into ndx
, since it is highly unlikely that there will be an alternative document indexing implementation and it is tightly coupled with the data structure that is used in ndx
?
Also, as you've noticed ndx-utils
is quite bad, so I want to move this functions to ndx-compat
and remove this package. In the documentation I'll just use lodash words()
function as a tokenizer.
from ndx.
Also, I want to change id
property in DocumentDetails
to doc
because in some use cases this indirection is useless and we can just store direct refs to document objects. It would also require small change in the addDocumentIndex()
function, id
argument will be replaced with an optional argument to specify custom key
for use cases that want to store some unique keys instead of direct refs.
from ndx.
Do you think it would be better if I move all code from ndx-index into ndx
It makes sense to me too.
I want to change id property in DocumentDetails to doc because in some use cases this indirection is useless and we can just store direct refs to document objects
Does it mean index will keep a reference to the original/full document
argument passed to addDocumentToIndex
? I'd like there will still be an option not storing a reference to a full document, as I don't want full documents to be kept in memory after they got indexed. I keep the original documents in one process and do indexing in another process I just take the search result ids from which and then prepare final search result in the first process. Communicating between processes is done using IPC scenario.
from ndx.
Does it mean index will keep a reference to the original/full document argument passed to
addDocumentToIndex
?
Only when id
isn't specified, I want to make it as a last optional argument in this function. Data structures will be the same, I'll just rename id
to doc
.
I'd like there will still be an option not storing a reference to a full document, as I don't want full documents to be kept in memory after they got indexed.
Yes, I understand that such use case is extremely important, I also using it in exactly the same way.
from ndx.
I want to make it as a last optional argument in this function
Just a note. If a function gets a lot of arguments, especially if some of them are optional, it might wort considering turning arguments to a single object argument as generally going with an object as an argument is a more flexible option and more self-documented.
from ndx.
I don't think that JIT in v8/others will inline this function and because of it escape analysis won't be able to figure out that it can remove this temporary allocation. Don't want additional memory allocation per each invocation since it is a low-level API.
from ndx.
Sure. Thanks for thorough analyzing and approach to building this library. One of the reasons I didn't enable serialization for above-mentioned program with initial search feature release is that indexing and search work quite fast.
from ndx.
Published in the beta
channel: ndx
, ndx-query
, ndx-compat
and ndx-serializable
.
Decided to keep it simple without any optional arguments, addDocumentToIndex()
API hasn't changed.
- Functions from
ndx-index
moved tondx
QueryResult
property namedocId
renamed tokey
.docId
implied that it should be some kind of an ID.
from ndx.
The desktop app mentioned above is working well with the following versions:
"ndx": "1.0.0-beta.2",
"ndx-query": "1.0.0-beta.2",
"ndx-utils": "1.0.0-beta.0",
I didn't notice a memory leaks. Closing the issue as resolved.
from ndx.
Related Issues (8)
- Search by ending HOT 2
- Recommended way to update an index? HOT 3
- [feature request] Rewrite "_vacuumIndex" in the way so it doesn't use a recursion technique HOT 2
- ability to search by field HOT 4
- add to uFuzzy benchmark HOT 3
- There might be a potential issue on this code line HOT 1
- Feature request: adding commons/shared/base project HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ndx.