aligusnet / informationretrieval Goto Github PK
View Code? Open in Web Editor NEWInformation Retrieval
Information Retrieval
It takes a few hundred milliseconds to load document from a storage.
We should not load the whole collection to obtain one document.
Partially is enough to make tokenizer's job easier.
Operations to support:
E.g. (AND query moon (NOT (OR sun star)))
DocumentId is unique identifier of the document. DocumentId consists of 2 IDs:
DocumentId has some disadvantages:
it creates holes in a series of document ids if number of documents in collection is less then 2^16 (which is almost always the case)
it does not really saves the space, e.g. in case of RangeDocumentId introducing intermediate class for storing ranges in collection (collectionId, (localId, length)[])[] does not save in case of small ranges;
Using GridSplitter
Index that does not cache whole postings list in memory and but postings from disk as soon as it is required.
<RootProjectName>(\.[Test|App|WPF])?
Simple Parallel.For is enough
DictionaryIndex uses 2 structures to store index:
There are a couple of issues with that:
Both of the structs can be represented as 32-bit integers. We should use integer representations of the structs to avoid unnecessary allocations and optimize index structure on disk.
store DocumentIDs using ranges only when it makes sense (e.g. there are 5 documents id in a row).
To save time on reading/writing to disk.
Because, as it turns out it takes most of the time of every text preprocessing steps:
In general we do not need to store any results of the listed steps to disk so we can pass the preprocessed text to an indexer.
Support asynchronous input and output of document blocks.
DAWG is Directed Acyclic Word Graph. .NET implementation available there: DAWGSharp
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.