jorendorff / indexor

A toy search engine

indexor - A toy search engine!

It's not done yet! But it can do some things:

1. Download https://www.dropbox.com/s/lv44vyl8ia46llx/sample.tar.bz2?dl=0 and unzip it next to this directory. It creates a directory named sample (that should be a sibling directory of this repo) with about 8,476 files in it.

2. Run ./build_index.py to try to build the index.

   This creates a few big files in the parent directory. With our sample from step 1, they take up half a GB of disk space. (It would be pretty easy to cut that in half; haven't bothered yet.)

3. Run ./read_index.py to read raw entries out of the index.

   This is something less than a real search engine. You can only query one term at a time, and the output is bare-bones. But it works! With our sample, it takes a few seconds to start up; after that, queries are fast.

indexor's Issues

Support storing the index

multiple similar entries

This is more of a fug and not a bug. Entries that are the same but are followed by punctuation are treated as different entries. We are not stripping out punctuation at this point, so that is to be expected. currently if you search for something like 'penny' it will have different results than 'penny.', 'penny;', and 'penny,'

To resolve this issue:

• we need to decide what punctuation should be stripped and what should stay
• we need to find the best way to strip punctuation out

This shouldn't be very difficult to solve for.

results return only words starting with A or B

lamesearch runs without barking, but results returned are limited to those starting with A or B. Cause unknown at this point.

Searching for `superman` should find the article on Superman

> superman
Marvel Comics        1024.txt
London Symphony Orchestra 3593.txt
Broadcast syndication 4622.txt
Dwight Howard        5134.txt
Track cycling        1555.txt
Emblem               4627.txt
San Miguel Beermen   6690.txt
Tim Duncan           4137.txt
Hans Zimmer          3628.txt
2008 in film         6189.txt
> batman
Marvel Comics        1024.txt
The X-Files          1539.txt
Geelong              2563.txt
Sex Pistols          1541.txt
Adult Swim           4621.txt
Joss Whedon          3616.txt
Frank Welker         5153.txt
George Clooney       3622.txt
Andy Warhol          0040.txt
Nicole Kidman        1067.txt

These are super classy results but come on.

Support searching for multiple words

> batman suit
no hits

unacceptable.