manabuo / gutengrep

Find whole sentences matching a regex in Project Gutenberg

gutengrep

Find whole sentences matching a regex in Project Gutenberg plain text files.

Example commands

gutengrep.py "^[^\w]*And then" "*.txt" --cache --sort --correct -o output/and-then.txt

gutengrep.py "^[^\w]*But why" "*.txt" --cache --sort --correct -o output/but-why.txt

gutengrep.py -i "whale" moby11.txt --sort --correct -o out\mobydick-whale.txt

Example output

Name	Sorted	Regex	Input	Word count
But why?	But why?	^[^\w]*But why	*.txt	7,572
And then!	And then!	[^\w]*And then	*.txt	85,014
The whale	The whale	whale	moby11.txt	50,913
Why	Why	[^\w]*Why	*.txt	184,832
Once upon a time	Once upon a time	-i once upon a time	*.txt	6,195
The End	The End	-i the end\.	*.txt	142,94
Happily ever after	Happily ever after	-i happily ever after	*.txt	271
Moonlit	Moonlit	-i moonlit	*.txt	52,345
Moonlight	Moonlight	-i moonlight	*.txt	3,186

See also nanogenmo.md.

Tips

Download the Project Gutenberg August 2003 CD and put all the files in the same directory.

When working on the whole corpus, use --cache to cut down on file operations. The first time it will build a cache file of all tokenised sentences. This first pass takes about 5 minutes on my MBP to go through the 597 books of the Project Gutenberg CD and extract its 3,583,390 sentences. Subsequent runs using the cache take about 40 seconds.

If searching just a single file, or a subset of files, make sure not to use --cache because it will use the cache file generated on the initial file spec.