fergiemcdowall / norch Goto Github PK
View Code? Open in Web Editor NEWA search server that can be installed with npm
A search server that can be installed with npm
Norch now has quite nice, yet undocumented, replication and snapshot functionality
It would be nice if the jQuery part of forage-document-processor was extracted out so that it could be easily edited and made interchangeable with other drop-in jQuery files. booming voice These files shall be known as "Adapters"
cannot install norch
C:\projects\sideLoad>node --version
v0.10.29
C:\projects\sideLoad>npm --version
1.4.14
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\request\node_
modules\hawk\node_modules\cryptiles\Makefile
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\reque
st\node_modules\hawk\node_modules\cryptiles\Makefile
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\request\node_modules\hawk\node_modules\boom\images\boom.png'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\request\node_
modules\hawk\node_modules\boom\images\boom.png
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\reque
st\node_modules\hawk\node_modules\boom\images\boom.png
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\request\node_modules\hawk\node_modules\hoek\images\hoek.png'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\request\node_
modules\hawk\node_modules\hoek\images\hoek.png
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\reque
st\node_modules\hawk\node_modules\hoek\images\hoek.png
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! error rolling back Error: EPERM, unlink 'C:\projects\sideLoad\node_modu
les\norch\node_modules\cheerio\node_modules\cheerio-select\node_modules\CSSselec
t\test\tools\bench.js'
npm ERR! error rolling back [email protected] { [Error: EPERM, unlink 'C:\projects\si
deLoad\node_modules\norch\node_modules\cheerio\node_modules\cheerio-select\node_
modules\CSSselect\test\tools\bench.js']
npm ERR! error rolling back errno: 50,
npm ERR! error rolling back code: 'EPERM',
npm ERR! error rolling back path: 'C:\projects\sideLoad\node_modules\norch
\node_modules\cheerio\node_modules\cheerio-select\node_modules\CSSselect
test\tools\bench.js' }
npm ERR! Error: ENOENT, open 'C:\projects\sideLoad\node_modules\norch\node_modul
es\request\node_modules\form-data\node_modules\combined-stream\node_modules\dela
yed-stream\package.json'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\request\node_
modules\form-data\node_modules\combined-stream\node_modules\delayed-stream\packa
ge.json
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\multer\node_modules\busboy\node_modules\readable-stream\node_modules\string_
decoder\README.md'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\multer\node_m
odules\busboy\node_modules\readable-stream\node_modules\string_decoder\README.md
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\multe
r\node_modules\busboy\node_modules\readable-stream\node_modules\string_decoder\R
EADME.md
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\cheerio\node_modules\htmlparser2\node_modules\readable-stream\lib_stream_wr
itable.js'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\cheerio\node_
modules\htmlparser2\node_modules\readable-stream\lib_stream_writable.js
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\cheer
io\node_modules\htmlparser2\node_modules\readable-stream\lib_stream_writable.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\multer\node_modules\busboy\node_modules\readable-stream\node_modules\inherit
s\LICENSE'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\multer\node_m
odules\busboy\node_modules\readable-stream\node_modules\inherits\LICENSE
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\multe
r\node_modules\busboy\node_modules\readable-stream\node_modules\inherits\LICENSE
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\cheerio\node_modules\cheerio-select\node_modules\CSSselect\test\tools\bench.
js'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\cheerio\node_
modules\cheerio-select\node_modules\CSSselect\test\tools\bench.js
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\cheer
io\node_modules\cheerio-select\node_modules\CSSselect\test\tools\bench.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\cheerio\node_modules\htmlparser2\node_modules\domhandler\tests\13-comment_in
_text.json'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\cheerio\node_
modules\htmlparser2\node_modules\domhandler\tests\13-comment_in_text.json
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\cheer
io\node_modules\htmlparser2\node_modules\domhandler\tests\13-comment_in_text.jso
n
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\multer\node_modules\busboy\node_modules\readable-stream\node_modules\core-ut
il-is\README.md'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\multer\node_m
odules\busboy\node_modules\readable-stream\node_modules\core-util-is\README.md
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\multe
r\node_modules\busboy\node_modules\readable-stream\node_modules\core-util-is\REA
DME.md
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\multer\node_modules\busboy\node_modules\dicer\test\fixtures\many-noend\part7
.header'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\multer\node_m
odules\busboy\node_modules\dicer\test\fixtures\many-noend\part7.header
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\multe
r\node_modules\busboy\node_modules\dicer\test\fixtures\many-noend\part7.header
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
npm ERR! Error: ENOENT, lstat 'C:\projects\sideLoad\node_modules\norch\node_modu
les\search-index\node_modules\natural\lib\natural\phonetics\phonetic.js'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues
npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "norch"
npm ERR! cwd C:\projects\sideLoad
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.14
npm ERR! path C:\projects\sideLoad\node_modules\norch\node_modules\search-index
node_modules\natural\lib\natural\phonetics\phonetic.js
npm ERR! fstream_path C:\projects\sideLoad\node_modules\norch\node_modules\searc
h-index\node_modules\natural\lib\natural\phonetics\phonetic.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack C:\Program Files\nodejs\node_modules\npm\node_modules\fst
ream\lib\writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (fs.js:107:15)
|
[email protected] install C:\projects\sideLoad\node_modules\norch\node_modules
search-index\node_modules\level\node_modules\leveldown
node-gyp rebuild
C:\projects\sideLoad\node_modules\norch\node_modules\search-index\node_modules\l
evel\node_modules\leveldown>node "C:\Program Files\nodejs\node_modules\npm\bin\n
ode-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild
gyp ERR! configure error
gyp ERR! stack Error: Python executable "python" is v3.4.2, which is not support
ed by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & <
3.0.0.
gyp ERR! stack at failPythonVersion (C:\Program Files\nodejs\node_modules\np
m\node_modules\node-gyp\lib\configure.js:108:14)
gyp ERR! stack at C:\Program Files\nodejs\node_modules\npm\node_modules\node
-gyp\lib\configure.js:97:9
gyp ERR! stack at ChildProcess.exithandler (child_process.js:645:7)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at maybeClose (child_process.js:755:16)
gyp ERR! stack at Socket. (child_process.js:968:11)
gyp ERR! stack at Socket.emit (events.js:95:17)
gyp ERR! stack at Pipe.close (net.js:465:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modu
les\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\projects\sideLoad\node_modules\norch\node_modules\search-index\n
ode_modules\level\node_modules\leveldown
gyp ERR! node -v v0.10.29
gyp ERR! node-gyp -v v0.13.1
gyp ERR! not ok
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! C:\projects\sideLoad\npm-debug.log
npm ERR! not ok code 0
Hi,
I am working on an alpha version of a website and we want to add a simple site search. A little googling led me to forage and it looks like it might work very well for us!
What I would like to do is to index our content (this is currently stored as a structured directory of markdown files). How might I go about indexing this into forage? Is it as simple as using the forage-document-processor command with a cusatom adaptor to do the conversion and then running the forage-indexer on the resulting files?
Any guidance would be greatly appreciated.
David
Hello,
I'm very interested in your projet. Really good work. Thank you.
A question : I don't see documents ? Only filters and facets. What's wrong ?
Facets could be more "visual" : chart, pie, map, tree, ...
JSON documents could be stored on noSql database before indexation. It would be nice to index CouchDB rather than a directory.
Crawling could be triggered directly from the browser that view the page you want to index. (kind of "bookmark")
Philppe.
For the test dataset:
"New" works
"York" works
"Yorksdad" works (and returns no results, since it is absent from the index)
"New York" works
"New Yorksdad" results in a catastrophic crash
We need a new logo since Forage is dead and Norch is back. Dawn of the dead!
At the moment any machine with access to the Forage end-point can add documents to the index. Any end-point that can process a change to the index must be placed under access control.
Make it possible to have different types of matcher
Only run URL from localhost
In the search function the $.getJSON("http://localhost/search?"... should be change to
$.getJSON("/search?...
Now it only works on the development machine/server.
Espen
Make it possible to delete a single document from the index along with all associated references
A small error in the documentation? Got error on singlequotes in JSON, it should be double-quotes, shouldn't it?
[
{
'id':'1',
'title':'A really interesting document',
'body':'This is a really interesting document',
'metadata':['red', 'potato']
},
{
'id':'2',
'title':'Another interesting document',
'body':'This is another really interesting document that is a bit different',
'metadata':['yellow', 'potato']
}
]
Could we implement some way of indexing a document without specifying the id?
That is, could we index this:
{
'title':'A really interesting document',
'body':'This is a really interesting document',
'metadata':['red', 'potato']
},
and autoassign it some id?
It it said that norch is under a MIT license, and a link to https://github.com/fergiemcdowall/norch/blob/master/LICENSE, but there is no license in that location. Also, there is no license file provided in the repo.
If not, any recommendations?
While running
sudo npm install -g norch
on ubuntu trusty
[email protected] install /usr/local/lib/node_modules/norch/node_modules/search-index/node_modules/level/node_modules/leveldown
node-gyp rebuild
npm http 304 https://registry.npmjs.org/apparatus
/bin/sh: 1: node: not found
gyp: Call to 'node -e "require('nan')"' returned exit status 127. while trying to load binding.gyp
I've always wanted a better bookmark search tool. I have hundreds of them and I hate labeling them with searchable tags. I want to search their metadata!
This looks like the perfect project to fork into a simple command line app.
Basically I just want to index my bookmarks (given some format: Safari, Chrome, Firefox, JSON, etc.) and then be able to search them. It would also be nice to save a snapshot. It will also be super badass to do this all through the command line too!
Anyways, I'm curious if anyone here wants to help me with this since I'm not terribly familiar with all the work that has been done here. I hope someone will agree that this is a pretty awesome tool. This is a very simple extension of what has already been implemented here.
I'd appreciate any help or any suggestions on how to get started.
Thanks
I would suggest moving the default port from 3000 as this is the port that expressjs. uses. This could cause a lot of unnecessary confusion.
Pigs snout, sniffing - as the o in Forage.
https://www.flickr.com/photos/eklem/13776125753/
Something similar to ElasticSearch (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-get.html), it is possible we store/retrieve JSON documents Forage?
Says homepage is forage.net
/getDoc?docID=0
doesnt seem to be working at the moment
This looks to probably be most prevalent in natural module, but other closures seem to be left hanging when indexing.
Will hopefully have at least a partial solution shortly for pull.
When running forage using path/to/forage.js directories are not treated as relative (i.e path.join(__dirname, 'xxxx'); )
This makes running forage via say forever (init.d) needing cd commands in script etc.
I have corrected code, and will submit pull shortly.
First of all, great work on Forage!
Second - it would be great if it supported partial queries, since this would ensure search-as-you-type is possible.
What do you think?
What about this - fergiemcdowall/search-index#3 ? Does "closed" mean that it's implemented?
Is there any plan to support chinese language for forage?
Vagrant wont play nice on OSX- see error below...
bash-3.2$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] -- 3000 => 3000 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] Mounting shared folders...
[default] -- /vagrant
[default] -- /tmp/vagrant-puppet/manifests
[default] -- /tmp/vagrant-puppet/modules-0
[default] Running provisioner: puppet...
Running Puppet with default.pp...
stdin: is not a tty
Could not find class apt for precise64.lyse.net at /tmp/vagrant-puppet/manifests/default.pp:10 on node precise64.lyse.net
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
puppet apply --modulepath '/etc/puppet/modules:/tmp/vagrant-puppet/modules-0' --detailed-exitcodes /tmp/vagrant-puppet/manifests/default.pp || [ $? -eq 2 ]
Stdout from the command:
Stderr from the command:
stdin: is not a tty
Could not find class apt for precise64.lyse.net at /tmp/vagrant-puppet/manifests/default.pp:10 on node precise64.lyse.net
bash-3.2$
Norch really needs a proper test suite. I like the look of this: http://thewayofcode.wordpress.com/2013/04/21/how-to-build-and-test-rest-api-with-nodejs-express-mocha/
Fergie, is there a specific reason (possibly performance on fetch) that fields are replicated across each 'field' key storage element.
I just created a 20,000 doc index (20 fields/doc) 100mb total, and have a data space of 9gb.
Still getting my head around index code, but it seems a this could be optimized away using a field cache within leveldb. Keyspace obviously cannot change, but valuespace could be tweaked at expense of second level field fetch as results are returned (possibly even streamed via a transform containing keys).
url from readme.
Was considering integrating into Etherpad, doable?
Yesterday, I tested forage-fetch and downloaded a lot of documents, but I had to remove the --starturl [url], or else I only ended up with one document. I can re-test the issue today.
Dosregard. Found a bug in indexer, will patch it today.
Save batch files on the search server so that they can be re-fed at will
It would be good to have some kind of crawler admin functionality acessable from the web, and with a content processing pipeline
Index up reuters-004 like so "curl --form [email protected] http://localhost:3030/indexer --form filterOn=metadata"
Search for 'negotiation': 3 reasonable hits come up
Try to refine by 'negotiation banks': no hits, when clearly there should be at least two?
At the moment, malformed JSON crashes the system. Make it such that a malformed JSON returns a pretty error, whilst keeping the system up
"Server not found"
Got an idea last week to use the browser as a virtual machine for Forage. Forage could then be added with a bookmarklet to any page. With some simple UI-stuff you could define the Forage Document Processor Adapter, set up rules for Forage Crawler, crawl (and process) and then possibly test-search with HTML5 local storage instead of levelDB. When the user is satisfied she or he could download the JSON-file with processed items + scripts for adding a search box, search result and navigators to a page.
The real benefit would be that the user would not need any server to test Forage and actually crawl a site. When page crawled the user can download JSON + setup-stuff or add it to a cloud service.
The user finds a page to crawl ...
... clicks the bookmarklet ...
... that adds Forage JavaScript-stuff to the page ...
... much like a browser plugin ...
... tests a jQuery selector satement ...
... and adds the field to the item when satisfied. This process is repeated until a full item is defined.
More to come on the crawl rule setup, test-search and downloading of data + adapter + crawl-rules
Utility function to get a specific document directly via ID without using search().
Useful for launching say a display.html page.
Will submit pull shortly
There are reports of documents disappearing when two or more batches are indexed concurrently (thanks @praveev)
The solution to this is probably to implement queuing functionality in Norch with https://github.com/learnboost/kue or similar.
Seems there's only symlinks from /usr/local/bin to forage (which is in /usr/local/lib/node_modules/forage/bin/ ).
Add symlinks for these also?
Would be nice to simply write a JSON inline and post it.
A similar idea to the search engine as a bookmarklet. #32 But since the bookmarklet thingy is not the first to happen, it's good to describe some of the ideas more likely to happen.
If the "search engine running in your browser" works, how about the index, and maybe the search engine it self, as torrents/webtorrents. Then, the one making a webapp or webservice will almost need no server/hardware power at all. You only need a seeder of the app/search engine and search index.
This is a big job, but it would be really nice with documentation matching at least one sample set of data. That means it's much easier to test all the functionality and figure out how stuff is working.
Stuff I think forage-fetch needs to handle:
These are thoughts based on earlier use of Fast ESP and Scrapy, so the way they solve it may not be the correct answer, but the outcome of it is needed, I think.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.