andreivisan / nutchelasticsearch

Hi, I am Andrei

I'm a Full Stack Developer and Tech Enthusiast!

• 🔭 I’m currently working on many projects!
• 🌱 I’m currently learning Machine Learning and Deep Learning
• 🥅 2023 Goals: Getting experienced with building Machine Learning projects
• ⚡ Fun fact: I love creating content to help people learn better and faster

Connect with me:

Languages and Tools:

nutchelasticsearch's Issues

Not generating elasticindex

I tried with this repository and followed all the steps. Still it is not generating any doc or index in ElasticSearch.

Here is the output (ES index name is nutch, cluster name is nutch)
...

LinkDb: merging with existing linkdb: TestCrawl2/linkdb
LinkDb: finished at 2014-04-14 18:46:47, elapsed: 00:00:02
Dedup on crawldb
Indexing 20140414184635 on ElasticSearch index -> nutch
Indexer: starting at 2014-04-14 18:46:52
Indexer: deleting gone documents: false
Indexer: URL filtering: false
Indexer: URL normalizing: false
Active IndexWriters :
ElasticIndexWriter
elastic.cluster : elastic prefix cluster
elastic.host : hostname
elastic.port : port
elastic.index : elastic index command
elastic.max.bulk.docs : elastic bulk index doc counts. (default 250)
elastic.max.bulk.size : elastic bulk index length. (default 2500500 ~2.5MB)

Indexer: finished at 2014-04-14 18:46:54, elapsed: 00:00:02
Mon Apr 14 18:46:55 PDT 2014 : Iteration 2 of 5
Generating a new segment
Generator: starting at 2014-04-14 18:46:56
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: false
Generator: normalizing: true
Generator: topN: 50000
-shouldFetch rejected 'https://nutch.apache.org/', fetchTime=1397611053313, curTime=1397526416071
Generator: 0 records selected for fetching, exiting ...

---

finally resolved the issue -- the elasticsearch port should be 9300 rather than 9200.
elasticsearch version should be 0.90.11 instead of 0.90.1 . You will have rebuild the nutch 1.8 after updating ivy.xml

nutch script still invokes solrindex rather than elasticindex

Hi Andrei,

thanks for creating this solution. It looks you have enhanced crawl and nutch scripts.
I reviewed these scripts. The nutch script has solrindex rather than elasticindex.

I followed your steps for nutch 2.2.1 and I am still facing the issue of no content in Elasticsearch.

Thanks
--Mahesh