graphaware / graph-aided-search Goto Github PK

View Code? Open in Web Editor NEW

154.0 154.0 42.0 874 KB

Elasticsearch plugin offering Neo4j integration for Personalized Search

Java 100.00%

elasticsearch elasticsearch-plugin java neo4j

graph-aided-search's People

Contributors

Stargazers

Watchers

Forkers

ikwattro longlongtech javamuller rucky2013 cuulee beifeizhou semanticbeeng daviddyball jiangwm louiekang hailiang-wang colinsongf olympicsoftware iskenderunlu guptam xycloud alperhankendi irkasper zhuzhengyi mikalv simonzhao777 calvinalvin afcarl jiyulongxu adrianprecub quanpinjie bigdataedison mbrukman union2find diogenescs sdiazben tiagoooliveira hipparic g2064 t1seungy kyhoolee luoluoluoxl iitians michaelcizmar jiangbin216 smithimage iq-scm

graph-aided-search's Issues

Does it work with embedded Neo4j?

I meet "Neo.ClientError.Security.Unauthorized",when i integrated neo4j3.1.3 and ES2.2.2

sent request "curl -X POST http://localhost:9200/neo4j-index/Movie/_search -d '{
"query" : {
"bool": {
"should": [{"match": {"title": "love"}}]
}
},
"gas-filter" :{
"name": "SearchResultCypherFilter",
"query": "MATCH (n:User)-[r:LIKES]->(m) WITH m, avg(r.rate) as avg_rate where avg_rate < 3 RETURN m.objectId as id",
"exclude": true,
"keyProperty": "objectId"
}
}'"

get an error : "{"error":{"root_cause":[{"type":"runtime_exception","reason":"Failed to parse a search response."}],"type":"runtime_exception","reason":"Failed to parse a search response.","caused_by":{"type":"runtime_exception","reason":"Cypher Execution Error, message is : {code=Neo.ClientError.Security.Unauthorized, message=No authentication header supplied.}"}},"status":500}"

first,i wouldn't setting dbms.security.auth_enabled=false,
help me!!!!

Elasticsearch version 6.X

Is there any way to use it with Elasticsearch version 6.x? I tried to install the Elasticsearch plugin in Elasticsearch version 6.X and failed.

Graph-Aided Search Result Filtering

I've duplicated the Movie database of Neo4j on Elasticsearch and it's indexed with the index nodes. It has two types Movie and Person. I am trying to make a simple Result Filtering with Graph-Aided Search using this curl command line:

curl -X GET localhost:9200/nodes/_search?pretty -d '{
"query": {
   "match_all" : {}
 },
 "gas-filter": {
         "name": "SearchResultCypherfilter",
         "query": "MATCH (p:Person)-[ACTED_IN]->(m:Movie) WHERE p.name= 'Parker Posey' RETURN  m.uuid as id",
         "shouldExclude": true,
         "protocol": "bolt"
   }
}'

But I get as results all the 171 nodes of both types Movie and Person in my index nodes. However, as my query says I want to return only the type Movie by its uuid. So basically it doesn't look to the gas-filter part.

Also when I put false as the value of shouldExclude I am getting the same results.

I am working with:

Elasticsearch 2.3.2
graph-aided-search-2.3.2.0
Neo4j-community 2.3.2.10
graphaware-uuid-2.3.2.37.7
graphaware-server-community-all-2.3.2.37
graphaware-neo4j-to-elasticsearch-2.3.2.37.1

Result that should be returning:

The uuid of the movie titled You've Got Mail.

I tried to follow this tutorial for the configuration, and I found out that index.gas.enable had the value false so I changed it and finished the configuration just like in the tutorial:

mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.hostname=http://localhost:7474
{"acknowledged":true}

mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.enable=true
{"acknowledged":true}

mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.user=neo4j
{"acknowledged":true}

mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.password=mypassword
{"acknowledged":true}

After that I tried to add the settings of boltHostname and bolt.secure but it didn't work and I had this error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"},"status":400}

So I closed my index to configure it and then opened it again:

mac$ curl -XPOST http://localhost:9200/nodes/_close
{"acknowledged":true}

mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.boltHostname=bolt://localhost:7687
{"acknowledged":true}

mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.bolt.secure=false
{"acknowledged":true}

mac$ curl -XPOST http://localhost:9200/nodes/_open
{"acknowledged":true}

After finishing the configuration I tried again on Postman the same gas-filter query that I was executing with curl and now I am getting this error:

{
  "error": {
    "root_cause": [
      {
        "type": "runtime_exception",
        "reason": "Failed to parse a search response."
      }
    ],
    "type": "runtime_exception",
    "reason": "Failed to parse a search response.",
    "caused_by": {
      "type": "client_handler_exception",
      "reason": "java.net.ConnectException: Connection refused (Connection refused)",
      "caused_by": {
        "type": "connect_exception",
        "reason": "Connection refused (Connection refused)"
      }
    }
  },
  "status": 500
}

I don't know which connection the error is talking about. I am sure I passed the correct password of Neo4j in the configuration. I've even stopped and restarted again the servers of Elasticsearch and Neo4j but still the same errors.

Any ideas?

Integration into Elastic v5.X?

Does this work with version 5?

Any roadblocks or issues?

Integrating neo4j 3.0.3 and es 2.3.2

Hi:

I tried following the steps outlined here.

http://graphaware.com/neo4j/2016/04/20/graph-aided-search-the-rise-of-personalised-content.html

The website uses older neo4j version which is 2.x,, so I replaced all downloads with neo4j 3.x

Now, I'm currently at the step where we are to Configure Graph-Aided Search by defining the URL of the Neo4j server instance and enabling it.

When we execute the following command:

$ curl -XPUT "http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.hostname=http://localhost:7474&index.gas.enable=true"

we got the following error:

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"neo4j-index","index":"neo4j-index"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"neo4j-index","index":"neo4j-index"},"status":404}

Knowledge graph approach

Is it possible to query neo4j first and get relevant views, which can be used to get the appropriate documents and search via elasticsearch?

Multiple filters

It is possible to use multiple filters in one query to the Elastic?

Multi-index query with the plugin

Is the gas-filter should support a multi-type query?
something like GET index1,index2/_search
?

unknown search element [gas-booster]

EDIT: I just find the error after few hours of research, an old instance from an other version of elasticsearch was still running in background. I close this mistake. Thanks for your work here ;)

Hello,

I get this error using the node.js official plugin of elasticsearch with neo4j.

Elasticsearch TRACE: 2016-11-02T22:26:57Z
  -> POST http://localhost:9200/neo4j-index/School%2CEmployer%2CUserFacebook%2CUserEmail%2CPlace%2CCompanyDomain/_search
  {
    "query": {
      "query_string": {
        "fields": [
          "facebookName^10",
          "goodGmailName^7",
          "gmailEmail^10",
          "country^15",
          "city^15",
          "schoolName^30",
          "employerName^20",
          "placeName^15",
          "zip^5",
          "street^7",
          "domain^5"
        ],
        "query": "allan",
        "use_dis_max": true
      }
    },
    "gas-booster": {
      "name": "SearchResultCypherBooster",
      "query": "MATCH (n) WHERE n:UserEmail or n:UserFacebook WITH n AS n, (CASE WHEN exists(n.facebookEmail) or exists((n)-[:STUDIED_IN]->()) or exists(n.hometown) or size((n)-[]->())>30 THEN 1 else 0 end) as score RETURN n.uuid AS uuid, score",
      "identifier": "uuid",
      "scoreName": "score",
      "operator": "+"
    },
    "gas-filter": {
      "name": "SearchResultCypherFilter",
      "query": "MATCH (n:UserEmail) WITH n MATCH (n)<-[:HAS_ACCOUNT]-(father) WITH n, father WHERE exists((father)<-[:KNOWS]-()) RETURN n.uuid AS id UNION MATCH (n:UserFacebook) WITH n MATCH (n)<-[:HAS_ACCOUNT]-(father) WITH n, father WHERE exists((father)<-[:KNOWS]-()) RETURN n.uuid AS id"
    }
  }
  <- 400
  {
    "error": {
      "root_cause": [
        {
          "type": "search_parse_exception",
          "reason": "failed to parse search source. unknown search element [gas-booster]",
          "line": 1,
          "col": 243
        }
      ],
      "type": "search_phase_execution_exception",
      "reason": "all shards failed",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
        {
          "shard": 0,
          "index": "neo4j-index",
          "node": "TafSfQzVT1CTuW2gRv7Acw",
          "reason": {
            "type": "search_parse_exception",
            "reason": "failed to parse search source. unknown search element [gas-booster]",
            "line": 1,
            "col": 243
          }
        }
      ]
    },
    "status": 400
  }

Elasticsearch DEBUG: 2016-11-02T22:26:57Z
  Request complete

Trace: [search_parse_exception] failed to parse search source. unknown search element [gas-booster], with { line=1 col=243 }
    at /Users/bondot_e/Workspace/shortouch-api/app/controllers/users/users.controller.js:65:11
    at tryCallOne (/Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/promise/lib/core.js:37:12)
    at /Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/promise/lib/core.js:123:15
    at flush (/Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/asap/raw.js:50:29)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

What's giving me a headache is that I'm currently working locally. But if I request elasticsearch hosted on my distant server everything is working fine...

I used elasticsearch v2.3.2 :
Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0_60

on my distant server :
Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0_91

and I tried with graph-aid-search v2.3.2.0 (the version which is working fine on my distant server), v.2.3.2.2 (recommended on the README) and the last release v2.4.1.3 without any success.

elasticsearch logs:

[2016-11-02 23:16:02,826][INFO ][node                     ] [Dirtnap] version[2.3.2], pid[30136], build[b9e4a6a/2016-04-21T16:03:47Z]
[2016-11-02 23:16:02,827][INFO ][node                     ] [Dirtnap] initializing ...
[2016-11-02 23:16:03,544][INFO ][plugins                  ] [Dirtnap] modules [reindex, lang-expression, lang-groovy], plugins [graph-aided-search], sites []
[2016-11-02 23:16:03,596][INFO ][env                      ] [Dirtnap] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [32gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-11-02 23:16:03,597][INFO ][env                      ] [Dirtnap] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-11-02 23:16:03,598][WARN ][env                      ] [Dirtnap] max file descriptors [10240] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-11-02 23:16:06,129][INFO ][node                     ] [Dirtnap] initialized
[2016-11-02 23:16:06,130][INFO ][node                     ] [Dirtnap] starting ...
[2016-11-02 23:16:06,227][INFO ][transport                ] [Dirtnap] publish_address {127.0.0.1:9301}, bound_addresses {[fe80::1]:9301}, {[::1]:9301}, {127.0.0.1:9301}
[2016-11-02 23:16:06,231][INFO ][discovery                ] [Dirtnap] elasticsearch/F78p3vSDST-83RfYIVaG5Q
[2016-11-02 23:16:09,274][INFO ][cluster.service          ] [Dirtnap] new_master {Dirtnap}{F78p3vSDST-83RfYIVaG5Q}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-11-02 23:16:09,315][INFO ][http                     ] [Dirtnap] publish_address {127.0.0.1:9201}, bound_addresses {[fe80::1]:9201}, {[::1]:9201}, {127.0.0.1:9201}
[2016-11-02 23:16:09,315][INFO ][node                     ] [Dirtnap] started
[2016-11-02 23:16:09,334][INFO ][gateway                  ] [Dirtnap] recovered [0] indices into cluster_state

So graph-aided-search is apparently successfully loaded.

I think I am following well my setup:

curl -XPOST 'localhost:9200/neo4j-index/_close'

curl -XPUT 'http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.hostname=http://localhost:7474&index.gas.enable=true'
curl -XPUT 'http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.user=neo4j&index.gas.neo4j.password=xxx'

curl -X PUT 'localhost:9200/neo4j-index/_settings' -d '{
"analysis": {
"filter": {
"autocomplete_filter": {
"type":     "edge_ngram",
"min_gram": 4,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type":      "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter",
"asciifolding"
]
}
}
}
}'

curl -XPOST 'localhost:9200/neo4j-index/_open'

And then I performed my mapping.

Any suggestion ?

No new recod in Elastic

Hello, i tried run module, but i have not seen any changes in elastic.
What i did:

I have windows and i put jar into

~\Documents\Neo4j\default.graphdb\plugins

Change config - add to row

elasticsearch.host_name=http://localhost:9200
elasticsearch.index_spec=people:Person(name).

My elastic used this default port .

Than run elastic and neo4j.
I used

CREATE (ee:Person {name: "Emil"})

used

MATCH (ee:Person) WHERE ee.name = "Emil" SET ee.name= ee.name

After that i called

curl -XGET localhost:9200/* and get nothing

Why my command

MATCH (ee:Person) WHERE ee.name = "Emil" SET ee.name= ee.name cannot create ne record in elastic

Configuration:

Windows 10
elastic 5.6
Neo4j 3.2.5

Elastic v2.4.6 support and release

Will there be a future release to support Elastic v2.4.6 as there are bug fixes to ES since v2.4.4? Also, the readme says v2.4.4 is supported but there is no release for that version. Will this be available soon?

Compatibility with neo4j 3.x

Hi, is this elastic search plugin compatible to communicate to neo4j 3.0.x?

ids parameters not present for SearchResultCypherFilter

Currently the SearchResultCypherBooster class is cognizant of the node ids which are to be boosted which significantly improves performance as one can leverage indexes, i.e., if one wanted to rank the search results by most recently created,

curl -X GET localhost:3988/nodes/_search?pretty -d '{
    "query": {
        "query_string": {
            "query": "foo"
        }
    },
    "gas-booster": {
        "name": "SearchResultCypherBooster",
        "query": "MATCH ()-[r:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id, r.timestamp AS score",
        "operator": "replace",
        "protocol": "bolt"
    }
}'

however for the SearchResultCypherFilter class it is not, i.e.,

curl -X GET localhost:3988/nodes/_search?pretty -d '{
    "query": {
        "query_string": {
            "query": "foo"
        }
    },
    "gas-filter": {
        "name": "SearchResultCypherFilter",
         "query": "MATCH ()-[:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id",
         "exclude": false,
         "protocol": "bolt"
    }
}'
{
  "error" : {
    "root_cause" : [ {
      "type" : "runtime_exception",
      "reason" : "Failed to parse a search response."
    } ],
    "type" : "runtime_exception",
    "reason" : "Failed to parse a search response.",
    "caused_by" : {
      "type" : "runtime_exception",
      "reason" : "Cypher Execution Error, message is : {code=Neo.ClientError.Statement.ParameterMissing, message=Expected a parameter named ids}"
    }
  },
  "status" : 500
}

I presume that the node ids would be present as the gas-filter condition should augment the same result set returned from ElasticSearch.

Removing the WHERE clause works, however performance is somewhat suboptimal, and about 10 times slower than an equivalent boost query i.e.,

time curl -X GET localhost:3988/nodes/_search?pretty -d '{
    "query": {
        "query_string": {
            "query": "foo"
        }
    }
}'

real	0m0.016s
user	0m0.003s
sys	0m0.003s

time curl -X GET localhost:3988/nodes/_search?pretty -d '{
    "query": {
        "query_string": {
            "query": "foo"
        }
    },
    "gas-booster": {
        "name": "SearchResultCypherBooster",
        "query": "MATCH ()-[r:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id, r.timestamp AS score",
        "operator": "replace",
        "protocol": "bolt"
    }
}'

real	0m0.104s
user	0m0.006s
sys	0m0.000s

time curl -X GET localhost:3988/nodes/_search?pretty -d '{
    "query": {
        "query_string": {
            "query": "foo"
        }
    },
    "gas-filter": {
        "name": "SearchResultCypherFilter",
         "query": "MATCH ()-[:CREATED]->(n:Entity) RETURN n.uuid AS id",
         "exclude": false,
         "protocol": "bolt"
    }
}'

real	0m1.148s
user	0m0.006s
sys	0m0.000s

Is this framework compatible with Elasticsearch 5.0 and Neo4J 3.1.1?

SearchResultCypherFilter Filtering Everything on Very Simple Query

Hi,

We're currently giving GAS ago for filtering search queries based on information returned from the graph. It's all very simple, so I'm not sure why it's not working. If I add "exclude": true to the query it does return results that don't match, like I would expect. As crazy as it sounds, if I run the graph query manually and run the search without the gas-filter clause, I can see the documents returned do contain the ids of the documents that are returned by the Cypher query. I'm stumped. Any help you could provide would be appreciated.

Graph Layout

(:Product {id: <int>})-[:SIMILAR_TO {weight: <int>}]->(other:Product)

Search Mapping

GET /products/product/_mapping
{
  "products" : {
    "mappings" : {
      "product" : {
        "properties" : {
          "id" : {
            "type" : "integer"
          }
        }
      }
    }
  }
}

Result when running the Cypher query directly

> MATCH (:Product {id: 1})-[r:SIMILAR_TO]->(other) RETURN other.id as id ORDER BY r.weight DESC
╒═══╕
│id │
╞═══╡
│ 5 │
├───┤
│ 2 │
├───┤
│ 4 │
├───┤
│ 3 │
└───┘

The GAS query that returns nothing:

{
  "query": {
    "match_all": {}
  },
  "gas-filter": {
    "query": "MATCH (:Product {id: 1})-[r:SIMILAR_TO]->(other) RETURN other.id as id ORDER BY r.weight DESC",
    "protocol": "bolt",
    "name": "SearchResultCypherFilter"
  },
  "size": 10,
  "fields": ["id"]
}

Query Output for above:

{
  "took": 602,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}

UPDATE: Added verbose logging to ES

.......
search_1     | [2017-02-16 16:29:27,934][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Boosting results: {
search_1     |   "took" : 14,
search_1     |   "timed_out" : false,
search_1     |   "_shards" : {
search_1     |     "total" : 5,
search_1     |     "successful" : 5,
search_1     |     "failed" : 0
search_1     |   },
search_1     |   "hits" : {
search_1     |     "total" : 1,
search_1     |     "max_score" : 7.918695,
search_1     |     "hits" : [ {
search_1     |       "_index" : "products",
search_1     |       "_type" : "product",
search_1     |       "_id" : "636279",
search_1     |       "_score" : 7.918695,
search_1     |       "fields" : {
search_1     |         "id" : [ 636279 ]
search_1     |       }
search_1     |     } ]
search_1     |   }
search_1     | }
search_1     | [2017-02-16 16:29:27,935][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading headers...
search_1     | [2017-02-16 16:29:27,935][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading hits...
search_1     | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading aggregations...
search_1     | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading suggest...
search_1     | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Creating new SearchResponse...
search_1     | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Rewriting overhead time: 62 - 14 = 48ms
.......

Is this deprecated?

Hi,

This is a useful concept because it can improve understanding of complex networks... do you plan to come back to it?

Just curious because I prefer active codebases!

Java Client API: failed to parse search source. unknown search element [gas-filter]

If i run this query over TCP with the Java Api (port 9300), i get this error:

SearchParseException[failed to parse search source. unknown search element [gas-filter]];

However, if i run this query over HTTP it works.

Any ideas? It seems like this plugin doesn't work with the Java API.

Cheers,
Clay

>  {
>   "size" : 100,
>   "query" : {
>     "bool" : {
>       "must" : {
>         "term" : {
>           "disabled" : false
>         }
>       }
>     }
>   },
>   "gas-filter" : {
>       "name" : "SearchResultCypherFilter",
>       "query" : "MATCH (a)-[:KNOWS*]->(d{referenceId:'522d8b956819338605a53948c5dcef1a'}) RETURN distinct a.referenceId as guid",
>       "exclude" : true
>     }
>   }