ongr-io / elasticsearchdsl

Query DSL library for Elasticsearch

ElasticsearchDSL

Introducing Elasticsearch DSL library to provide objective query builder for Elasticsearch bundle and elasticsearch-php client. You can easily build any Elasticsearch query and transform it to an array.

If you need any help, stack overflow is the preferred and recommended way to ask ONGR support questions.

If you like this library, help me to develop it by buying a cup of coffee

Version matrix

Elasticsearch version	ElasticsearchDSL version
>= 7.0	>= 7.0
>= 6.0, < 7.0	>= 6.0
>= 5.0, < 6.0	>= 5.0
>= 2.0, < 5.0	>= 2.0 (not supported)
>= 1.0, < 2.0	1.x (not supported)
<= 0.90.x	not supported

Documentation

The online documentation of the bundle is here

Try it!

Installation

Install library with composer:

$ composer require ongr/elasticsearch-dsl

elasticsearch-php client is defined in the composer requirements, no need to install it.

Search

Elasticsearch DSL was extracted from Elasticsearch Bundle to provide standalone query dsl for elasticsearch-php. Examples how to use it together with Elasticsearch Bundle can be found in the Elasticsearch Bundle docs.

If you dont want to use Symfony or Elasticsearch bundle, no worries, you can use it in any project together with elasticsearch-php. Here's the example:

If you are using Symfony there is also the ElasticsearchBundle which provides full integration with Elasticsearch DSL.

The library is standalone and is not coupled with any framework. You can use it in any PHP project, the only requirement is composer. Here's the example:

Create search:

 <?php
  require 'vendor/autoload.php'; //Composer autoload

  $client = ClientBuilder::create()->build(); //elasticsearch-php client
  
  $matchAll = new ONGR\ElasticsearchDSL\Query\MatchAllQuery();
  
  $search = new ONGR\ElasticsearchDSL\Search();
  $search->addQuery($matchAll);
  
  $params = [
    'index' => 'your_index',
    'body' => $search->toArray(),
  ];
  
  $results = $client->search($params);

Elasticsearch DSL covers every elasticsearch query, all examples can be found in the documentation

elasticsearchdsl's Issues

A helper to be able execute functional tests easy

Something same as https://github.com/ongr-io/ElasticsearchBundle/blob/v1.0.0-alpha.4/Test/AbstractElasticsearchTestCase.php but without managers, repositories and services. The objective is to be able easily create index, insert dummy data, delete index.

Group aggregation by it's kind

There are too many aggs and names are starting to duplicate. There is introduced pipeline namespace, in addition, we could move bucket and metric also to it namespace group.

Don't forget to keep BC and add deprecations.

function_score has problem

with this function

private function applyQuery(array &$function, BuilderInterface $query = null)
    {
        if ($query) {
            $function['query'] = $query->toArray();
        }
    }

When i execute these codes below:

$functionScoreQuery = new FunctionScoreQuery(new MatchAllQuery());
$stateTermQuery = new TermQuery('state', 'FINISH');
$functionScoreQuery->addWeightFunction(100, $stateTermQuery);
$stateTermQuery = new TermQuery('state', 'PENDING');
$functionScoreQuery->addWeightFunction(90, $stateTermQuery);

I cannot generate "filter" for attribute state

for example

.....

"function_score": {
                        "query": {
                            "match_all": []
                        },
                        "functions": [
                            {
                                "weight": 100,
                                "filter": {
                                    "term": {
                                        "state": "FINISH"
                                    }
                                }
                            },
                            {
                                "weight": 90,
                                "filter": {
                                    "term": {
                                        "state": "PENDING"
                                    }
                                }
                            }
                        ]
                    }

.....
Please let me know how can i create filter, it's always generate query

Add support for pipeline aggregations

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html

Its a while now that they are implemented and its high time ONGR receives them as well

Add additional mention of parameters usage

There is still quite difficult to realize the ParametersTrait usage. We need to add additional information in query and aggregation topics regarding parameters.

include ES 2.0 in travis testing env

Right now we are using only simpel elasticsearch service provided from travis. Lets include custom ES versions the latest 1.* and 2.*

Same as here: ongr-io/ElasticsearchBundle#453

ReversedNestedAggregation cannot be used without inner aggregations

Currently aggregations have to be added to the ReverseNestedAggregation, otherwise an exception will be thrown:

ElasticsearchDSL/src/Aggregation/ReverseNestedAggregation.php

Line 75 in d218a69

throw new \LogicException("Reverse Nested aggregation `{$this->getName()}` has no aggregations added");

Here is a use case: I want to aggregate on nested objects, but return the doc_count of the top level documents (because several nested objects can match). This can be achieved by:

"aggs": {
    "color_nested": {
      "nested": {
        "path": "variants"
      },
      "aggs": {
        "color": {
          "terms": {
            "field": "variants.color"
          },
          "aggs": {
            "color_reverse": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }

As illustrated by this query, in this case scenario there is no point in having additional aggregations and it is supported by Elasticsearch. I think we should remove the exception here, what do others think?

Consider implement fluent interface pattern to the endpoints

There is already a suggestion with aggregation #140

Consider to add elasticsearch-php to the composer.

Maybe we should add elasticsearch-php client dependency in requirements.?

There is a lack of documentation on how to use sort

addDecayFunction without options

addDecayFunction does not work if no options are passed. There is no function with "0" name.

{
  "functions": [
    {
      "linear": {
        "myField": {
          "origin": 0,
          "scale": 100
        }
      },
      "0": []
    }
  ]
}

IMHO options should be siblings of "myField" and not "linear". Otherwise you will not be able to add decay on multi value fields.

Missing timeout and other parameters in a request body search

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#_parameters_5

Create context suggester

Create completion suggester

script_score query does not support [params] in ElasticSearch 5.0

https://www.elastic.co/guide/en/elasticsearch/guide/master/script-score.html

Add parameters processing in getArray() for output in all aggregations

Ability to use Nested Child Query with parent child inner hits

Hi,

I'm currently using the HasChildQuery class to include additional documents within my result set. I need to include the inner_hits part of the query in order to get the right results but it looks like this ability isn't part of the HasChildQuery implementation when dealing with parent/child documents (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#parent-child-inner-hits).

Do you know if this is planned to be included in later versions? I saw that inner_hits support was added for other queries but not for parent child ones.

TopHitsAggregation does not use parameters.

$topHitsAggregation->addParameter('_source', ['include' => ['title']]);

Has no effect on resulting array.

fields was changed to stored_fields in elasticsearch 5.0

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-stored-fields.html#search-request-stored-fields

Unnecessary processing of parameters in aggregations

There are two aggregations that have processArray() in their getArray() methods. These calls should be removed:

ElasticsearchDSL/src/Aggregation/DateHistogramAggregation.php

Line 85 in 1dc07a5

$out = $this->processArray($out);

ElasticsearchDSL/src/Aggregation/TopHitsAggregation.php

Line 139 in 1dc07a5

$output = $this->processArray($output);

Some queries should support multiple fields

I have tested range query and found out that it works with multiple fields (tested with ES 2.1.1). This library does not support that.

Example:

{
    "range": {
        "price": {
            "gt": 50
        },
        "title": {
            "gt": "ccc"
        }
    }
}

Currently if you want to achieve something like that using our library you need to create two range queries and put them into bool.

We should check how many queries support multiple fields and think how we could add support for it.

Edit: prefix and regexp queries also supports multiple fields (tested on ES 2.1.1).

parsing_exception: no [query] registered for [missing] in ElasticSearch 5.0

Deprecated in 2.2.0.
Use exists query inside a must_not clause instead.

How to use filtered

Is there a way to set filtered node in query node?

Example Query:
{ "query":{ "filtered":{ "query":{ "query_string":{ "query":"test", "fields": ["title", "content","abstract","page_title_s"], "minimum_should_match":"30%" } }, "filter":{ "bool":{ "should":[ { "term":{ "CType":"bullets" } }, { "term":{ "CType":"text" } }, { "term":{ "CType":"textpic" } }, { "term":{ "CType":"header" } }, { "term":{ "CType":"table" } } ] } } } } }

Rename `Suggest` to the `TermSuggest`

It's actually a term suggest, so the name should be correct.

toArray function should return the name as a key, and SuggestEndoint has to handle suggest key return. e.g. the structure should be like this:

...
"suggest" : {
  "my-suggest-1" : {
    "text" : "the amsterdma meetpu",
    "term" : {
      "field" : "body"
    }
  },
  "my-suggest-2" : {
    "text" : "the rottredam meetpu",
    "term" : {
      "field" : "title"
    }
  }
}

So here's what we have to do:

• Refactor a SuggestEndpoint to handle more than one suggestion and provide a suggest key.
• Rename Suggest to the TermSuggest
• Add $name as a key to return the structure in toArray function in TermSuggest.

ParameterTrait not used in FieldSort class

Parameters can be passed only trough construct when adding sort.

Split tests structure to the functional and unit

MatchAllQuery() not working with Elasticsearch version 5 if no parameters are provided

If i instantiate an new MatchAllQuery Object with for example a boost of 1 MatchAllQuery(['boost' => 1]) everything works as intended:
"match_all":{"boost":1}

However if i provide no parameters to MatchAllQuery(), the getParameters() returns an empty array which remains an array when encoded to JSON:
"match_all":[]
That raises an exception inside Elasticsearch:

[match_all] query malformed, no start_object after query name

Elasticsearch expects an object even if it is empty.

Expected JSON:
"match_all":{}

Cover all query classes with tests

All query classes SHOULD be reviewed. There might be deprecated queries or changes in query structure.

Multiple Nested Arguments

I'm looking to reproduce the following:

"aggs": {
    "stock_type": {
      "terms": {
        "field": "stock_type"
      },
      "aggs": {
        "year": {
          "terms": {
            "field": "year",
            "size": 500
          },
          "aggs": {
            "make": {
              "terms": {
                "field": "make",
                "size": 500
              },
              "aggs": {
                "model": {
                  "terms": {
                    "field": "model",
                    "size": 500
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
       

Basically, I'm looking for:

• Stock > Year > Make > Model

I have 3 days of experience with ES but I am stepping into a working established project.

The example code implies some background knowledge in ES.

Any help or suggestion would be greatly appreciated.

FiltersAggregation generates invalid array

The FiltersAggregation adds filters inside an array with the type as key. Though, this generates the following array for a TermQuery with the 'term'-key twice in the stack:

array(
    'filtername' => array(
        'term' => array(
            'term' => array(
                'fieldname' => 'fieldvalue'
            )
        )
    )
)

This filter will throw an exception when sending it with a query to an Elasticsearch server.

I can supply a PR to fix this issue if you want.

No alive nodes found in your cluster on after 2.2.0 update

Hi,

After upgrading from the elasticsearch-dsl 2.1.2 to 2.2.0, we are receiving the No alive nodes found in your cluster. We reverted back and it is working fine.

Is something changed on the version release? we are also using shield along with our ES node.

Regards,

Cannot set "size" parameter for TermsAggregation

According to the ES documentation, when performing a "terms" aggregation, ElasticSearch will by default return the top 10 terms. If you want more, you have to pass an additional "size" parameter.

However, there is no way to set that parameter on ONGR\ElasticsearchDSL\Aggregation\Bucketing\TermsAggregation .

It seems to me that this class (and perhaps other aggregation classes?) should either:

1. Have a getter/setter for the "size" parameter, or
2. Support an optional "parameters" array in the constructor allowing the passing of any additional parameters. This matches the classes in ONGR\ElasticsearchDSL\Query.

Of course option 2 is more flexible, but perhaps too much so?

I'd be happy to implement either one and make a PR, but I'd like to know that a solution is not already in the pipeline, and also get your input on which of the two options you'd prefer.

Change BoolFilter to BoolQuery

There is a left over after Filters deprecation.

ElasticsearchDSL/src/Search.php

Line 223 in 1a38c05

public function addFilter(BuilderInterface $filter, $boolType = BoolFilter::MUST, $key = null)

Raw DSL element needed

When building a Query, sometimes we need a feature that is not supported by ESB yet. Some RawXXX is needed of form:

$search->addAggregation(new RawAggregation([
    'name' => 'foo',
    'field' => 'field,
    'aggregations' => [/*...*/],
]));

Which would inject custom aggregation introduced to Elastisearch, for example, 2 days ago.

Other types of elements might be added too, like RawFilter, etc.

origin: ongr-io/ElasticsearchBundle#322

Add options support to SpanNotQuery

Removes matches which overlap with another span query. The span not query maps to Lucene SpanNotQuery. Here is an example:

{
    "span_not" : {
        "include" : {
            "span_term" : { "field1" : "hoya" }
        },
        "exclude" : {
            "span_near" : {
                "clauses" : [
                    { "span_term" : { "field1" : "la" } },
                    { "span_term" : { "field1" : "hoya" } }
                ],
                "slop" : 0,
                "in_order" : true
            }
        }
    }
}

The include and exclude clauses can be any span type query. The include clause is the span query whose matches are filtered, and the exclude clause is the span query whose matches must not overlap those returned.
Read More...

There is no possibility to add top level inner hits.

More info here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#top-level-inner-hits

Create phrase suggester

BoolQuery::getQueries with ie. BoolQuery::MUST triggers 'Undefined index: must'

For my particular use case I'm re-using the query (instances) in aggregations. When using the BoolQuery::getQueries method with a $boolType argument. It will trigger a notice that an array index is not defined when no query instances have been added to the boolType.

From the docblock you'd assume that it would always return an (empty) array. Also because there's currently no other way to read the added query instances from the BoolQuery instance with the specified boolType.

I can provide a PR for this if you'd like.

Nested query with bool doesn't work when there is only one MUST condition.

$bool = new BoolQuery();
$bool->add(new MatchQuery('some.field', 'someValue'), BoolQuery::MUST);

$nested = new NestedQuery('urls', $bool);

Will produce:

{
  "nested": {
    "path": "urls",
    "query": {
      "bool": {
        "match": {
          "some.field": {
            "query": "someValue"
          }
        }
      }
    }
  }
}

Missing must condition inside bool...

Add "type" support to IdsQuery

Filters documents that only have the provided ids. Note, this query uses the _uid field.

{
    "ids" : {
        "type" : "my_type",
        "values" : ["1", "4", "100"]
    }
}

The type is optional and can be omitted, and can also accept an array of values. If no type is specified, all types defined in the index mapping are tried.

Check if there are any new queries or aggs that we still do not support.

None of the span queries have documentation

It's time to add functional tests

I think we need to add functional tests to check queryig on both 2.x and 5.x elasticseach versions.

Review differences in search body between Elasticsearch 2.x and 5.

There were some issues where 's hard to keep BC across 2.x and 5.x. Most likely there is a need for the new major version by dropping support for Elasticsearch 2.x.

If FunctionScore does not have any functions breaks query

multiple termQuery not working

Hi !
I am trying to execute this query but it seems that it's not supported to add multiple termQuery.

<?php
       $search = new Search();
        $termQueryForTag1 = new TermQuery("tag", "wow");
        $termQueryForTag2 = new TermQuery("tag", "elasticsearch");
        $termQueryForTag3 = new TermQuery("tag", "dsl");

        $search->addQuery($termQueryForTag1);
        $search->addQuery($termQueryForTag2);
        $search->addQuery($termQueryForTag3, BoolQuery::MUST);

        $queryArray = $search->toArray();
        echo "<pre>";var_dump($queryArray);die(); 

Just the laste query was token

array(1) {
  ["query"]=>
  array(1) {
    ["term"]=>
    array(1) {
      ["tag"]=>
      string(3) "dsl"
    }
  }
} 

Is it a bug or I am doing it wrongly ?

PS: I am using ElasticSearch 2.3.3 and elasticsearch-dsl 2.2.

Thanks in advance,

Support more than 2 levels in aggregation find function

Support other parameters in termsAggregation

Hello,

According to this documentation
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

Terms Aggregation can support many parameters :
size, min_doc_count, order,...

Can you add to this ?
Thanks

Provide more information in README

At least it would be nice to point users to documentation.

At best [IMHO] we could provide all the main information directly in README. Nothing could be simpler than to open repository and see what is it, when to use it and how to use it.

Add coveralls integration with tests code coverage