Getting different (wrong) result when using PHP client about elasticsearch-php

[polyfractal]

Ah, I think I see the problem. It's the way Elasticsearch operates, which can sometimes cause strange results when dealing with it from an algorithm (usually when doing integration tests).

By default, Elasticsearch performs a refresh operation every second. This refresh makes new documents visible to search...until the refresh is executed the documents are effectively invisible to search operations. A count request is just a special type of search that counts the total number of documents, so it is influenced by this refresh interval.

When you are indexing into ES rapidly, it is possible to index and then call a Count request in just a few milliseconds...way below the 1s threshold. When you run the curl request manually, enough time has elapsed so the docs are "visible".

To fix this in your test code, just add a refresh command:

$client = new \Elasticsearch\Client();

$deleteParams['index'] = 'test';
$client->indices()->delete($deleteParams);

$indexParams['index'] = 'test';
$client->indices()->create($indexParams);

$doc = new \stdClass();
$doc->id = 123;
$doc->field = "abc";
$doc->field2 = "xyz";

$params = array();
$params['id'] = $doc->id;
$params['index'] = 'test';
$params['type'] = 'item';
$params['body'] = (array)$doc;
$client->index($params);

// This refresh command will force a refresh and you'll see correct counts
$client->indices()->refresh(array('index' => 'test'));

$result = $client->count(array('index' => 'test'));

Array
(
    [count] => 1
    [_shards] => Array
        (
            [total] => 5
            [successful] => 5
            [failed] => 0
        )

)

Of course, this is just for testing...you shouldn't call a refresh after every document is indexed or you will hurt your indexing speed and performance.

[bretrzaun]

Yes - that did help, I added the refresh just before the count and everything works as expected now.
Thanks for your help ! Made my starting experience with Elasticsearch very pleasant.

[polyfractal]

I'm a little confused, the output from your curl command also shows a count of zero?

[count] => 0

??

If you can paste your entire set of commands I'd be happy to run them myself and see if I can recreate the situation. Are you on Elasticsearch 1.0 or an older version?

[bretrzaun]

I am sorry - I copied in the wrong example.
Now updated...

[polyfractal]

Ok, that makes more sense :)

Can you paste a set of commands which recreate the issue? It's much easier to debug an example than to just start digging through the code. Thanks!

[bretrzaun]

I posted the commands I am using and the log output here:

http://pastebin.com/5u6dWNZV

PS: Yes - I am using Elasticsearch 1.0.0

[polyfractal]

Great! Glad to help...some parts of Elasticsearch can be confusing for new users, and you ran into one of them. I'll see about adding a blurb to the docs about this problem for future users.

Lemme know if you run into anything else, bug or otherwise! :)

[eddiejaoude]

@polyfractal @bretrzaun It is possible to update the index in a single request with an additional parameter...I have updated your example as follows

$client = new \Elasticsearch\Client();

$deleteParams['index'] = 'test';
$client->indices()->delete($deleteParams);

$indexParams['index'] = 'test';
$client->indices()->create($indexParams);

$doc = new \stdClass();
$doc->id = 123;
$doc->field = "abc";
$doc->field2 = "xyz";

$params = array();
// -------------------------
$params['refresh'] = true; // no need for a separate refresh request
// -------------------------
$params['id'] = $doc->id;
$params['index'] = 'test';
$params['type'] = 'item';
$params['body'] = (array)$doc;
$client->index($params);

$result = $client->count(array('index' => 'test'));

[polyfractal]

Yep, you can indeed do this :)

I generally hide this option from new users because it is easy to slap a "refresh" onto all your indexing commands...and then forget that "refresh" has been toggled. This becomes very expensive because it is refreshing on each new document.

I prefer to introduce the concept as an explicit API call so that people are aware that it is an additional operation which is being performed, so they consider the overhead of calling it repeatedly.

But you're right, you can absolutely add it to individual commands instead of a second request.

[eddiejaoude]

I prefer to introduce the concept as an explicit API call so that people are aware that it is an additional operation which is being performed, so they consider the overhead of calling it repeatedly.

Fair point 😄 👍

[eddiejaoude]

Where is the curl & json equivalent of this in the ElasticSearch Docs? I found it once before, but can't seem to find it again

[polyfractal]

@eddiejaoude It's described (briefly, without code sample) here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-refresh

[eddiejaoude]

Thanks @polyfractal , that is the Documentation I had found too. I was looking for an actual example.

However, I got more information from the IRC & I documented it here https://github.com/TransformCore/elasticsearch-example-docs/blob/master/docs/4-runtime-parameters/refresh-parameter.md#example