dakrone / elasticsearch-in-action

Offical code repository for the Elasticsearch in Action book from Manning

This repository contains the code examples from the book Elasticsearch in Action.

When do I need them?

Because it's nice to have some sample data to run various searches from the book's listings. Throughout most chapters, there's an example use-case of running a get-together site, where you have people organizing themselves into groups and hosting events. Code samples provide you with some data and a script to index that data.

How do I index the sample data?

# clone the repository
git clone https://github.com/dakrone/elasticsearch-in-action.git

# switch to a branch that matches your version. Master works with 1.x and 2.x
# but we currently support 5.x, 6.x and 7.x as well:
git clone https://github.com/dakrone/elasticsearch-in-action.git -b 7.x

# index the sample data
elasticsearch-in-action/populate.sh

I have questions or suggestions for these samples

You have feedback? We'd be glad to hear it from you on the book's forum. To get there, go to the live book and click on the discuss button on the top-right. You can also send pull requests or open issues on this repository.

Happy reading and Elasticsearch-ing!

elasticsearch-in-action's Issues

6.x minor bug

This word should be "aliases",test on 6.6.1.

there is a missing quotation here,cause the indexing fail.

thx

when using branch 6.x with elasticsearch 7.x, met error 400.

when using branch 6.x with elasticsearch 7.x, met error 400.

elasticsearch version:

curl 'localhost:9200'
{
  "name" : "7a7216cec4ae",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "6IW0Nq7qRKSXlh5tfkNI6A",
  "version" : {
    "number" : "7.0.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "b7e28a7",
    "build_date" : "2019-04-05T22:55:32.697037Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

branch and version:

* 0a6a376 - (origin/6.x) Fixed mapping typo (4 weeks ago) <Radu Gheorghe>

the error logs as below:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400}

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [relationship_type] of type [text] in document with id '100'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [relationship_type] of type [text] in document with id '100'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 2:24"}},"status":400}

Problem with populate.sh indexing events.

I've installed Elastic Search 5 with X-pack. I modified the ./populate.sh script to workaround the authentication issue. However, when I run ./populate.sh, I get:

Indexing events...
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
...

typo in populate.sh

In running populate.sh, I noticed an error message for Templates (UTF8 error).

In the second template (about lines 336-339 of the populate.sh file), there are
curly double quotes instead of normal double-quote character (this in two lines).

WHen these are changed to normal double-quote characters, populate.sh runs without error.
So, I think you should change populate.sh in the repository.

Errors in populate.sh

Hello,

I am trying to run populate.sh on my station, but I am running into a few issues which narrow down to wrong HTTP method being called when creating get-together index:

Creating 'get-together' index...
{"error":"Incorrect HTTP method for uri [/get-together] and method [POST], allowed: [HEAD, DELETE, GET, PUT]","status":405}

and content-type not being specified when populating the index (the fact that index wasn't created is being ignored at this point):

{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

I am running the script on a Fedora machine:

$ uname -a
Linux localhost.localdomain 4.18.13-100.fc27.x86_64 #1 SMP Wed Oct 10 18:34:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The ES version is 6.4.2.

Could you please update the script so I can pull and re-execute it. Or else advise what am I doing wrong?

populate.sh Not available in version 7.0

curl -s -H 'content-Type:application/json' -XPOST "$ADDRESS/get-together/event/100?parent=1"

The 7.0 syntax seems to have changed and the declaration with 'parent=1 'will not pass.

ability to specify elasticsearch host in populate.sh

the localhost is fixed there, it should be a variable from the command line argument which defaults to localhost

read statement does not work

Hi
I used the populate.sh script and the 'read' command did not work.
It just goes on without asking for my confirmation.
It started to work when i changed #!/bin/sh to #!/bin/bash.

[application/x-www-form-urlencoded] is not supported","status":406}

[application/x-www-form-urlencoded] is not supported","status":406}

error index type in event

when we create data by populate.sh,string type is not 'string', it created to 'type':'text'.In code case 7-10,we need "aggregation" on field "attendees",at this moment will cause error :
Fielddata is disabled on text fields by default. Set fielddata=true on [attendees] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
my es version is :5.6.8

populate.sh gives {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

$ curl -XGET 'http://localhost:9200'
{
"name" : "damir-VirtualBox",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "l5HYNXRqS5ScoAkUl3MYnA",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

sudo ./populate.sh
-same thing without sudo.
WARNING, this script will delete the 'get-together' and the 'myindex' indices and re-index all data!
Press Control-C to cancel this operation.

Press [Enter] to continue.

Creating 'get-together' index...
{"error":"Incorrect HTTP method for uri [/get-together] and method [POST], allowed: [DELETE, GET, PUT, HEAD]","status":405}
Done creating 'get-together' index.

Indexing data...
Indexing groups...
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done indexing groups.
Indexing events...
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done indexing events.
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [get-together]","resource.type":"index_or_alias","resource.id":"get-together","index_uuid":"na","index":"get-together"}],"type":"index_not_found_exception","reason":"no such index [get-together]","resource.type":"index_or_alias","resource.id":"get-together","index_uuid":"na","index":"get-together"},"status":404}
Done indexing data.

Creating Templates.
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done Creating Templates.

Adding Dynamic Mapping
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done Adding Dynamic Mapping

Adding Aliases
{"error":"Incorrect HTTP method for uri [/november_2014_invoices] and method [POST], allowed: [DELETE, GET, PUT, HEAD]","status":405}
{"error":"Incorrect HTTP method for uri [/december_2014_invoices] and method [POST], allowed: [DELETE, GET, PUT, HEAD]","status":405}
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done Adding Aliases
Adding Filter Alias
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done Adding Filter Alias

Adding Routing Alias
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Done Adding Routing Alias

"can't specify parent if no parent field has been configured for ES 6.x

It's not indexing in ES 6.x, need help.
Thanks

Second Edition?

This book has been an invaluable Elasticsearch reference for me over the years. The query syntax in 7.x (and probably earlier) has started to diverge a bit from the examples in the book. If there's any chance of a second edition, I would be first in line to buy it. I wanted to post this on the Manning Forum but it says it is "410 Gone"... an HTTP code I have not seen that often.

populate.sh fails

Elasticsearch version: 5.1.1

output

$ ./populate.sh 
WARNING, this script will delete the 'get-together' and the 'myindex' indices and re-index all data!
Press Control-C to cancel this operation.

Press [Enter] to continue.

Creating 'get-together' index...
No handler found for uri [/get-together] and method [POST]
Done creating 'get-together' index.

Indexing data...
Indexing groups...
{"_index":"get-together","_type":"group","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
{"_index":"get-together","_type":"group","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
{"_index":"get-together","_type":"group","_id":"3","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
{"_index":"get-together","_type":"group","_id":"4","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
{"_index":"get-together","_type":"group","_id":"5","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
Done indexing groups.
Indexing events...
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}
Done indexing events.
{"_shards":{"total":10,"successful":5,"failed":0}}
Done indexing data.


Creating Templates.
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : false]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [date]: Root mapping definition has unsupported parameters:  [store : false]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : false]"}},"status":400}
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"failed to parse template source"}],"type":"illegal_argument_exception","reason":"failed to parse template source","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('“' (code 8220 / 0x201c)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@5fdeb449; line: 6, column: 12]"}},"status":400}
Done Creating Templates.

Adding Dynamic Mapping
{"acknowledged":true,"shards_acknowledged":true}
Done Adding Dynamic Mapping

Adding Aliases
No handler found for uri [/november_2014_invoices] and method [POST]
No handler found for uri [/december_2014_invoices] and method [POST]
{"error":{"root_cause":[{"type":"parsing_exception","reason":"[alias_action] failed to parse field [add]","line":14,"col":3}],"type":"parsing_exception","reason":"[aliases] failed to parse field [actions]","line":14,"col":3,"caused_by":{"type":"parsing_exception","reason":"[alias_action] failed to parse field [add]","line":14,"col":3,"caused_by":{"type":"illegal_argument_exception","reason":"Can't repeat param [add]"}}},"status":400}
Done Adding Aliases
Adding Filter Alias
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"december_2014_invoices","index_uuid":"_na_","index":"december_2014_invoices"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"december_2014_invoices","index_uuid":"_na_","index":"december_2014_invoices"},"status":404}
Done Adding Filter Alias

Adding Routing Alias
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"december_2014_invoices","index_uuid":"_na_","index":"december_2014_invoices"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"december_2014_invoices","index_uuid":"_na_","index":"december_2014_invoices"},"status":404}
Done Adding Routing Alias

error creating templates

Creating Templates.
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : false]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [date]: Root mapping definition has unsupported parameters:  [store : false]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : false]"}},"status":400}
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : true]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [date]: Root mapping definition has unsupported parameters:  [store : true]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [store : true]"}},"status":400}
Done Creating Templates.

what is the fix for this?

populate.sh contain chinese symbol quote, in row 336

another pattern of join type

see below
https://gitee.com/beesman/elasticsearch7x/blob/master/mapping-es7.json
https://gitee.com/beesman/elasticsearch7x/blob/master/populate7.sh
reference:
https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html

Error Indexing events

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}

Error while running populate.sh script

According to Readme file, I tried to run the populate.sh script on MacOS. Some curl commands ran fine but for some commands, I am getting following error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"}],"type":"illegal_argument_exception","reason":"Can't specify parent if no parent field has been configured"},"status":400}

---

OS: Mac OS Yosemite
curl 7.43.0
ElasticSearch: 5.2.0

Command executed (from populate.sh):

curl -s -XPOST "http://localhost:9200/get-together/event/113?parent=5" -d'{
  "host": "Dave",
  "title": "Elasticsearch at Rangespan and Exonar",
  "description": "Representatives from Rangespan and Exonar will come and discuss how they use Elasticsearch",
  "attendees": ["Dave", "Andrew", "David", "Clint"],
  "date": "2013-06-24T18:30",
  "location_event": {
    "name": "Alumni Theatre",
    "geolocation": "51.51558,-0.117699"
  },
  "reviews": 3
}'

Authentication needed

I've installed Elastic Search 5 with the steps here and when I try to run the ./populate.sh script, I get authentication errors:

Creating 'get-together' index...
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/get-together]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/get-together]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

It seems that installing X-Pack turned on authentication which needs turned back off.

Of course, then I had to use ./populate.5x.sh from the still open pull request #7.