2019-05-02: Add, update and delete nested objects in Elasticsearch documents about iridakos-posts

[THEjoezack]

Great post, I didn't realize you could do this with nested objects. Thanks!

[jwilcken]

Excellent post! Used your update example for work. Thank you!

[alvaroseparovich]

THANK YOU ALOT MAN!
The documentation on elastic site is good, but it seems to be not so deep.
This is what I was looking for.

[iridakos]

@THEjoezack Thank you, I also didn't know about this functionality until recently. 👍

[mohittyagi6677]

@iridakos How to add a field to an existing nested object. What would the PUT query look like?

[iridakos]

@mohittyagi6677 Hi,

Suppose we have already created the following index:

PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "users": {
          "type": "nested",
          "properties": {
            "firstname": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

If we wanted to add another property to the users nested object, we would execute the following:

PUT my_index/_mapping/_doc
{
  "properties": {
    "users": {
      "type": "nested",
      "properties": {
        "lastname": {
          "type": "text"
        }
      }
    }
  }
}

[JFeoOks]

@iridakos
Thank you!

What can you say about performance?
'To update, add, or remove a nested object, we have to reindex the whole document'
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html#nested-objects

[benjieperez]

I want to knew why this specific code you've given is not working on me, its not updating all of my fields inside the nested object..
"script": {
"source": "def targets = ctx._source.cats.findAll(cat -> cat.breed == params.current_breed); for(cat in targets) { cat.breed = params.breed }",
"params": {
"current_breed": "European",
"breed": "European Shorthair"
}
}

I rework it as this:
"script": {
"source": "def targets = ctx._source.eng.findAll(); for(cat in targets) { cat.soc_mm_score = params.new_mm }",
"params": {
"new_mm": "1"
}
},
"query": {
"match": {
"cli_id": "696668"
}
}

[edrimon]

Hello, great information ! Based on that artile, I am trying to add nested tags in existing mappings.
I want to add tag "keyw" and "weight" inside the "content". My existing mappings are:

{
  "microlearning" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "raw_as_text" : {
            "path_match" : "meta.raw.*",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "attachment" : {
          "type" : "binary"
        },
        "attributes" : {
          "properties" : {
            "group" : {
              "type" : "keyword"
            },
            "owner" : {
              "type" : "keyword"
            }
          }
        },
        "content" : {
          "type" : "text"
        }, 
...

and I am using the following code to add the needed nested ones:

PUT /microlearning
{
	"mappings": {
		"properties": {
			"content": {
				"tags": {
					"type": "nested",
					"properties": {
						"keyw": {
							"type": "keyword"
						},
						"weight": {
							"type": "float"
						}
					}
				}
			}
		}
	}
}

However the result is erroneous, like:

{
  "error": {
    "root_cause": [
      {
        "type": "resource_already_exists_exception",
        "reason": "index [microlearning/NRUB38smQim_EsUpb_LJzQ] already exists",
        "index_uuid": "NRUB38smQim_EsUpb_LJzQ",
        "index": "microlearning"
      }
    ],
    "type": "resource_already_exists_exception",
    "reason": "index [microlearning/NRUB38smQim_EsUpb_LJzQ] already exists",
    "index_uuid": "NRUB38smQim_EsUpb_LJzQ",
    "index": "microlearning"
  },
  "status": 400
}

What am I missing here?

BR, Makis

[iridakos]

@iridakos
Thank you!

What can you say about performance?
'To update, add, or remove a nested object, we have to reindex the whole document'
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html#nested-objects

@JFeoOks
That is correct, the whole document has to be reindexed.

[iridakos]

I want to knew why this specific code you've given is not working on me, its not updating all of my fields inside the nested object..
"script": {
"source": "def targets = ctx._source.cats.findAll(cat -> cat.breed == params.current_breed); for(cat in targets) { cat.breed = params.breed }",
"params": {
"current_breed": "European",
"breed": "European Shorthair"
}
}

I rework it as this:
"script": {
"source": "def targets = ctx._source.eng.findAll(); for(cat in targets) { cat.soc_mm_score = params.new_mm }",
"params": {
"new_mm": "1"
}
},
"query": {
"match": {
"cli_id": "696668"
}
}

Hi @benjieperez,
A little hard to understand without the index information. Are you using the _update on a specific document as described in the post? I am confused with the query > match > cli part of your query.

[iridakos]

Hi @edrimon,

Hello, great information ! Based on that artile, I am trying to add nested tags in existing mappings.
I want to add tag "keyw" and "weight" inside the "content". My existing mappings are:

{
  "microlearning" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "raw_as_text" : {
            "path_match" : "meta.raw.*",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "attachment" : {
          "type" : "binary"
        },
        "attributes" : {
          "properties" : {
            "group" : {
              "type" : "keyword"
            },
            "owner" : {
              "type" : "keyword"
            }
          }
        },
        "content" : {
          "type" : "text"
        }, 
...

and I am using the following code to add the needed nested ones:

PUT /microlearning

^ This line is to create an indexing and not to update it.
You should be using: PUT /microlearning/_mapping/_doc

{
	"mappings": {
		"properties": {
			"content": {
				"tags": {
					"type": "nested",
					"properties": {
						"keyw": {
							"type": "keyword"
						},
						"weight": {
							"type": "float"
						}
					}
				}
			}
		}
	}
}

However the result is erroneous, like:

{
  "error": {
    "root_cause": [
      {
        "type": "resource_already_exists_exception",
        "reason": "index [microlearning/NRUB38smQim_EsUpb_LJzQ] already exists",
        "index_uuid": "NRUB38smQim_EsUpb_LJzQ",
        "index": "microlearning"
      }
    ],
    "type": "resource_already_exists_exception",
    "reason": "index [microlearning/NRUB38smQim_EsUpb_LJzQ] already exists",
    "index_uuid": "NRUB38smQim_EsUpb_LJzQ",
    "index": "microlearning"
  },
  "status": 400
}

What am I missing here?

That's why you get an error that the index with this name already exists.

BR, Makis

But even if you did it the proper way, there's another problem here.

In the original index, the content has been defined as text. You can't add nested properties to a text property.

mapper [content] of different type, current_type [text], merged_type [ObjectMapper]

Cheers!

[edrimon]

But even if you did it the proper way, there's another problem here.

In the original index, the content has been defined as text. You can't add nested properties to a text property.

Hi, are you sure? I was checking this example below which is using nested tags to text properties..

https://conan.is/blogging/weighted-elasticsearch.html

[iridakos]

In your first snippet, content is declared as text:

"content" : {
  "type" : "text"
},

When you try to update the mappings, you use:

"properties": {
  "content": {
    "tags": {
	"type": "nested",
	"properties": {
	  "keyw": {
	    "type": "keyword"
	  },
	 "weight": {
	   "type": "float"
	 }
      }
    }
  }
}

in other words, you are trying to add a property tags inside the previously defined property content which is of type text.

Any chance you copy/pasted the code erroneously?
Did you mean:

"properties": {
    "tags": {
	"type": "nested",
	"properties": {
	  "keyw": {
	    "type": "keyword"
	  },
	 "weight": {
	   "type": "float"
	 }
      }
    }
}

[edrimon]

Yes, now I see. It is really too interesting.. thanks..!

Makis

[edrimon]

I did like you said and it worked, the nested fields are added to the mappings. But now I want to populate those fields of an existing document, for example to add a new keyw with value "X" and weight 0.8 and also Y,0.2.

I cannot find a clear way after googling to perform it? Some say use scripting , some other to try directly but it is not clear.. How could I just update the current document and populate for example two pairs of the nested fields keyw,weight there?

Actually here in the document says for update , delete etc , but I do not understand how I can insert values to the fields that I just added to the mappings, for an existing document (but having empty nested fields as they were just created in the mappings)

[lsivasub]

Hi, Thanks for the well explained article. I have a scenario, where each object within the nested document with different values. Taking your cats example, let's say I have updated the mapping to have another property "age" of type "short". Now I want to set the age of every cat cat.name:"Irida" to age 2, cat.name:"Phoebe" to age 4, etc. how will I do that. I am trying by iterating over the cats and params list, yet have not figured it out. Thanks

[xavi-developer]

Thank you very much for this awesome post, I have been searching for something like this for months.

I wanted to point an issue I am having. I have an User index with uuid, name and email fields that are set on document creation. When user creates a comment, I want to execute the following script sending the comment data within params:

ctx._source.comments.add(params.subentityItem)

If user has previous comments (therefore "comments" field is already present on the document) it works properly, but if the document doesn't have the property it crashes with error

Cannot invoke "Object.getClass()" because "receiver" is null

I have tried some ways to create "comments" field dynamically (source)

  "query": {
   "bool": {
    "exists": {
      "field": "comments"
    }
   }
  }

but it gives the error

[UpdateRequest] unknown field [query]

Is there any way to create that field dynamically when trying to modify it? I could add the field with empty value during the document creation, but the idea is to have it only when needed

Thank you very much

[rsajpon]

Thanks a lot, just what I was looking for, not found in Elasticsearch docs 👍

[brianbooth]

Thank you! This article should be required reading for anyone before using nested types. I've wasted too much time trying to figure this out and your article made it clear instantly.

But I think I spotted a harmless typo. On the web page, in the section 'Remove a nested object', you have the following:

Confirm the addition with:
GET iridakos_nested_objects/human/1
Cat removed:

I'm pretty sure you meant removal

[brianbooth]

Hi @xavi-developer,

I'm really new to this, but I think something like this should work for you.

"script": {
    "source": """ 
        if (ctx._source.comments == null) ctx._source.comments = new ArrayList();  
        ctx._source.comments.add(params.subentityItem);
    """,
    "params" : {
        "subentityItem" : {
                ...
        }
    }
}

[xavi-developer]

Hi @brianbooth

thank you for the tip, I'll give it a try!

[chinmayishetty7]

Hello @iridakos , I found this link very useful. Is there a way of finding the index of a particular item in the cats collection and use a set method replace an item?