The elasticsearch-log-dump from kfparri

elasticsearch-log-dump's People

Contributors

Stargazers

Watchers

elasticsearch-log-dump's Issues

Can't dump more than 10,000 records with search

While testing with the elastic data set. I found the limit of the number of records I can pull with search. After two pulls of 5000 records I get this response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_phase_execution_exception",
        "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [15000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "logstash-2015.05.18",
        "node" : "wi236Q7MR6mIs62MfhG8Ug",
        "reason" : {
          "type" : "query_phase_execution_exception",
          "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [15000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
        }
      }
    ]
  },
  "status" : 500
}

I need to investigate changing this logic to use the scroll functionality(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html)

Recommend Projects