Could not find a valid timestamp from the index about curator

[untergeek]

Your date is in the wrong format (for curator). It expects the default YYYY.MM.dd (see http://logstash.net/docs/1.4.0/outputs/elasticsearch#index) or even YYYY.MM.dd.HH (for hourly). There is currently no other accepted formatting. We may try to create a way to use strftime and accept defined timestamps, but that will be rather difficult as it involves matching joda-time with python's date libraries. That will be quite a yak shave...

Everything else looks great, though.

[xinity]

any "simple way" to "rename" my current indices ?

[electrical]

@xinity easiest way is to create aliases.
See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html for more info on that.

[untergeek]

I don't think you can delete an index on alias, but it's worth trying. Please let us know if that works.

[xinity]

just tried , deleting an alias doesn't delete the index itself :)
lucky me i'm not in production mode, but very soon, so changing the timestamp of my indices will do no harm.

btw, having an option to define the timestamp format would be awesome!!

[untergeek]

Mapping will be a lot bigger than just figuring out indices. That part is as simple as:
joda YYYY.MM.dd vs. python %Y.%m.%d

The complexity happens if somebody wants to start using week numbers. How do I delete based on elapsed days? Or months? The masking of dailies is not a problem. Getting tricky date math to play nice for creating complex aliases (both things like, "yesterday" and "last_week", as well as combining 7 days of regular dailies into a weekly alias, or a monthly, etc.) and other time management issues. That's why this isn't just about adding in strftime mapping.

[deepakas]

I am having the same problem. My Flume Log Stash Serializer create the index in the format raw_json-2014-05-12 . Also I had the same issue when I tried to delete marvel indices which has the format YYYY.MM.dd .

/usr/lib/python2.6/site-packages/curator/curator.py -d 3 -p .marvel
2014-05-14T18:07:44.691 INFO main:333 Job starting...
2014-05-14T18:07:44.692 INFO new_conn:180 Starting new HTTP connection (1): localhost
2014-05-14T18:07:44.694 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.003s]
2014-05-14T18:07:44.695 INFO main:359 Deleting indices older than 3 days...
2014-05-14T18:07:44.698 INFO log_request_success:49 GET http://localhost:9200/.marvel_/_settings?expand_wildcards=closed [status:200 request:0.002s]
2014-05-14T18:07:44.722 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.07
2014-05-14T18:07:44.722 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.08
2014-05-14T18:07:44.722 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.09
2014-05-14T18:07:44.722 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.10
2014-05-14T18:07:44.722 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.11
2014-05-14T18:07:44.723 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.12
2014-05-14T18:07:44.723 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.13
2014-05-14T18:07:44.723 ERROR find_expired_indices:201 Could not find a valid timestamp from the index: .marvel-2014.05.14
2014-05-14T18:07:44.723 INFO index_loop:309 DELETE index operations completed.
2014-05-14T18:07:44.723 INFO main:379 Done in 0:00:00.037963.
[root@node05 github]# ls -ltr /var/lib/elasticsearch/bigdatadev/nodes/0/indices/.marvel
/var/lib/elasticsearch/bigdatadev/nodes/0/indices/.marvel-2014.05.10:
total 8
drwxr-xr-x. 2 elasticsearch elasticsearch 4096 May 10 01:00 _state
drwxr-xr-x. 5 elasticsearch elasticsearch 4096 May 10 01:00 0

/var/lib/elasticsearch/bigdatadev/nodes/0/indices/.marvel-2014.05.11:

[untergeek]

Marvel

@deepakas your prefix for marvel is missing the -. A prefix is supposed to be everything up to the date portion. See the help output where it shows logstash-:

curator --help
…
  -p PREFIX, --prefix PREFIX
                        Prefix for the indices. Indices that do not have this
                        prefix are skipped. Default: logstash-
…

With that, your marvel pruning command-line should read:

/usr/lib/python2.6/site-packages/curator/curator.py -d 3 -p .marvel-

Flume

Your Flume serializer is using a different separator. You can use the SEPARATOR configuration option to change that:

curator --help
…
  -s SEPARATOR, --separator SEPARATOR
                        Time unit separator. Default: .
…

So for that instance, you'd use a line more like:

/usr/lib/python2.6/site-packages/curator/curator.py -d 3 -p raw_json- -s \-

Being a dash, you may need to escape it with a backslash as above.

[untergeek]

Will be closing this at the one month mark if no further comments.

[faxm0dem]

this was helpful: I also had the issue while forgetting the -

[untergeek]

Updated the wiki FAQ to link to and explain this better. Closing.