Once you’re running syslog-ng | logstash | elasticsearch for to archive all the syslogs from your servers, there is probably something you didn’t noticed: syslog don’t care about year.
So when 2015 coming, Logstash don’t know about it and push into elasticsearch the year of it’s last starts. So your January 2015 syslogs coming to January 2014 ones…
How I fixed it ? I’ve restarted logstash when I understood the problem, but it was somehow too late.
Some solution is explained here: https://discuss.elastic.co/t/syslog-date-without-year/29834 but you need some more code to handle the year move and using @timestamp don’t allow you to replay piece of logs when you need…

Your elasticsearch don’t want to accept connexions or queries anymore after a restart or a “Java heap” memory problem ?

# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elasticsearch",
"status" : "red",
[...]
}

You will need to check if one the of indexes is “red” status by requesting the indices list. The command is:

# curl 'localhost:9200/_cat/indices?v'
health status index                pri rep docs.count docs.deleted store.size pri.store.size
red    open   my_index-2014.07.21   3   1   10107061            0      1.7gb          1.7gb
yellow open   my_index-2014.07.22   3   1    6728769            0      1.1gb          1.1gb

What is not said in Elasticsearch documentation is that you will need to delete the datas of the index that is in “red” status to be able to restart elasticsearch. I didn’t find any tool to repair indexes… So datas are losts. You was putting them in the cloud, they gone to space…
That experience made me want to move to another noSQL database… To Be Continued 🙂