I was using ELK stack with Filebeat. My setup was like, I was having my log files on one server and ELK stack on another remote server.

server1 - Log files with Filebeat installed.
server2 - ElasticSearch, Logstash and Kibana installed.

I was parsing my log files using Filebeat and sending those logs to Logstash using Filebeat with Logstash having configuration as shown -

Everything was correct and I was able to create Index in ElasticSearch with name "file1". Then I was testing something and I had to create a new index for the same log files. So, I created a new index called "file2". Here arises the problem, I was not able to see my new index. Then I tried to check configuration files for Logstash and Filebeat both. Everything was correct. I even deleted my old index "file1". Still, I wasn't able to create a new Index. Where was the problem? I kept on hitting the internet for a solution but no luck. I tried every single thing. I debugged Filebeat, Logstash, ElasticSearch. Still, no problem. Then I thought of modifying my log files. And as soon as I modified my log file, I was able to see new index created in ElasticSearch. Then I got to know that it's some bug. I first thought that it's a bug in ElasticSearch so I thought of sending a PR in ElasticSearch repo. Then I thought I should first ask on ElasticSearch discussions. And after asking on ElasticSearch discussions I got to know that it's not a bug. It's one of the features of Filebeat.

The problem was that Filebeat tails files, so unless you append data or create a new file it will not process any data and thus not create any new index. That is why when I was modifying my log file I was able to see new index created in ElasticSearch. So, if you expect the file to be reprocessed you will need to modify the registry file where Filebeat keeps track of files processed.