I have an Elasticsearch cluster on AWS ec2 instance. It is t3.small with 2 vcores and 2 GB ram. I have installed Elasticsearch & kibana. For extensions, I have installed Heartbeat and Metricbeat. The database I'm working with is mongo DB and all my data is no-SQL. I feed my engine from my MongoDB cluster which is in my local machine with a script. I feed my engine and run the queries from my app and also from the console. So far so good. Everything is fine. Well, the cluster is always yellow it is not green.
The problem starts after hitting multiple requests on the engine. After 50 or 60 search queries the data just disappears. Well somehow my engine is forcefully dumping my indices and it's not being able to restore those data ( obviously I have no snapshot and no restore point ) and I keep getting lose those data. I have to manually feed the engine again and again. Well at first I had 1 GB ram so I thought upgrading would fix the issue but after upgrading to 2 GB ram it didn't stop. Well, now the data stays there for some more time.
So here are my DB configs.
- I have 70K + no SQL documents.
- Which contains text and geo_point types
- I make post request on my engine through my front end application.
- I don't have logstash installed, but metricbeat is not showing any error logs.
- All my elastic search engine setup is for
Testing purposesthis isnot the production mode. - We will upgrade when we go to the production mode.
So I need to know
- What is the reason behind this and
- how to prevent this huge data loss
So please help me or just suggest how to solve this huge problem.
Thank you