Delete unused data elasticsearch

user_4104887 0
Viewed 2227

I'm new in using elasticseach. I to use elasticsearch to aggregate logs. My problem is with the storage, I deleted all indices and now I have only one index.

When I call /_cat/allocation?v disk.indices is 23.9mb and disk.used is 16.4gb. Why is this difference? How can I remove unused data or how can I remove properly indices?

I ran the command:

curl -XPOST "elasticsearch:9200/_forcemerge?only_expunge_deletes=true"

But I didn't see any improvement.

Output of _cat/allocation?v :

shards disk.indices disk.used disk.avail
12       24.3mb    16.4gb     22.7gb

Output of _cat/shards?v :

index        shard prirep state      docs  store ip           node
articles     0     p      STARTED    3666 24.2mb 192.168.1.21 lW9hsd5
articles     0     r      UNASSIGNED                          
storage_test 2     p      STARTED       0   261b 192.168.1.21 lW9hsd5
storage_test 2     r      UNASSIGNED                          
storage_test 3     p      STARTED       0   261b 192.168.1.21 lW9hsd5
storage_test 3     r      UNASSIGNED                          
storage_test 4     p      STARTED       0   261b 192.168.1.21 lW9hsd5
storage_test 4     r      UNASSIGNED                          
storage_test 1     p      STARTED       0   261b 192.168.1.21 lW9hsd5
storage_test 1     r      UNASSIGNED                          
storage_test 0     p      STARTED       0   261b 192.168.1.21 lW9hsd5
storage_test 0     r      UNASSIGNED                          
twitter      3     p      STARTED       1  4.4kb 192.168.1.21 lW9hsd5
twitter      3     r      UNASSIGNED                          
twitter      2     p      STARTED       0   261b 192.168.1.21 lW9hsd5
twitter      2     r      UNASSIGNED                           
twitter      4     p      STARTED       0   261b 192.168.1.21 lW9hsd5
twitter      4     r      UNASSIGNED                          
twitter      1     p      STARTED       0   261b 192.168.1.21 lW9hsd5
twitter      1     r      UNASSIGNED                          
twitter      0     p      STARTED       0   261b 192.168.1.21 lW9hsd5
twitter      0     r      UNASSIGNED                          
.kibana      0     p      STARTED       4 26.4kb 192.168.1.21 lW9hsd5

Thanks

elasticsearch
2 Answers
user_10503890

You might be facing some side effects of a _forcemerge on an a non-read-only index:

Warning: Force merge should only be called against read-only indices. Running force merge against a read-write index can cause very large segments to be produced (>5Gb per segment), and the merge policy will never consider it for merging again until it mostly consists of deleted docs. This can cause very large segments to remain in the shards.

In this case I would suggest to first make the index read-only:

PUT your_index/_settings
{
  "index": {
    "blocks.read_only": true
  }
}

Then to do force merge again and enable back writing to the index:

PUT your_index/_settings
{
  "index": {
    "blocks.read_only": false
  }
}

In case this does not work, you can do a reindex from an old index into a new index and then delete the old index.

Is there a better way of deleting old logs?

Looks like you want to delete old log messages. Although you could issue a delete by query, there is in fact a better way: using Rollover API.

The idea is to create a new index every time the old index gets too big. Writes will happen into a fixed alias, and Rollover API will make alias point into a new index when the old one is too old or too big. Then to delete the old data you will only have to delete the old indexes.

Hope that helps!

user_50959570
Related