ElasticSearch composite agreggation with after_key and size

user_6310743 0
Viewed 2446

I have question. If I use histogram composite aggregation with setting size. For example I have integer range. Each number bucket in aggregation represents 1 day (Using integer for date is not important).

If I use size and say I want to get 10 buckets in result. Can I reliably use after_key to select date of my agregation?

Because if I filter by specific date range I will also get buckets outside of filtered range because some documents will be there since histogram is created on interval field (Same is true for date histogram on data_range field type).

I my idea is to use combination of after_key to set real start date and size to pick how many days of result I want.

What worries me is from documentation:

The after_key is usually the key to the last bucket returned in the response, but that isn’t guaranteed. Always use the returned after_key instead of derriving it from the buckets.

Will I miss some buckets if is say: after_key: 10 size: 5 after_key: 15 size: 5

Or maybe better question will be, what are the cases when last returned bucket does not match next after_key?

This would be usefull also for Kibana to display time correctly, but also some buckets may represent 30 years of range and only 1 month is usually queried. So if I can paginate properly I can save a lot.

elasticsearch aggregation composite
1 Answers

As far as I understand You want to have scroll functionality with composite aggregation. I am right the you are on the right track.

Think of it as offset key in SQL but instead of providing the row number you are specifing the last row values.

It works in a way that while doing composite aggregation it will skip the rows till the row mentioned in the after_key and return the next set or length size

The Only Place you may miss/overshot/etc some data in case new documents are index between those two calls. Like for your first 5 docs you get from 1 to 5 then some new documents indexed and are inserted/lies between those first results. You can miss those as in you second call you are asking for records after 5.

Otherwise you are fine.

I use composite aggregation alot as it help read aggregated data that is not possible with normal search API.

Just keep Track of the after_key and when the length of the buckets become zero you can close the loop.

user_88561690
Related