Currently, we have have ElasticSearch instance with around 100m+ records that contain a lat/lng, created_at, item_id and user_id. We would like to answer these questions:
1) How many item_id are in a nearby X,Y with a radius of Z in the last T days?
2) How many unique users are a nearby X,Y with a radius of Z in the last T days?
3) How many total items, and unique users are a nearby X,Y with a radius of Z in the last T days, group by month?
We've tried MongoDB, MySQL and now ElasicSearch and we're getting no where in terms of performance (constantly under 5s). Is the only solution to move to a data-ware house model here like Redshift? Anyone have any recommendations?
Below is the search query to get
GET items/item/_search
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"geo_distance": {
"distance": "10mi",
"loc": {
"lat": 40.712784,
"lon": -74.005941
}
}
},
{
"range": {
"cat": {
"gte": "now-1d/d",
"lte": "now/d"
}
}
}
]
}
}
}
},
"aggs": {
"distinct_users": {
"cardinality": {
"field": "uid.hash"
}
},
"distinct_checkins": {
"cardinality": {
"field": "iid.hash"
}
}
},
"size": 1
}