In our application we have an elasticsearch cluster that is shared for all of our clients.
The types of requests made against this cluster are computationally intense and may take minutes to complete. Because of the type of data in the cluster, the types of requests we get, and the irregularity at which they are used, we can't predict when these requests will be made or do any caching before hand. If multiple clients make requests at the same time, they will experience slower response speeds than normal.
For most clients this isn't an issue. Their data isn't large enough to make a noticeable difference (3s -> 10s occasionally isn't a big deal). But for larger clients, the time difference may be minutes and is very noticeable.
What we'd like, above all else, is consistency- even if these operations were slower on average. To do that, we'd like to give these special clients dedicated nodes, while all other clients use the shared nodes. At a first glance, it seems the only way to do that is to create a dedicated cluster.
But this adds overhead in any application that interfaces with elasticsearch to first do a lookup for the cluster to route to. Ideally, we'd be able to create dedicated nodes within the cluster. This way, the application doesn't need to be aware of cluster routing, and the index configuration can be shared between these "virtual clusters".
Each document has a client id, which we can use to distribute across nodes using _routing. But this has its own problems. Firstly, this doesn't allow us to create a generic default cluster. Secondly, this may mean the dedicated nodes share client data with other clients- the goal is to get consistent speeds by removing node resource contention. And finally, this doesn't allow us to allocate multiple nodes for a given route.
Is there a way to create a routing rule that nodes can be added to explicitly. e.g. I want to add 3 nodes to routing key 582123.
Is there a way to create a default routing rule for nodes that don't match existing routes? If not we could always have an explicit default route. We'll still need to do a route lookup application side but it would still reduce complexity in a multi-cluster scenario.