Ignite cache query performance issue

user0Hhh1 0
Viewed 51

Configuration:

CPU 3 cores, RAM 10 GB and storage 1TB external. Ignite cluster nodes:5. Persistence Mode is enabled. Currently 80GB of storage is present. Number of backups: 1.

We see performance issues while querying data from one particular cache which has 15,00,00,000 records. This cache created as partitioned mode and the related data is collocated with the affinity key.

Earlier ignite cluster had 3 nodes and scaled them up to 5. After increasing it we could see the improvement in the performance with other caches. This particular cache query is not responding.

Also observed that one node is using 2 cores and 2 nodes are using less than 1 core and remaining 2 nodes are not using the CPU. All nodes are participating in the baseline topology and using 9GB of RAM.

Expecting suggestions on the configuration (increasing the nodes/RAM/CPU or changes to configuration) and improving the query performance.

ignite c#
2 Answers

Try to check if page replacement occurred. Jmx metric DataRegionMetrics.PagesReplaced() will show it. If page replacement started you may try to increase the amount of RAM available to the nodes.

Vladimir Malinovskii0

Since you use a custom affinity key and see that some of the nodes are idle during the query execution then it could be related to the uneven data distribution between the partitions and as result between the nodes. The data distribution between the partitions is related to the affinity key, e.g. OrganizationId is chosen as an affinity key for the Orders cache, in case there is one organization that has a majority of orders then all of them will be stored in one partition in one node, which will lead to higher load on this node.

You can check the entry count in each partition by using Java API:

org.apache.ignite.IgniteCache#sizeLong(int partition, CachePeekMode... peekModes)

And the partition distribution between the nodes by using the control.sh script:

control.sh --cache distribution null <CACHE_NAME>

In case the data distribution is uneven you can try to choose another field as an affinity key, which will give you a more even distribution, or you can introduce a surrogate key for that.

You can find more details regarding the data partitioning here.

Igor Belyakov0
Related