Apache load balancer data syncing across multiple applications

user_6332430 0
Viewed 551

I have a setup where two instances of same application deployed behind a load balancer and both of the applications are talking to the same database.

The issue I'm having is that when data inserted into the database via one of the application, the very same data is not accessible/visible to the other application.

These two applications have exactly same configuration in terms of database abstraction framework, spring framework, etc. In addition, these two applications run on different tomcat servers on different physical servers.

What could be the cause to the issue?

Edit

Hibernate is configured as follow:

enter image description here

Edit 2

Here's the ehcache.xml

<cache name="org.hibernate.cache.internal.StandardQueryCache"
       maxElementsInMemory="10000"
       eternal="false"
       timeToLiveSeconds="86400"
       overflowToDisk="false"
       memoryStoreEvictionPolicy="LRU"/>

<defaultCache
        maxElementsInMemory="10000"
        eternal="false"
        timeToLiveSeconds="86400"
        overflowToDisk="false"
        memoryStoreEvictionPolicy="LRU"/>
ehcache load-balancing tomcat hibernate java
3 Answers

You are most probably experiencing cache replication issues, probably due to missing/incorrect cache coordination configuration.

Assumption

You have enabled both second level cache and query cache in Hibernate, which is fine of course and you got a performance boost. The problem is that you use 2 nodes, each one having its own cache or they don't invalidate caches correctly.

For example, node A performs a query for all entities Z (i.e. list of product categories: Z1, Z2, ...) and caches the result. Later on, node B adds a new product category entity Zn. Node A will not use that new entity until its own cache is invalidated; until then it will use the cached result set which does not include Zn.

You can verify this assumption if the problem goes away when you disable query caching (i.e. set <property name="hibernate.cache.use_query_cache">false</property>).

Solutions

Option A

Optimize cache invalidation. This greatly depends on your business needs but some suggestions would be:

  • Reduce cache TTL for certain queries until you find a sweet spot (time span you can tolerate missing/extra values). You can use EhCache regions for that.
  • Disable caching altogether for certain entities

Example:

Create a region for short-lived (i.e. 5 minutes) query results in ehcache.xml:

<cache name = "short-lived"
maxElementsInMemory = "500"
eternal = "false"
timeToIdleSeconds = "600"
timeToLiveSeconds = "600"
overflowToDisk = "false"
/>

For specific queries that need to be frequently refreshed:

Query query = session.createQuery("FROM Z");
query.setCacheable(true);
query.setCacheRegion("short-lived");

More on the topic here.

Option B

This is the real solution. You can have an shared cache for all nodes. While it's definitely harder to setup and configure, it gives you the best possible performance. For example, cached queries will run once per cluster, not once per node as it is now.

Take a look at Terracotta.

user_15051460

There are different types of possibility of causing this issue.

1) Check for the database access privileges. Also check is your application is able to commit the data into database. You need to debug different scenarios

  1. Insert data from cluster_1 and try to read the same from cluster_2
  2. Repeat step 1 with different order
  3. Check in SQL database are there any difference while inserting data(owner info,object version).

2) You can create a cluster of tomcat server in (Clustering/Session Replication How-To)

user_68219360

Do you have any ORM in your application that is communicate with DB? I suppose that, so here is my answer: Each application has its own caching mechanism via ORM (like Hibernate). If one of them change data, changes will be reflected on its cache but the otherone doesn't know anything about changes in DB. You can do one of following options:

  1. Disable ORM cache in application (Quickest solution)
  2. Wrap DB with another application that act as model layer + cache server
  3. Create a messaging system to broadcast change event between instances and update cache in case of trigger. (not a good idea though)
user_127365400
Related