Parallel processing in multi server environment, Java

user_3227576 0
Viewed 107

I have a system which receives a lot of messages (by JMS) from a provider. The messages are consumed by currently up to 4 servers. The provider sends some of these messages doubled and at the same millisecond... some messages depends on each other. For example one message contains the customer information, the next about his products.

Currently the software, which is deployed on up to 4 servers, listens to the queue and processes the message. Processing means some mapping and storing the data into our database, which is a Oracle or PostGres Database (depends on the installation, but always a relational database). Concurrent modifications are detected by optimistic locking and we have an generic retry mechanism which fixes this issues, when they occur. So no big pain on updates.

But inserts are a problem... for example we receive 2 customer messages in the same ms... the software checks if the customer exists... if it not, it creates it... but if this happens at the same time, both of them try to create the customer. We solved this issue by DB-Constraints (the customerNumber must be unique) or in more complicated cases (where a DB-Constraints is not suitable) by pessimistic locking over a database table (an entry for insert is generated in the locking table, the other insert thread fails to create this entry and knows, that a insert is ongoing and stops it's processing). This worked so far... i guess not the best and prettiest solution, but it worked.

Problem is our processing load grows, we find more and more issues where inserts duplicates data... which causes a lot of problems with the rest of our business logic and must be prevented... the current way is to secure these methods with pessimistic locking by using locking db table...

Currently i'm not sure if we are going into the right direction and may be our design is wrong from the ground... can anyone offer me alternative ideas for this issue? May be some design patterns with advantages and disadvantages?

Our current alternative is to group and load balance the messages (for example with KAFKA). This should make sure that the messages of one customer are processed by one thread => so the concurrent insert is not possible. Opinions about this?

Environment: Java (EJB), Wildfly, Oracle/PostGre DB

parallel-processing java
1 Answers

One of the options is to group tasks so that 1 server processes a 100 of requests, 2nd server will process the next 200 requests. You can even group them by customer name so that 2 servers don't concurrently work with the same customer.

This can be done without Kafka, with plain old relational DB. You just need to insert your tasks into DB. Then each service could SELECT from the table like this (this is PG syntax, Oracle can support the same with small changes):

select * from tasks 
order by customer 
limit 100
for update skip locked

This allows to:

  1. Batch the tasks in groups
  2. Group tasks by fields (like customer)
  3. Lock in progress records (for update)
  4. And allow other services to skip and go to the next batch (skip locked)

But this does require some delay to gather a batch. On the other hand batch processing is usually more efficient, so you may end up handling requests faster.

user_8866970
Related