Can I use the same state.dir for multiple instances in Kafka streams?

user_2534090 0
Viewed 2438

My use case is to get the latest value for any key in a Kafka topic. For this, I am currently using GlobalStateStore.

The problem is that, if I have different application.id for each instance, a different state dir is being created.

What I would want is to reuse the existing state store dir across all instances to avoid duplicating the data. This is because, all of my application instances run on the same machine, so there is no point of having different state directories.

I have done the following:

Use the same application.id and created 2 stream instances, The 2 instances pointing to the same state.dir but the state store instances are different (java objects are different) statestore1 and statestore2

and then as a test I have done the following:

while(true)
{
 new Thread(()-> stateStore1.get(key)).start();
 new Thread(()-> stateStore2.get(key)).start();
}

and then I started producing values in the topic...

I have observed that all the state store instances (statestore1 and statestore2) are updating the same state store (i.e. same state.dir) (since that is the code)

I did not encounter any problems in the tests doing that. But.. I suppose that a write lock should be acquired before writing the data to the state.dir. Now, if for some reason, statestore1 did not release the lock, then statestore2 will be going on waiting?

Is my above approach safe or are there any other ways?

Update:

The use-case is that there is some common data which I would want to share across different processes (different JVM instances). Since the data is common there is no need to have a global store for each process (since it would be redundant).

All processes should have the ability to read and write to that store because each process is independent of each other. So if one process goes down, the other can do its work (store the latest value in the global store and get it when needed).

apache-kafka-streams apache-kafka
2 Answers

If you run all on the same machine, and you want to share a global store, you should not use multiple instances, but multiple threads within an instance.

Global stores are designed to be replicated per instance, and this is exactly what you want to avoid.

user_49530790

First of all, the state_dir takes the parameter of the directory where a folder with the value of application.id will be created.

So, for example, if the state_dir is C:\tmp and the application.id is test, then the folder will be C:\tmp\test

So, if the state_dir is given to be C:\tmp\test rather than C:\tmp with the same application.id or even a different application.id, a new directory will be created in the C:\tmp\test\ as C:\tmp\test\test or C:\tmp\test\some_other_application_id.

These doesn't conflict with each other. There will still be multiple global stores.

However, if the state_dir is given to be C:\tmp for both of the KafkaStreams instances, then there will be an exception stating

Could not lock global state directory. This could happen if multiple KafkaStreams instances are running on the same host using the same state directory

For the use case of having a common place for all the instances, it is better to go with a database as sink to Kafka (either using the Kafka connect API or writing from streams to a database manually).

If the use case doesn't require complex SQL-like queries, then you can have your own RocksDB which will be writing to the same directory for all instances running on the same machine.

Kafka doesn't seem to support such a use-case out-of-the-box.

user_25340900
Related