How to boot up a Microservice from events

user_2911506 0
Viewed 117

Say I have a shop application and I want to make some complicated validations for that operation.

  1. Events are the single source of truth in my system.
  2. Adding a product is represented by a ProductAdded message.
  3. The microservice responsible for validating the product reads a message, validates it, and produces a ProductValidated message.

But what happens if I want to microservice to boot up from zero? On bootup, each and every message is reprocessed, resulting in a redundant and duplication of validation for each consumed message. This could be solved by first reading all messages from the messaging queue and when all messages are loaded, start an asynchronous process of validation.

But how can it ensure that all messages are loaded? maybe messages are produced quicker than the process of building the state from events. A solution could be querying the messaging queue for the total number of messages at a given moment. Then, reading all of them and process them. Then, query and process again.

The problem with this one is that it doesn't seem to me like a typical solution for this challenge. I want to find out what is a popular practice to do in this situation.

microservices apache-kafka event-driven message-queue
1 Answers

You have few options:

A KTable, which you aggregate by shopping cart (each shopping cart cannot have the same product twice). To prevent this from growing too big, records need to be 'tombstoned', so another thing needs to tell the app that a shopping cart is no more.

Remember that to do any kind of aggregation in Kafka, you need local storage. If you don't want or cannot have local storage, Kafka is the wrong tool.

I don't fully understand your points under But there is a problem with the microservice validation process. First it says no caching or local storage, and the second point says load everything (which implies caching in local storage).

--- Edit

You can check is this example from Confluent that does a validation on orders: https://github.com/confluentinc/kafka-streams-examples/tree/5.4.1-post/src/main/java/io/confluent/examples/streams/microservices .

If I understand you correctly, you can have a local storage that doesn't have a changelog, so you can re-populate it on restart.

Check the class InventoryService.java, there you can see how to create a separe store. The line you want to omit is .withLoggingEnabled(), as that creates a changelog topic.

final StoreBuilder reservedStock = Stores
  .keyValueStoreBuilder(Stores.persistentKeyValueStore(RESERVED_STOCK_STORE_NAME),
    Topics.WAREHOUSE_INVENTORY.keySerde(), Serdes.Long())
builder.addStateStore(reservedStock);

The 2 other things you'll need to do are:

  • Configure the stream to go back to the ealiest record config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
  • Have a bit of code that finds the store and wipes it before you build the stream. Check this blog post by Confluent, the section Local State Stores, which explains part of how to find the directory where the local files are stored, so you can wipe the directory.
user_6088200
Related