Ensuring consistency with Kafka Schema and OpenAPI specification

user_3011570 0
Viewed 614

A number of API/microservices provide access to critical resources including Kafka topics. The API/microservice messages are validated using an OpenAPI specification that defines the API/microservice contract. Once the microservice validates the message it is published to a Kafka topic, at which point the message is (again) validated against Kafka's schema registry.

The problem is that there are two message definitions upon which messages are validated (the OpenAPI spec and the Kafka's schema registry) and it is a challenge to ensure both message definitions are in sync.

With this in mind, I have a few questions:

  • is there a way to convert OpenAPI specs to Kafka schema registry format (and vice-versa)?
  • is there a way to allow Kafka to verify against an OpenAPI spec instead of registry (probably not a great solution as native Kafka capabilities should be used)?
  • is there a way to allow an API/Microservice to validate its messages against a Kafka schema instead of OpenAPI spec (again, probably not a good approach since OpenAPI specs are the standard approach to define messages for APIs)?

Lastly, which of the above makes the most sense. Are there any other better alternatives?

confluent-schema-registry microservices openapi apache-kafka
1 Answers

After many attempts to solve this problem I have concluded that there is an architecture approach that will allow JSON Schema, OpenAPI specs, and Schema Registry to work well together (I have a prototype working but it is too complex to show here). Here is the general approach...

First a few facts that underpin my approach:

  • Fact: OpenAPI (v3.1+) is a superset of JSON Schema; the request and response definitions are 100% compliant to JSON Schema
  • Fact: Schema Registry supports JSON Schema; JSON Schemas appear to be first class citizens: they can be stored and queried in/by Schema Registry; And libraries (although not easily found) are available that allow JSON Schema to work with Kafka (consumers, producers, KSQLDB for Confluent) and its connectors

The implication of this is that JSON Schema would need to be the primary way to define messages flowing through rather than AVRO (I think this may not be looked upon very fondly by the Kafka community). But by using JSON Schema is allows OpenAPI, Schema Registry, and JSON Schema to play nice together!

In other words a generalized solution would have:

  • OpenAPI becomes the primary way of defining APIs that frequently are front-ends in an event streaming solution
  • JSON Schemas define the OpenAPI request/response formats
  • OpenAPI specs, with a modest bit of tooling (several are available), OpenAPI specs can be assembled/bundled from JSON Schema parts, so now OpenAPI and JSONN Schemas are finally harmonized
  • Since API request/response form "events" in an event mgmt system, and there definition is independent of the OpenAPI spec, JSON schemas now define events
  • And finally, since JSON Schemas are in Schema Registry, all events are defined in a Kafka-friendly way and accessible in any normal Kafka fashion
user_30115700
Related