We currently have a data pipeline setup where we are reading raw data from a single Kafka topic using Logstash and write it to ElasticSearch.
The data in this topic is in JSON format, but each row can belong to a completely different business domain, so it may have completely different schema. For example:
record 1: "{"id":1,"model":"model2","updated":"2017-01-1T00:00:00.000Z","domain":"A"}
record 2: "{"id":"some_compound_key","result":"PASS","domain":"B"}
You can see that not only is the schema different, but it is actually conflicting (id is an integer in the first record, and a string in the second).
There are only two guarantees - each record is a valid JSON record, and each has a "domain" field. Even records with the same domain value can sometimes have different schemas.
We now have a requirement to enrich and transform this data as it goes through the pipeline (instead of doing it later with an ETL), and we're looking into several ways of accomplishing it. The caveat is that since the data has no unified schema, the transformation needs to be done on a row by row basis:
1) Continue using Logstash - it is possible to model the transformation pipeline that we need, per domain, using a set of Logstash filters and conditionals.
It is also easy to maintain and deploy since Logstash reloads the configuration periodically in runtime, so to change/add transformation logic we only need to drop a new config file in the conf directory.
The downside however is that it is very hard to enrich data with Logstash from external sources.
2) Use Kafka Streams - This seems like an obvious choice since it integrates well with Kafka, allows joining data from multiple streams (or external sources) and has no schema requirements - it is easy to transform the data row by row.
Here the downside is that it is difficult to modify the transformation logic in runtime - we need to either recompile and redeploy the app, or wrap it with some API that would generate and compile Java code in runtime, or some other complex solution.
3) Use Spark Streaming - we are already using Spark for batch processing, so it would be great if we could use it for streaming as well to keep our stack as simple as possible.
However, I'm not sure if Spark can even support streaming data that doesn't have a single schema, nor if it's possible to perform transformations on a per-row basis.
All of the examples I've seen (as well as our own experience with Spark batch processing) assume that the data has a well defined schema, which is not our use case.
Can anyone shed some light on whether what we need is possible with Spark Streaming (or Structured Streaming), or should we stick with Logstash / Kafka Streams?