What is the advantage of storing schema in avro?

Question

We need to serialize some data for putting into solr as well as hadoop.

I am evaluating serialization tools for the same.

The top two in my list are Gson and Avro.

As far as I understand, Avro = Gson + Schema-In-JSON

If that is correct, I do not see why Avro is so popular for Solr/Hadoop?

I have searched a lot on the Internet, but cannot find a single correct answer for this.

Everywhere it says, Avro is good because it stores schema. My question is what to do with that schema?

It may be good for very large objects in Hadoop where a single object is stored in multiple file blocks such that storing schema with each part helps to analyze it better. But even in that case, schema can be stored separately and just a reference to that is sufficient to describe the schema. I see no reason why schema should be part of each and every piece.

If someone can give me some good use case how Avro helped them and Gson/Jackson were insufficient for the purpose, it would be really helpful.

Also, official documentation at the Avro site says that we need to give a schema to Avro to help it produce Schema+Data. My question is, if schema is input and the same is sent to output along with JSON representation of data, then what extra is being achieved by Avro? Can I not do that myself by serializing an object using JSON, adding my input schema and calling it Avro?

I am really confused with this!

user_2250246 · Answer

Having schema is good from schema evolution perspective but isn't something that cannot be done in JSON. Its exactly the same thing with JSON as in Avro.

Main thing with Avro is that it's binary. So let us say, your record is:

{
  {"name": "emp_name", "type":"string"},
  {"name":"dob", "type":"string"},
  {"name":"age", "type":"int"}
}

Avro will store the data portion for the above as: "John"1990102032

Here is some kind of marker to demarcate between the field-values.

So without the schema, the above data is useless ! You don't know what those values mean without the schema.

That's the difference between binary and non-binary formats. Binary formats have just the values and that makes them super compact. Text formats like JSON and XML have schema repeated over and over again, thus making them more than 2x bulkier than binary.

Another comparison: An integer in binary takes only 32 bits but an integer in JSON form is a number stored in text format.

{"some_number": 43847384}

The above number in JSON is stored not like a 32-bit integer in JSON, it's all text and takes up 8 chars * 8 bits = 64 bits or even more depending on the length of the number.

So schema in JSON is not optional. It is a MUST if you have to work with binary formats.

What is the advantage of storing schema in avro?

3 Answers