Where is an Avro schema stored when I create a hive table with 'STORED AS AVRO' clause?

Viewed 13418

There are at least two different ways of creating a hive table backed with Avro data:

  1. Creating a table based on an Avro schema (in this example, stored in hdfs):

    CREATE TABLE users_from_avro_schema ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ('avro.schema.url'='hdfs:///user/root/avro/schema/user.avsc');

  2. Creating a table by specifying hive columns explicitly with STORED AS AVRO clause:

    CREATE TABLE users_stored_as_avro( id INT, name STRING ) STORED AS AVRO;

Am I correct that in the first case the metadata of users_from_avro_schema table are not stored in Hive Metastore, but inferred from the SERDE class reading the avro schema file? Or maybe the table metadata are stored in the Metastore, added on table's creation, but then what is the policy for synchronising hive metadata with the Avro schema? I mean both cases:

  1. updating table metadata (adding/removing columns) and
  2. updating Avro schema by changing avro.schema.url property.

In the second case when I call DESCRIBE FORMATTED users_stored_as_avro there is no avro.schema.* property defined, so I don't know which Avro schema is used to read/write data. Is it generated dynamically based on the table's metadata stored in the Metastore?

This fragment of Programming Hive book discusses inferring info about columns from the SerDe class, but on the other hand HIVE-4703 removes this from deserializer info form columns comments. How can I check then what is the source of column types for a given table (Metastore or Avro schema)?

2 Answers
Related