How do I use an Avro type in another Avro type, without defining it again?

user_1202506 0
Viewed 2022

I have defined an Avro type called "some.package.SourceMetadata" in a json file called commonSourceMetadata.avsc:

{
  "type": "record",
  "name": "SourceMetadata",
  "namespace": "some.package",
  "fields": [..]
}

and I am trying to use it in another avsc file, using an imports keyword I found in Apache Avro 1.7.3:

  "imports": ["commonSourceMetadata.avsc"],
  "fields": [
    {
      "name": "sourceMetadata",
      "type": "some.package.SourceMetadata"
    },

I also tried:

  "fields": [
    {
      "name": "sourceMetadata",
      "imports": ["commonSourceMetadata.avsc"],
      "type": "some.package.SourceMetadata"
    },

The Apache Avro version is 1.9.

Still I get:

org.apache.avro.SchemaParseException: "some.package.SourceMetadata" is not a defined name. The type of the "sourceMetadata" field must be a defined name or a {"type": ...} expression.
    at org.apache.avro.Schema.parse(Schema.java:1637)

It is also clear to me that it does not take into account imports or import because I tried to import an inexistent file and no complains there.

How can i make it a defined name?

We are using Scala, SBT and the AvroHugger plugin to generate Scala case classes from the avsc Avro type definitions.

avro-tools avro
1 Answers

The Avro schema format (in JSON, usually with the .avsc extension) is defined by the specification.

The Avro IDL format is a different, non-JSON syntax, designed to be familiar for developers writing RPC protocols.

The import keyword is only supported in IDL. This is intentional, since sharing a schema (JSON) shouldn't require also sharing a filesystem.

It looks AvroHugger provides two ways to do what you want:

  1. Simply include all avsc files when generating your case classes and AvroHugger should be able to resolve the names, or
  2. Use the IDL format (also supported by AvroHugger) which permits the import keyword.
user_14375930
Related