kotlin write string to orc file with apache orc java

jake wong 0
Viewed 16

I'm using apache orc 1.8. Following the short example in the documentation here: https://orc.apache.org/docs/core-java.html I'm failing to be able to write string out to the orc file.

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
import org.apache.orc.OrcFile
import org.apache.orc.TypeDescription
import org.apache.orc.Writer


    val conf = Configuration()
    val schema = TypeDescription.fromString("struct<x:string,y:string>")
    val writer = OrcFile.createWriter(
        Path("./my-file.orc"),
        OrcFile.writerOptions(conf)
            .setSchema(schema)
    )



    val batch = schema.createRowBatch()
    val x = batch.cols[0] as BytesColumnVector
    val y = batch.cols[1] as BytesColumnVector
    for (r in 0..99) {
        val row = batch.size++
        x.vector[row] = r.toString().toByteArray(Charsets.UTF_8)
        y.vector[row] = r.toString().toByteArray(Charsets.UTF_8)
        // If the batch is full, write it out and start over.
        if (batch.size == batch.maxSize) {
            writer.addRowBatch(batch)
            batch.reset()
        }
    }
    if (batch.size != 0) {
        writer.addRowBatch(batch)
        batch.reset()
    }
    writer.close()

When I use spark to read the orc file back with:

val df = spark.read().format("orc").load("./my-file.orc")
df.show()
df.printSchema()

it shows:

+---+---+
|  x|  y|
+---+---+
|   |   |
|   |   |
|   |   |
|   |   |

I dont understand what is the issue here. I think the error lies in this line: r.toString().toByteArray(Charsets.UTF_8). But i'm not sure what I can do to fix it.

Any ideas?

orc java kotlin
1 Answers

I found out "I think"

Made the following changes:

...
...
val schema = TypeDescription.createStruct().addField("x", TypeDescription.createString()).addField("y", TypeDescription.createString())

...
...
for (r in 0..99) {
        val row = batch.size++

        x.setVal(row, r.toString().toByteArray(Charsets.UTF_8))
}
jake wong0
Related