segmentio/parquet fails to read files written with Julia and/or Python

Juanbuntu 0
Viewed 31

The code below, which was taken almost verbatim from the segmentio/arrow, fails to read .parquet files written with Python and/or Julia libs. When the code returns from the call to parquet.ReadFile("file") the rows contain 0 values for int64 or "" for strings. The reading fails with codec = {ZSTD, GZIP, or SNAPPY}

type FiRowType struct{ x1, x2, x3 int64 }
func RdFiFile() {
    rows, err := parquet.ReadFile[FiRowType]("fileName_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for _, c := range rows {
        fmt.Printf("%+v\n", c)
    }
}

type FsRowType struct{ x1, x2, x3 string }
func RdFsFile() {
    rows, err := parquet.ReadFile[FsRowType]("fileName_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for _, c := range rows {
        fmt.Printf("%+v\n", c)
    }
}

The Golang code does not return an error, i.e., err == nil. The code returns the right number of rows and columns, some info in the metadata seems to be correct (like the originator of the file, date of creation etc). I created the files using Julia:

using Parquet
function WrForGo( )
  min = 1 
  max = 10
  # arrays of size (10,3). ai is int and as is 
String 
  ai = Array{Int64, 2}(undef, 10,3)
  as = Array{String, 2}(undef, 10,3)
  for i =1:max
      for j=1:3
        as[i,j] = string(i, pad=2) * "_" * string(j,pad=2)
        ai[i,j] = (j-1)*10 + i
      end
    end

    dfi = DataFrame(ai, :auto) ; dfs = DataFrame(as, :auto)
    print( dfi ) ;  print( dfs )
    Parquet.write_parquet( prqDir * "fi_ZSTD.parquet", compression_codec = "ZSTD", dfi)
    Parquet.write_parquet( prqDir * "fs_ZSTD.parquet", compression_codec = "ZSTD", dfs)

    Parquet.write_parquet( prqDir * "fi_GZIP.parquet", compression_codec = "GZIP", dfi)
    Parquet.write_parquet( prqDir * "fs_GZIP.parquet", compression_codec = "GZIP", dfs)

    Parquet.write_parquet( prqDir * "fi_SNAPPY.parquet", compression_codec = "SNAPPY", dfi)
    Parquet.write_parquet( prqDir * "fs_SNAPPY.parquet", compression_codec = "SNAPPY", dfs)

end
go parquet python
1 Answers

In Go variables that start with a lower case character not exported so cannot be updated from other packages (e.g. segmentio/parquet-go). Try the below:

package main

import (
    "fmt"
    "log"

    "github.com/segmentio/parquet-go"
)

type FiRowType struct {
    X1 int64 `parquet:"x1,optional"`
    X2 int64 `parquet:"x2,optional"`
    X3 int64 `parquet:"x3,optional"`
}

func RdFiFile() {
    rows, err := parquet.ReadFile[FiRowType]("fi_ZSTD.parquet")
    if err != nil {
        log.Fatal(err)
    }
    for i, c := range rows {
        fmt.Printf("%d %+v\n", i, c)
    }
}

func main() {
    RdFiFile()
}
Brits0
Related