Alternatives to NDJSON?

Mascarpone 0
Viewed 32

I need a disk serialization format which:

  • Supports utf8 or binary data

  • Supports multiple messages per file (like newline delimited json)

  • (possibly) Is schemaless

  • (possibly) Has both a node and a rust implementation

I couldn't find a way for msgpack or CBOR to support multiple messages per file in go, although it's supported by cbor (sequences). At the moment I'm playing with asn.1 and it seems nice but I was wondering if there was a better alternative.

go serialization
1 Answers

I made it work both with asn.1 and cbor, I need to try msgpack now

package main

import (
    "bytes"
    "github.com/fxamacker/cbor/v2"
    "log"
)

func main() {

    type Record struct {
        Payload string
        Counter int
    }

    r1 := Record{
        "hello", 1}
    r2 := Record{
        " world", 2}

    var buff []byte

    b, err := cbor.Marshal(r1)
    if err != nil {
        log.Fatal(err)
    }
    buff = append(buff, b...)

    b, err = cbor.Marshal(r2)
    if err != nil {
        log.Fatal(err)
    }
    buff = append(buff, b...)

    log.Println(buff)

    var out1, out2 Record
    decoder := cbor.NewDecoder(bytes.NewReader(buff))

    err = decoder.Decode(&out1)
    if err != nil {
        log.Fatal(err)
    }
    log.Println(out1)

    err = decoder.Decode(&out2)
    if err != nil {
        log.Fatal(err)
    }
    log.Println(out2)

}
Mascarpone0
Related