How to compare types of different, generic HLists?

Viewed 87

So I have a user defined input of keys, and the user is also expected to provide serializers for these keys. I am trying to statically ensure that the serializers provided work with the user provided types. Example user usage would be something like:

case class Keys(key1: String, key2: Int)

val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, IntSerializer)

toSerializedKeys(keys, keyDescriptor)

but I would like something like this to fail at compile time:

val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, StringSerializer)

toSerializedKeys(keys, keyDescriptor)

To abstract over arity and attempt to provide compile time type matching on the keys/serializers I'm trying to use shapeless. My current solution works fine given explicit HLists. But as soon as I try to apply this to generic HLists things fall apart.

Current solution:

Assume we have the following user provided serializers:

trait SerializerFn[T] {
  def toBytes(t: T): ByteBuffer
}

object StringSerializerFn extends SerializerFn[String] {
  override def toBytes(t: String): ByteBuffer = ???
}

object IntSerializerFn extends SerializerFn[Int] {
  override def toBytes(t: Int): ByteBuffer = ???
}

I currently create a type class, helper function, and instances to handle serialization of HLists:

trait Serializer[HA <: HList, HB <: HList] {
  def toBufs(keys: HA)(fns: HB): List[ByteBuffer]
}

object Serializer extends SerializerInstances {
  def toBufs[HA <: HList, HB <: HList](keys: HA)(fns: HB)(
    implicit 
    serializer: Serializer[HA, HB]
  ): List[ByteBuffer] = {
    serializer.toBufs(keys)(fns)
  }
}

trait SerializerInstances {
  implicit val hnilSerializer: Serializer[HNil, HNil] =
    new Serializer[HNil, HNil] {
      override def toBufs(keys: HNil)(fns: HNil): List[ByteBuffer] = Nil
    }

  implicit def hconsSerializer[H, Fn <: SerializerFn[H], TA <: HList, TB <: HList](
    implicit
    hconsSerializer: Serializer[TA, TB],
  ): Serializer[H :: TA, Fn :: TB] =
    new Serializer[H :: TA, Fn :: TB] {
      override def toBufs(keys: H :: TA)(fns: Fn :: TB): List[ByteBuffer] =
        List(fns.head.toBytes(keys.head)) ::: hconsSerializer.toBufs(keys.tail)(fns.tail)
    }
}

The static type checking I'm going for here is happening in hconsSerializer where we restrict the head of both HLists to be an H and SerializerFn[H], respectively.

And to test it, this compiles fine:

val testKeys = "3" :: "4" :: HNil
val testFns = StringSerializerFn :: StringSerializerFn :: HNil

val testSerializedKeys = Serializer.toBufs(testKeys)(testFns)

and as expected, this doesn't compile:

val testKeys = "3" :: "4" :: HNil
val testFns = StringSerializerFn :: IntSerializerFn :: HNil

val testSerializedKeys = Serializer.toBufs(testKeys)(testFns)

However, when trying to generalize this (so that we can actually use this in a constructor that we provide to users), things fall apart, (I've omitted the Generic conversion from the user provided Product to HList for brevity):

def toSerializedKeys[ReprA <: HList, ReprB <: HList](
  keys: ReprA,
  fns: ReprB
): List[ByteBuffer] = 
  Serializer.toBufs(keys)(fns)

For reference, intellij complains that the implicit Serializer can't be found, and explicitly passing hconsSerializer shows:

Required: Serializer[ ReprA, ReprB]
Found:    Serializer[Nothing :: HNil, Nothing :: HNil]

and Scalac says:

[error]  found   : [H, Fn <: com.twitter.beam.io.manhattan.SerializerFn[H], TA <: shapeless.HList, TB <: shapeless.HList]com.twitter.beam.io.manhattan.Serializer[H :: TA,Fn :: TB]
[error]  required: com.twitter.beam.io.manhattan.Serializer[ReprA,ReprB]

What kind of implicit evidence do I need to be able to perform this kind of recursive type checking on generic HLists? Is accomplishing something like this even possible? I'm new to Shapeless.


Also, if comparing HList types like this isn't possible. Another solution could be something that allowed the user to define individual implicit key serializers, instead of a full KeyDescriptor that we convert to an HList, e.g something like:

implicit val intSerializer = IntSerializerFn
implicit val strSerizlizer = StringSerializerFn

However, I tried something like this and then tried mapping over the Keys HList with a Poly1 that takes an implicit SerializerFn, but I ran into similar issues when trying to work with a generic HLists.

2 Answers

If you are using Scala3 (aka dotty), here is a simple solution using stdlib only for your purpose:

import scala.deriving.Mirror.ProductOf as PF
import java.nio.ByteBuffer

trait SerializerFn[T]:
  def toBytes(t: T): ByteBuffer

object StringSerializerFn extends SerializerFn[String]:
  override def toBytes(t: String): ByteBuffer = ???

object IntSerializerFn extends SerializerFn[Int]:
  override def toBytes(t: Int): ByteBuffer = ???

case class SI(key1: String, key2: Int)

case class SII(key1: String, key2: Int, key3: Int)

def toSerializedKeys[K <: Product, S <: Tuple](keys: K, sers: S)
                                              (using pf: PF[K], ev: S <:< Tuple.Map[pf.MirroredElemTypes, SerializerFn]): Unit = {}

// success
toSerializedKeys(SI("a", 1), (StringSerializerFn, IntSerializerFn))
toSerializedKeys(SII("a", 1, 2), (StringSerializerFn, IntSerializerFn, IntSerializerFn))

// compile fail
// toSerializedKeys(SI("a", 1), (StringSerializerFn, StringSerializerFn))
// toSerializedKeys(SII("a", 1, 2), (IntSerializerFn, IntSerializerFn, IntSerializerFn))

I guess you just missed evidence serializer: Serializer[ReprA, ReprB] in

def toSerializedKeys[ReprA <: HList, ReprB <: HList](
  keys: ReprA,
  fns:  ReprB
): List[ByteBuffer] =
  Serializer.toBufs(keys)(fns)

So just replace this with

def toSerializedKeys[ReprA <: HList, ReprB <: HList](
  keys: ReprA,
  fns:  ReprB
)(implicit serializer: Serializer[ReprA, ReprB]): List[ByteBuffer] =
  Serializer.toBufs(keys)(fns)

Then

case class Keys(key1: String, key2: Int)

val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, IntSerializer)

toSerializedKeys(keys, keyDescriptor)

compiles (as expected) and

val keys = Keys("Foo", 2)
val keyDescriptor = (StringSerializer, StringSerializer)

toSerializedKeys(keys, keyDescriptor)

doesn't (as expected).

Just noticed that Alec advised the same in comments.

Related