How to implement unsafePartsOf using lens

I'm trying to implement a simple version of unsafePartsOf with the specialized signature:

unsafePartsOf :: Traversal s t a b -> Lens s t [a] [b]

After trying to understand how it works, I come up with this (repl).

myUnsafePartsOf :: Traversal s t a b -> Lens s t [a] [b]
myUnsafePartsOf traverse = lens getter setter
  where
    -- getter :: s -> [a]
    getter s = s ^.. traverse
    -- setter :: s -> [b] -> t
    setter s xs = evalState (s & traverse %%~ assignElem) xs
    assignElem _a = state $ \case
        [] -> error "list not long enough"
        (b : bs) -> (b, bs)

I think that using lens :: (s -> a) -> (s -> b -> t) -> Lens s t a b to separate the definition getter and setter will make them easier to understand.

However, this function doesn't type check because ^.. operator (toListOf) has a (specialized) signature of toListOf :: Traversal' s a -> s -> [a] where the original s and t (as well as a and b) would get unified.

I then tried to implement a more general version of toListOf (the same repl) with the following code:

toListOf :: Traversal s t a b -> s -> [a]
toListOf traverse s = execState (traverse getElem s) []
  where getElem a = modify (a:)

but failed again because that would require to unify b with () due to the usage of modify. I understand that there is no way for me to create an arbitrary b in the getElem so this function is impossible.

Note that it's actually possible to define the safe partsOf using lens in a similar way:

myPartsOf :: Traversal' s a -> Lens' s [a]
myPartsOf t = lens getter setter
  where
    getter s = s ^.. t
    -- setter :: s -> [a] -> s
    setter s xs = evalState (s & t %%~ assignElem) xs
    assignElem a = state $ \case
        [] -> (a, [])
        (x : xs) -> (x, xs)

Is it possible to define unsafePartsOf using lens? If so how can I do it?