Custom view shifts when applied its starting position to itself later

If I store my custom view's starting position, but later ask it to go to that position (fx via animation), it goes to an offset position. This is really annoying.

class MyCustomView(context: Context, attributeSet: AttributeSet) :
    ImageView(context, attributeSet) {

    private val binding: CustomViewBinding =
        CustomViewBinding.inflate(LayoutInflater.from(context), this, true)

    private var startX: Float = binding.toggleImage.x

    init {
        setOnClickListener {
            binding.toggleImage.x = startX
        }
    }
}