How to update RecyclerView item without animation?

I have a RecyclerView. When I click a button inside an item in RecyclerView, I want to change the color of a View in that item. The following is my code and it works fine. But, the problem is the item will have an animation which is ugly. I want to update the item without the animation. How should I do that? By the way, I don't want to turn off the animation, only for this click event.

  public class ViewHolder extends RecyclerView.ViewHolder implements View.OnClickListener {
        public ImageView imageView;
        public Button button;

        public ItemViewHolder(View view) {
            super(view);
            //do something
            button.setOnClickListener(this);
        }

        @Override
        public void onClick(View view) {
            //change color
            notifyItemChanged(getAdapterPosition());
        }
    }