How to fix Weapon Rotation in Unity? (2D)

I'm having issues with a rotation script I have, what the script does it's simple, it rotates the player graphics weapon to look at the mouse, and change the scale of it if the player looks at the left or right. This used to work fine but when I added additional sprites to the player meaning, if he looks up, the body will move up a little bit, same for when looking down (If you see the video you'll understand what I mean)

Well the problem is that when the player looks down, since the body also moves a bit, the character starts having a seizure. This doesn't happen when looking up.

I've recorded a short video to show the problem.

https://streamable.com/c0p5ls

Here's the code:

public class Rotating : MonoBehaviour

{

public PlayerController player;
public float x;


public Vector3 flipArm;
public Transform playerT;
public Transform rotatingPivot;

public State state;

public enum State
{
    Normal,
    Frozen
}

void Start()
{
    x = transform.localScale.x;
    flipArm = transform.localScale;
    player = GetComponentInParent<PlayerController>();

}

void Update()
{
    switch(state)
        {
            case State.Normal:
            Rotate();
            break;

        }
}

private void Rotate()
{

    Vector3 difference = Camera.main.ScreenToWorldPoint(Input.mousePosition) - transform.position;
    difference.Normalize();

    float rotZ = Mathf.Atan2(difference.y, difference.x) * Mathf.Rad2Deg;

    transform.rotation = Quaternion.Euler(0f, 0f, rotZ + 0);
    //print(difference.x);

    Vector3 aimLocalScale = Vector3.one;
    if (difference.x >= 0)
    {
        //transform.rotation = Quaternion.Euler(0f, 0f, rotZ);
        aimLocalScale.x = +1f;
        aimLocalScale.y = +1f;
        player.armRight = true;
    }
    else
    {
        //transform.rotation = Quaternion.Euler(0f, 0f, rotZ+180);
        aimLocalScale.y = -1f;
        aimLocalScale.x = -1f;
        player.armRight = false;
    }
    transform.localScale = aimLocalScale;
}

}

I'm aware this is being caused by the weapon and how it's located on the arm, when I didn't have those body sprites of looking up or down, this problem didn't happen.

I've included some screenshots on how the weapon moves when looking up or down, hopefully it helps for something.

BodyNormal:

BodyUp

BodyDown

Yet I've tried some solutions and they didn't work, if anyone could provide a solution I would appreciate it very much, but also if possible an explanation of why this is happening since I fail to comprehend why this is.

Thanks.