:EDIT:

Three viable options seem to exist, so I'll cover those and then you can choose whatever works best for you. Ultimately I still think the root cause here is that you're trying to interact with the physics object (Rigidbody) while at the same time enabling Unity to interact with the physics object. The ways around this all revolve around either disabling Unity's ability to interact with it (fixing the constraints), or removing your interaction (adding a FixedJoint), or trying to not interact with it at the same time (moving your interaction to Update).

Option 1: Fixed Joint

Instead of doing the steps to update positions and velocities, just add a FixedJoint and then remove that joint on drop:

private FixedJoint fixedJoint;
public void BreakConnection()
{
    Destroy(fixedJoint);
}

public void PickupObject()
{
    fixedJoint = gameObject.AddComponent<FixedJoint>();
    fixedJoint.connectedBody = currentlyPickedUpObject.GetComponent<Rigidbody>();
}

Option 2: Move Positioning Code to Update

If you do this in Update() instead of FixedUpdate() then this also appears to remove the jitter, but there's a chance that you might not be getting the physics interactions that you want, so definitely test with this before committing to it:

private void Update()
{
    fixedJoint.connectedBody
    if (PickupParent != null)
    {
        currentDist = Vector3.Distance(PickupParent.position, pickupRB.position);
        currentSpeed = Mathf.SmoothStep(minSpeed, maxSpeed, currentDist / maxDistance);
        currentSpeed *= Time.fixedDeltaTime;
        pickupRB.transform.position = PickupParent.position;
        Vector3 direction = PickupParent.position - pickupRB.position;
        pickupRB.velocity = direction.normalized * currentSpeed;
    }
}

Option 3: Apply Constraints

This would require you to continue to do the updating (which you can avoid by adding a FixedJoint, in Option 1 above) but doesn't create the joint, so it leaves the pickupParent alone. Adding a FixedJoint will link the pickup and parent together which may result in pushback on the parent if/when the pickup collides with something.

You can just use the all/none constraint options or you could cache the existing constraints and reset them on break. I'll include that option here because it's (a tiny bit) more involved.

private RigidbodyConstraints priorConstraints;    // <--- NEW
public void BreakConnection()
{
    pickupRB.constraints = priorConstraints;    // <--- NEW
    currentlyPickedUpObject = null;
    lookObject = null;
    physicsObject.pickedUp = false;
    currentDist = 0;
    pickupRB.useGravity = true;
}

public void PickupObject()
{
    physicsObject = lookObject.GetComponentInChildren<PhysicsObjects>();
    currentlyPickedUpObject = lookObject;
    pickupRB = currentlyPickedUpObject.GetComponent<Rigidbody>();
    priorConstraints = pickupRB.constraints;    // <--- NEW
    pickupRB.constraints = RigidbodyConstraints.FreezeAll;    // <--- NEW
    physicsObject.playerInteractions = this;

    pickupRB.isKinematic = true;
    //  pickupRB.transform.position = PickupParent.position;
    pickupRB.transform.parent = PickupParent.transform;

    //StartCoroutine(physicsObject.PickUp()); 
}

:Original post:

I still think you're having issues because you're trying to manually override the physics system by setting the position and velocity. Try disabling the rigidbody when you pick it up and adding it as a child of the PickupParent. You'll keep whatever colliders are present and as long as the PickupParent has a ribidbody itself then this is a valid approach because rigibodies use the sum of all colliders on all child game objects.

You should only have one rigidbody on an object tree and it should be at the root level, so I'm assuming your pickup items are all root objects with no parents.

Since you're attaching them to you there really isn't anything to do in FixedUpdate anymore:

private void FixedUpdate()
{
    if (currentlyPickedUpObject != null)
    {
        //currentDist = Vector3.Distance(PickupParent.position, pickupRB.position);
        //currentSpeed = Mathf.SmoothStep(minSpeed, maxSpeed, currentDist / maxDistance);
        //currentSpeed *= Time.fixedDeltaTime;
        //pickupRB.transform.position = PickupParent.position;
        //Vector3 direction = PickupParent.position - pickupRB.position;
        //pickupRB.velocity = direction.normalized * currentSpeed;
    }
}

You haven't provided the code for pickup, but I believe you can effectively disable the rigidbody without deleting it by setting isKinematic to true, so change its parent and set that on pickup then remove the parent and clear isKinematic on BreakConnection.

void PickupObject()
{
    pickupRb.isKinematic = true;
    pickupRb.transform.parent = pickupParent.transform;
    // other stuff to do here
}

void BreakConnection()
{
    pickupRb.isKinematic = false;
    pickupRb.transform.parent = null;
    // other stuff to do here
}

If you're continuing to have jitter then the other thing to do would be to check the Interpolate setting on the pickupParent because that's the object you're snapping to (here and in your original code).

The problem is that you're trying to directly change the transform position in your code:

pickupRB.transform.position = PickupParent.position;

But it's not a right way to handle rigidbodies movement. It is the reason of shaking, and, moreover, it causes huge overhead due to recalculating all the rigidbody physics each time you set transform position directly. Instead of doing this, use rigidbody.MovePosition along with making the rididbody kinematic while it is attached to the player:

pickupRB.MovePosition(PickupParent.position)

To make it easier, you could also use the FixedJoint, as Chuck suggests