Scale some object but not others in a Fabric group?

I have a group that contains a text object and a rectangle object. When the group is scaled I want the rectangle object to change size but the text object to remain the same size.

Any way to accomplish this?