I am trying to build a scene in my 2D game which has a panel whose elements are arranged as in this image:

The ScrollRect should dynamically add rows after users provide values to the InputField, select an option from the Dropdown, and then click the Button. This all works fine, except that the viewable area of for the ScrollRect has the same vertical size as its siblings within the parent panel.
This happens because the GridLayoutGroup of the parent panel requires the child cell sizes to be specified, and if I specify a value which makes sense for the top two panels it is too short for the ScrollRect. Putting a ContentSizeFitter on the parent panel does not help (also, despite Unity warning against it, there is a ContentSizeFitter on the content of the ScrollRect since without one the newly added rows will not appear when scrolling).
So my question is:
what do I need to do to allow the vertical sizing of the children of the top level panel to be based on their sizes, when one child's vertical size will change dynamically?
Thanks!