The "problem" is straightforward, now let me see if I can describe it in words.
You've got two image views showing, say, the basil image: the one in the scroll view, and the full-screen one when you present. They are both set at Aspect Fill. But the aspect ratio itself is different for these two image views. Therefore, in order to be displayed in Aspect Fill in a different image view, the image shifts in the way it is shown.
The reason you find this upsetting at the end of the dismiss is that you don't notice it happening at the start of the present, but if you slow down your video (you can turn on Slow Animations on the simulator, it will help), you can see it there too:
When you tap an image in the scroll view, there is a jump as the image changes its display for the new image view, and then the image view grows to its true size as the transform is removed (animated).
When you tap the full screen image, the transform is applied (animated) and then there is a jump as the image view is taken away to reveal the old image view that was behind it.
If you make both image view Scale To Fill, that won't happen. Of course then there is a distortion of the image, but it is the same distortion so that the image appears to grow and shrink smoothly right the way through. Try it and you will at least be satisfied by the sense of what I'm saying.
(See https://www.apeth.com/stretch.gif.)
If you don't want to do that, then you will have a lot more work to do; you will need to figure out how to actually compensate for the difference in aspect ratio by changing the aspect ratio of the image to match. Or, you could make the goal size of the fullscreen image view match the aspect ratio of the small image view (and part of it will just end up offscreen).
The point is, you cannot display the same image using Aspect Fill in image view with different aspect ratios and expect the image to appear the same way in both. Either the image needs to be different (to compensate) or the aspect ratios need to be the same.