UICollectionViewCompositionalLayout minimum item size

I think the way to solve this kind of problem is to solve it first for the most degenerate, simple case you can think of, and then add any desired refinements.

So let's ignore all the stuff about spacing and ask ourselves: how would we make a simple grid of square cells 150-or-more on a side? These cells will touch the sides of the collection view and will touch one another on all sides; there is no spacing of any kind in this degenerate case.

Here's a way to do that. We will use a section provider function so that we know what the size of the collection view is:

let compositionalLayout = UICollectionViewCompositionalLayout(sectionProvider: { 
    (sectionIndex, environment) -> NSCollectionLayoutSection? in

How many of these squares can fit in the width of the collection view? That's a simple problem in integer division:

    let w = environment.container.effectiveContentSize.width
    let max = Int(w) / 150

Now, we're going to dictate the number of items per row as a number (max), so the item width doesn't matter; we'll use a dummy value. The item height will be the group height, and we'll cross that bridge when we come to it:

    let itemSize = NSCollectionLayoutSize(
        widthDimension: .absolute(10), // this is a dummy value
        heightDimension: .fractionalHeight(1))
    let item = NSCollectionLayoutItem(layoutSize: itemSize)

Okay, let's cross that bridge. The group height needs to be the same as an item width, so as to make the cells square. An item width will be the collection view width divided by the number of items per group; that is the number we already calculated as max:

    let groupSize = NSCollectionLayoutSize(
        widthDimension: .fractionalWidth(1),
        heightDimension: .absolute(w/CGFloat(max))) // square
    let group = NSCollectionLayoutGroup
        .horizontal(layoutSize: groupSize, subitem: item, count: max)

And we're done:

    let section = NSCollectionLayoutSection(group: group)
    return section

Try it out. It will help if your cell has a border so you can see where the cell edges are. You will see that we now have square cells larger than 150, both in portrait and in landscape, though of course the cell dimension is different in portrait than in landscape.

Okay, so having solved the problem for the degenerate case, let's think about a more sophisticated case. Let's posit a space of 10 points between everything: a margin of 10 round the outside, plus 10 points vertically and horizontally between cells. All we have to do is add three lines:

    group.interItemSpacing = .fixed(10)
    let section = NSCollectionLayoutSection(group: group)
    section.interGroupSpacing = 10
    section.contentInsets = .init(top: 0, leading: 10, bottom: 0, trailing: 10)

That actually looks pretty good, but our math is no longer working correctly. We can see this because the cells are no longer quite square.

So let's just insert the insets and spacings into our math. The maximum width of our cell area is now 160, not 150, because the cells are going to be robbed of 5 points on each side. And the width of the collection view must be reduced by 10 points, because it's going to robbed of 10 points on a side, which is 20, but 10 of those points are put back by the calculation we just did. So, carrying that through everywhere, we end up with this:

let compositionalLayout = UICollectionViewCompositionalLayout(sectionProvider: {
    (sectionIndex, environment) -> NSCollectionLayoutSection? in
    
    // how many 150-or-larger squares can fit in our width?
    let w = environment.container.effectiveContentSize.width - 10
    let max = Int(w) / (150 + 10)
    
    let itemSize = NSCollectionLayoutSize(
        widthDimension: .absolute(10), // this is a dummy value
        heightDimension: .fractionalHeight(1))
    let item = NSCollectionLayoutItem(layoutSize: itemSize)
    
    let groupSize = NSCollectionLayoutSize(
        widthDimension: .fractionalWidth(1),
        heightDimension: .absolute(w/CGFloat(max) - 10)) // square
    let group = NSCollectionLayoutGroup
        .horizontal(layoutSize: groupSize, subitem: item, count: max)
    
    group.interItemSpacing = .fixed(10)
    
    let section = NSCollectionLayoutSection(group: group)
    
    section.interGroupSpacing = 10
    section.contentInsets = .init(top: 0, leading: 10, bottom: 0, trailing: 10)
    
    return section
})

That looks pretty good! Here's a more general version without the magic numbers. We start by extracting all the magic numbers as properties, plus I've generalized by changing some of the numbers (the margin):

var square = 150
var padding : CGFloat = 10
var margin : CGFloat = 16

Our layout now looks like this:

let compositionalLayout = UICollectionViewCompositionalLayout(sectionProvider: {
    [self] (sectionIndex, environment) -> NSCollectionLayoutSection? in
                
    let w = environment.container.effectiveContentSize.width - CGFloat(margin)*2 + padding
    let max = Int(w) / (square + Int(padding))
    
    let itemSize = NSCollectionLayoutSize(
        widthDimension: .absolute(10), // this is a dummy value
        heightDimension: .fractionalHeight(1))
    let item = NSCollectionLayoutItem(layoutSize: itemSize)
    
    let groupSize = NSCollectionLayoutSize(
        widthDimension: .fractionalWidth(1),
        heightDimension: .absolute(w/CGFloat(max) - padding)) // square
    let group = NSCollectionLayoutGroup
        .horizontal(layoutSize: groupSize, subitem: item, count: max)
    
    group.interItemSpacing = .fixed(padding)
    
    let section = NSCollectionLayoutSection(group: group)
    
    section.interGroupSpacing = padding
    section.contentInsets = .init(top: 0, leading: margin, bottom: 0, trailing: margin)
    
    return section
})

That does seem to do what we wanted! Just to make sure, I've written a test case for it:

let vc = ViewController(collectionViewLayout: UICollectionViewFlowLayout())
vc.loadViewIfNeeded()
vc.collectionView.register(UICollectionViewCell.self, forCellWithReuseIdentifier: "cell")
for _ in 1..<3 {
    vc.margin = CGFloat(Int.random(in:7..<20))
    vc.padding = CGFloat(Int.random(in:7..<20))
    vc.square = Int.random(in:140..<170)
    var foundMin = false
    for w : CGFloat in stride(from: 280, to: 500, by: 1) {
        for h : CGFloat in stride(from: 280, to: 800, by: 10) {

            vc.view.frame = CGRect(x: 0, y:0, width: w, height: h)
            vc.view.layoutIfNeeded()
            let cell = vc.collectionView.visibleCells.first!
            let sz = cell.bounds.size
            XCTAssertEqual(sz.width, sz.height, accuracy: 0.1)
            XCTAssertGreaterThanOrEqual(sz.width, CGFloat(vc.square))
            if sz.width == CGFloat(vc.square) { foundMin = true }
        }
    }
    XCTAssertTrue(foundMin)
}

The idea here is to show that our cells are always square and that they are sometimes the desired minimum but never less — which is exactly the condition you imposed in the terms of the problem. We use dependency injection to randomize the initial conditions.