How do I make everything outside of a CAShapeLayer black with an opacity of 50% with Swift?

Your impulse to use a mask is correct, but let's think about what needs to be masked. You are doing three things:

• Dimming the whole thing. Let's call that the dimming layer. It needs a dark semi-transparent background.

• Drawing the white rounded rect. That's the shape layer.

• Making a hole in the entire thing. That's the mask.

Now, the first two layers can be the same layer. That leaves only the mask. This is not trivial to construct: a mask affects its owner in terms entirely of its transparency, so we need a mask that is opaque except for an area shaped like the shape of the shape layer, which needs to be clear. To get that, we start with the shape and clip to that shape as we fill the mask — or we can clip to that shape as we erase the mask, which is the approach I prefer.

In addition, your code has some major flaws, the most important of which is that your shape layer has no size. Without a size, there is nothing to mask.

So here, with corrections and additions, is your code; I made this the entirety of a view controller, for testing purposes, and what I'm covering is the entire view controller's view rather than a particular subview or sublayer:

override func viewDidLoad() {
    super.viewDidLoad()
    self.view.backgroundColor = .red
}
private var didInitialLayout = false
override func viewDidLayoutSubviews() {
    super.viewDidLayoutSubviews()
    if didInitialLayout {
        return
    }
    didInitialLayout = true
    let screenSize = UIScreen.main.bounds
    let cardLayer = CAShapeLayer()
    cardLayer.frame = self.view.bounds
    self.view.layer.addSublayer(cardLayer)

    let cardWidth = 350.0 as CGFloat
    let cardHeight = 225.0 as CGFloat
    let cardXlocation = (screenSize.width - cardWidth) / 2
    let cardYlocation = (screenSize.height / 2) - (cardHeight / 2) - (screenSize.height * 0.05)
    let path = UIBezierPath(roundedRect: CGRect(
            x: cardXlocation, y: cardYlocation, width: cardWidth, height: cardHeight),
            cornerRadius: 10.0)
    cardLayer.path = path.cgPath
    cardLayer.strokeColor = UIColor.white.cgColor
    cardLayer.lineWidth = 8.0
    cardLayer.backgroundColor = UIColor.black.withAlphaComponent(0.5).cgColor

    let mask = CALayer()
    mask.frame = cardLayer.bounds
    cardLayer.mask = mask

    let r = UIGraphicsImageRenderer(size: mask.bounds.size)
    let im = r.image { ctx in
        UIColor.black.setFill()
        ctx.fill(mask.bounds)
        path.addClip()
        ctx.cgContext.clear(mask.bounds)
    }
    mask.contents = im.cgImage
}

And here's what we get. I didn't have a preview layer but the background is red, and as you see, the red shows through inside the white shape, which is just the effect you are looking for.

The shape layer can only affect what it covers, not the space it doesn't cover. Make a path that covers the entire video and has a hole in it where the card should be.

let areaToDarken = previewLayer.bounds // assumes origin at 0, 0
let areaToLeaveClear = areaToDarken.insetBy(dx: 50, dy: 200)
let shapeLayer = CAShapeLayer()
let path = CGPathCreateMutable()
path.addRect(areaToDarken, ...)
path.addRoundedRect(areaToLeaveClear, ...)
cardLayer.frame = previewLayer.bounds // use frame if shapeLayer is sibling
cardLayer.path = path
cardLayer.fillRule = .evenOdd // allow holes
cardLayer.fillColor = black, 50% opacity