Downsample CIImage at initialization

I have an iOS app with a CoreImage pipeline in which I apply several processing steps (cropping/transforming/apply adjustments etc.) to a CIImage that's used to back an AVPlayer. The images are read from local files, usually through the CIImage.init(contentsOf: URL) initializer. I also implemented a caching mechanism on them because they need to be reused several times and recreating them is quite expensive.

Everything works great until I work with images up to 4K resolution; instead, if I use bigger images (for instance panoramic pictures from iPhone) rendering is very slow: I cannot even maintain a 30fps playback on the AVPlayer. Following this best practice article by Apple, I tried implementing a downsampling mechanism on large images before using them.

• First I tried a simple rescaling of the CIImage that's read from disk, but this doesn't seem to produce any benefit at all.

// Slower processing, less memory usage 
func CIDownsampleImage(from url: URL, maxPixelSize: CGFloat = 3840) -> CIImage {
  let image = CIImage(contentsOf: url)!
  let scaleFactor = maxPixelSize / max(image.extent.width, image.extent.height)
  guard scaleFactor > 1 else { return image }
  let transform = CGAffineTransform(scaleX: scaleFactor, y: scaleFactor)
  return image.transformed(by: transform)
}

• Second, I tried using ImageIO to get a downsized version of the image, follwing, among other, this very useful post. This indeed seems to improve the performance problem, but it causes a potential big memory overhead compared to the other approach. As I said, I want to cache images for reuse; now that CIImage is backed by a CGImage, the CGImage with the actual bitmap data needs to be retained as well and this can easily consume several dozens of MB per image (in particular, since we are talking about large images).

// Faster processing, but need to keep a CGImage in memory
func CGSourceDownsampleImage(from url: URL, maxPixelSize: CGFloat = 3840) -> CIImage {
  let source = CGImageSourceCreateWithURL(url as CFURL, nil)!
  let options: [CFString: Any] = [
    kCGImageSourceThumbnailMaxPixelSize: maxPixelSize,
    kCGImageSourceCreateThumbnailFromImageAlways: true
  ]
  let cgImage = CGImageSourceCreateThumbnailAtIndex(source, 0, options as CFDictionary)!
  return .init(cgImage: cgImage)
}

Is there any other approach to keep both the performance improvement and the absence of memory overhead? I was wondering if there's a way to build a CIImage that, when rendered, processes just an already downsampled version of the image in the file. For instance, I saw there is a CIImage.init(contentsOf: URL, options: [CIImageOption: Any]?) init, but there doesn't seem to be a CIImageOption to set a rescale factor or similar.

Another possibility would be to produce an already downsampled version of the image into a separate file, but I would prefer to try the other approach if possible.