Use MetalView with SwiftUI? How do I put something to display in there?

user_5637988 0
Viewed 2032

I'm stuck with SwiftUI and Metal up to the point of being about to give up.

I got this example from https://developer.apple.com/forums/thread/119112?answerId=654964022#654964022 :

import MetalKit
struct MetalView: NSViewRepresentable {
    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }
    func makeNSView(context: NSViewRepresentableContext<MetalView>) -> MTKView {
        let mtkView = MTKView()
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.enableSetNeedsDisplay = true
        if let metalDevice = MTLCreateSystemDefaultDevice() {
            mtkView.device = metalDevice
        }
        mtkView.framebufferOnly = false
        mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
        mtkView.drawableSize = mtkView.frame.size
        mtkView.enableSetNeedsDisplay = true
        return mtkView
    }
    func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) {
    }
    class Coordinator : NSObject, MTKViewDelegate {
        var parent: MetalView
        var metalDevice: MTLDevice!
        var metalCommandQueue: MTLCommandQueue!
        
        init(_ parent: MetalView) {
            self.parent = parent
            if let metalDevice = MTLCreateSystemDefaultDevice() {
                self.metalDevice = metalDevice
            }
            self.metalCommandQueue = metalDevice.makeCommandQueue()!
            super.init()
        }
        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        }
        func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            let rpd = view.currentRenderPassDescriptor
            rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
            rpd?.colorAttachments[0].loadAction = .clear
            rpd?.colorAttachments[0].storeAction = .store
            let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
            re?.endEncoding()
            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }
    }
}

... but I can't get my head around how to use this MetalView(), which does seem to work when I call it from a SwiftUI view, to display data. I want to use it to display a CIImage which will be filtered and manipulated with CIFilters...

Can someone please point me in the right direction on how to tell this view how to display something? I think I need it to display the content of a texture but tried countless hours and ended up starting from scratch for more countless times...

This is how I run my image filters now but it results in very slow sliders, which is why I decided to try learning about Metal... but it's been really time-consuming and. frustrating due to the lack of documentation...

func ciExposure (inputImage: CIImage, inputEV: Double) -> CIImage {
    let filter = CIFilter(name: "CIExposureAdjust")!
    filter.setValue(inputImage, forKey: kCIInputImageKey)
    filter.setValue(inputEV, forKey: kCIInputEVKey)
    return filter.outputImage!
}

I think I need to take that filter.outputImage and pass it on to the MetalView somehow?

Any help is really, really appreciated...

core-image metal swiftui swift macos
3 Answers

Apple's WWDC 2022 contained a tutorial/video entitled "Display EDR Content with Core Image, Metal, and SwiftUI" which describes how to blend Core Image with Metal and SwiftUI. It points to some new sample code entitled "Generating an Animation with a Core Image Render Destination" (here).

This sample project is very CoreImage-centric (which should suit your purposes nicely), but I wish Apple would post more sample-code examples showing Metal integrated with SwiftUI.

user_140953880

I have a small Core Image + SwiftUI sample project on Github that might be a good starting point for you. It doesn't cover a lot yet, but it demonstrates how to display filtered camera frames already.

Especially check out the draw function of the view. It's used to render a CIImage into the MTKView (you can do the same in your delegate's draw function).

user_5410160

Ok so this does the trick for me:

func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            
            let colorSpace = CGColorSpaceCreateDeviceRGB()

            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            
            let rpd = view.currentRenderPassDescriptor
            rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
            rpd?.colorAttachments[0].loadAction = .clear
            rpd?.colorAttachments[0].storeAction = .store
            
            let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
            re?.endEncoding()
                
            context.render((AppState.shared.rawImage ?? AppState.shared.rawImageOriginal)!,
                to: drawable.texture,
                commandBuffer: commandBuffer,
                bounds: AppState.shared.rawImageOriginal!.extent,
                colorSpace: colorSpace)
            
            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }

AppState.shared.rawImage is my CIImage texture I got from my filtering function.

The context is made somewhere else but should be:

context = CIContext(mtlDevice: metalDevice)

Next up is adding the centering part of the code provided by Frank Schlegel.

user_56379880
Related