Take a Video with ARKIT

user_6773569 0
Viewed 2935

Hello Community,

I try to build a App with Swift 4 and the great upcoming ARKit-Framework but I am stuck. I need to take a Video with the Framework or at least a UIImage-sequence but I dont know how.

This is what I've tried:

In ARKit you have a session which tracks your world. This session has a capturedImage instance where you can get the current Image. So I createt a Timer which appends the capturedImage every 0.1s to a List. This would work for me but If I start the Timer by clicking a "start"-button, the camera starts to lag. Its not about the Timer i guess because If I invalidate the Timer by clicking a "stop"-button the camera is fluent again.

Is there a way to solve the lags or even a better way?

Thanks

arkit video-capture swift list
4 Answers

Use a custom renderer.

Render the scene using the custom renderer, then get texture from the custom renderer, finally covert that to a CVPixelBufferRef

- (void)viewDidLoad {
    [super viewDidLoad];

    self.rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    self.bytesPerPixel = 4;
    self.bitsPerComponent = 8;
    self.bitsPerPixel = 32;
    self.textureSizeX = 640;
    self.textureSizeY = 960;

    // Set the view's delegate
    self.sceneView.delegate = self;

    // Show statistics such as fps and timing information
    self.sceneView.showsStatistics = YES;

    // Create a new scene
    SCNScene *scene = [SCNScene scene];//[SCNScene sceneNamed:@"art.scnassets/ship.scn"];

    // Set the scene to the view
    self.sceneView.scene = scene;

    self.sceneView.preferredFramesPerSecond = 30;

    [self setupMetal];
    [self setupTexture];
    self.renderer.scene = self.sceneView.scene;

}

- (void)setupMetal
{
    if (self.sceneView.renderingAPI == SCNRenderingAPIMetal) {
        self.device = self.sceneView.device;
        self.commandQueue = [self.device newCommandQueue];
        self.renderer = [SCNRenderer rendererWithDevice:self.device options:nil];
    }
    else {
        NSAssert(nil, @"Only Support Metal");
    }
}

- (void)setupTexture
{
    MTLTextureDescriptor *descriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm_sRGB width:self.textureSizeX height:self.textureSizeY mipmapped:NO];
    descriptor.usage = MTLTextureUsageShaderRead | MTLTextureUsageRenderTarget;

    id<MTLTexture> textureA = [self.device newTextureWithDescriptor:descriptor];
    self.offscreenTexture = textureA;
}

- (void)renderer:(id <SCNSceneRenderer>)renderer willRenderScene:(SCNScene *)scene atTime:(NSTimeInterval)time
{
    [self doRender];
}

- (void)doRender
{
    if (self.rendering) {
        return;
    }
    self.rendering = YES;
    CGRect viewport = CGRectMake(0, 0, self.textureSizeX, self.textureSizeY);

    id<MTLTexture> texture = self.offscreenTexture;

    MTLRenderPassDescriptor *renderPassDescriptor = [MTLRenderPassDescriptor new];
    renderPassDescriptor.colorAttachments[0].texture = texture;
    renderPassDescriptor.colorAttachments[0].loadAction = MTLLoadActionClear;
    renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1.0);
    renderPassDescriptor.colorAttachments[0].storeAction = MTLStoreActionStore;

    id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];

    self.renderer.pointOfView = self.sceneView.pointOfView;

    [self.renderer renderAtTime:0 viewport:viewport commandBuffer:commandBuffer passDescriptor:renderPassDescriptor];

    [commandBuffer addCompletedHandler:^(id<MTLCommandBuffer> _Nonnull bf) {
        [self.recorder writeFrameForTexture:texture];
        self.rendering = NO;
    }];

    [commandBuffer commit];
}

Then in the recorder, set up the AVAssetWriterInputPixelBufferAdaptor with AVAssetWriter. And convert the texture to CVPixelBufferRef:

- (void)writeFrameForTexture:(id<MTLTexture>)texture {
    CVPixelBufferPoolRef pixelBufferPool = self.assetWriterPixelBufferInput.pixelBufferPool;
    CVPixelBufferRef pixelBuffer;
    CVReturn status = CVPixelBufferPoolCreatePixelBuffer(nil, pixelBufferPool, &pixelBuffer);
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
    void *pixelBufferBytes = CVPixelBufferGetBaseAddress(pixelBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
    MTLRegion region = MTLRegionMake2D(0, 0, texture.width, texture.height);
    [texture getBytes:pixelBufferBytes bytesPerRow:bytesPerRow fromRegion:region mipmapLevel:0];

    [self.assetWriterPixelBufferInput appendPixelBuffer:pixelBuffer withPresentationTime:presentationTime];
    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
    CVPixelBufferRelease(pixelBuffer);
}

Make sure the custom renderer and the adaptor share the same pixel encoding.

I tested this for the default ship.scn and it and it only consume 30% CPU compared to almost 90% compared to use snapshot method for every frame. And this will not pop up a permission dialog.

user_12107920
Related