I've written a screen-recording app that writes out H.264 movie files using VideoToolbox and AVWriter. The colors in the recorded files are a bit dull compared to the original screen. I know that this if because the colorProfile is not stored in the video-file.
This is closely related to How to color manage AVAssetWriter output
I have created a testbed to show this on github ScreenRecordTest
If you run this app you can start recording using CMD-R and stop it using the same CMD-R (You have to start and stop recording once to get a fully written movie-file). You will find the recording in your /tmp/ folder under a name similar like this: "/tmp/grab-2018-10-25 09:23:32 +0000.mov"
When recording the app shows two live images: a) the frame gotten from CGDisplayStream -and- b) the cmSampleBuffer that came out of the compressor.
What I found out is that the IOSurface that is returned from CGDisplayStream is not color-managed, so you'll notice the "dull" colors already before compression. If you un-comment line 89 in AppDelegate.swift
// cgImage = cgImage.copy(colorSpace: screenColorSpace)!
this live-preview will have the correct colors. Now this is only for displaying the IOSurface before compression. I have no idea how to make the other live-preview (after compression) (line 69 in AppDelegate) show correct colors (say: how to apply a colorProfile to a CMSampleBuffer) or most important how to tag the written video-file with the right profile so that when opening the .mov file I get the correct colors on playback.