Pure red came back as (234, 0, 2): getting color right with AVAssetWriter

Shotnix records the screen with ScreenCaptureKit, writes H.264 with AVAssetWriter, and exports edited videos through Core Image. In September I measured what happens to colors on the way through, and two things were wrong:

• Recordings shifted colors. Pure red came back as (234, 0, 2) and pure green as (20, 255, 8).
• Exports came out lighter than the editor's preview. Mid-gray 128 came back as 146.

Neither was a bitrate problem. Both were Rec.709 meaning one thing on the way in and another on the way out.

Measure with solid colors

Eyeballing color doesn't work, so the test does a round trip. It fills frames with solid colors, encodes them with the recorder's own settings, decodes them with AVAssetImageGenerator, and averages a patch in sRGB with CIAreaAverage:

let colors: [(UInt8, UInt8, UInt8)] = [(255, 0, 0), (0, 255, 0), (0, 0, 255), (128, 128, 128), (255, 200, 0), (0, 122, 255)]
// ...six frames of each color through RecordingEngine.videoSettings, then:
let image = try await generator.image(at: time).image
let ci = CIImage(cgImage: image)
var px = [UInt8](repeating: 0, count: 4)
context.render(
    ci.applyingFilter("CIAreaAverage", parameters: [kCIInputExtentKey: CIVector(cgRect: ci.extent.insetBy(dx: 60, dy: 60))]),
    toBitmap: &px, rowBytes: 4,
    bounds: CGRect(x: 0, y: 0, width: 1, height: 1),
    format: .RGBA8,
    colorSpace: CGColorSpace(name: CGColorSpace.sRGB)
)
// Each channel must come back within 3 levels of what went in.

The frames going in are tagged the way ScreenCaptureKit tags sRGB frames, so the test sees what the real recorder sees.

Recordings: tell the writer what it's writing

The recorder's H.264 settings had no AVVideoColorPropertiesKey. The frames themselves were tagged, so the file still got primaries and a transfer function from them. What it didn't get was a YCbCr matrix. The encoder turned RGB into YCbCr with the BT.709 matrix and didn't write that down, so on the way back the decoder fell back to BT.601.

You can check that on paper. Pure red through BT.709 is Y′ 0.2126 and Cr 0.5. BT.601 turns it back into R = 0.2126 + 1.402 × 0.5 = 0.914, which is 233, within a level of the 234 the test measured. Green works out to (20, 255, 8) the same way.

Gray came through untouched, and that's the tell. Grays have no chroma, so the matrix never touches them. Only saturated colors moved.

The fix has two halves. ScreenCaptureKit delivers the frames as sRGB, and that's now explicit:

streamConfig.colorSpaceName = CGColorSpace.sRGB

And the writer is told the output is standard HD video color, Rec.709:

AVVideoColorPropertiesKey: [
    AVVideoColorPrimariesKey: AVVideoColorPrimaries_ITU_R_709_2,
    AVVideoTransferFunctionKey: AVVideoTransferFunction_ITU_R_709_2,
    AVVideoYCbCrMatrixKey: AVVideoYCbCrMatrix_ITU_R_709_2,
],

Now the matrix is written into the file, and the decoder uses the same one the encoder did. Recorded colors round-trip within 1 to 3 levels. The camera writer got the same properties.

Exports: don't render into Rec.709

The exporter renders every frame with Core Image into a CVPixelBuffer. It rendered straight into CGColorSpace.itur_709, which sounds right for HD video, and tagged the buffer as Rec.709 to match.

When I fixed it, I blamed BT.709's camera curve in general. Measuring it properly for this post showed something more specific: Apple's stack has two ideas of what the Rec.709 curve is.

• Core Graphics' itur_709 encodes with a plain 2.4 gamma, the display curve from BT.1886. Mid-gray 128 landed in the buffer as 135.
• Decoding a video tagged Rec.709, AVAssetImageGenerator uses a gamma of about 1.96, roughly the inverse of the camera curve. That 135 came back as 146.

Encode with 2.4, decode with 1.96, and every mid-tone comes back lighter. A pure 1.96 fits all three grays I tried: 64 came back as 84 and 192 as 201. Black and white don't move, since any gamma leaves 0 and 1 alone, so the picture looks washed out rather than broken.

The fix is to stay out of itur_709 entirely. Render in sRGB, exactly what the editor's preview shows, tag the buffer as sRGB, and let the encoder convert to the Rec.709 output the writer is set up for:

private let colorSpace = CGColorSpace(name: CGColorSpace.sRGB) ?? CGColorSpaceCreateDeviceRGB()

// For each frame:
CVBufferSetAttachment(buffer, kCVImageBufferCGColorSpaceKey, colorSpace, .shouldPropagate)
CVBufferSetAttachment(buffer, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, .shouldPropagate)
CVBufferSetAttachment(buffer, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_sRGB, .shouldPropagate)
context.render(image, to: buffer, bounds: CGRect(origin: .zero, size: outputSize), colorSpace: colorSpace)

The encoder now does the sRGB to Rec.709 conversion itself, with the same curve the decoder uses. Mid-gray goes in as 128 and comes back as 127. Exports match the preview within 1 to 2 levels, and a second test renders a frame the way the preview does, exports the same project, and compares the two.

With the color offset gone, the quality numbers made sense too: 43 to 50 dB PSNR through zoom moves, even at the smallest export preset. The bitrate had never been the problem.

A short checklist

• Set AVVideoColorPropertiesKey on every video writer. For SDR HD that's the Rec.709 primaries, transfer function and matrix.
• Tag the pixel buffers you append with the color space they're really in.
• Render in the space you preview in (sRGB on the Mac) and let the encoder convert to the output space.
• Keep a solid-color round trip in your tests, with pure colors and a few mid-grays. Matrix bugs leave grays alone and curve bugs leave black and white alone, so you need both.

Shotnix is a free, open-source screen recorder for Mac. The recorder fix is 02267bc and the exporter fix is 5b0c3dc.