AVVideoQualityKey: why 1.0 broke our recordings and 0.8 made them 8× smaller

Shotnix records the screen with ScreenCaptureKit and writes the video with AVAssetWriter. In May, one line went into the recorder's compression settings: AVVideoQualityKey: 1.0. Here's what they looked like, shortened:

var compression: [String: Any] = [
    AVVideoAverageBitRateKey: quality.bitrate(width: format.width, height: format.height, fps: fps, codec: format.codec),
    AVVideoExpectedSourceFrameRateKey: fps,
    AVVideoMaxKeyFrameIntervalKey: fps,
    AVVideoQualityKey: 1.0,
    AVVideoAllowFrameReorderingKey: false,
]
// For H.264:
compression[AVVideoH264EntropyModeKey] = AVVideoH264EntropyModeCABAC
compression[AVVideoProfileLevelKey] = AVVideoProfileLevelH264HighAutoLevel

Top quality sounds harmless. It wasn't, and it stayed in until September.

What 1.0 does

With AVVideoQualityKey at 1.0, Apple's H.264 encoder goes near-lossless. It writes several times the bitrate you set with AVVideoAverageBitRateKey, and it switches the profile to High 4:4:4 Predictive. Some players, browsers and websites can't open that profile at all.

Note the last line of the settings: the profile was set to AVVideoProfileLevelH264HighAutoLevel the whole time. The encoder wrote 4:4:4 anyway.

You can check the profile without any tools. It's the second byte of the avcC box in the track's format description: 100 is High, 244 is High 4:4:4 Predictive. The test reads it directly:

let formats = try await track.load(.formatDescriptions)
let description = try XCTUnwrap(formats.first)
let atoms = CMFormatDescriptionGetExtension(
    description,
    extensionKey: kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms
) as? [String: Any]
let avcC = try XCTUnwrap(atoms?["avcC"] as? Data)
XCTAssertEqual(avcC[avcC.startIndex + 1], 100, "profile_idc 100 is High; 244 is High 4:4:4 Predictive")

The first fix was to delete the line. Recordings went back to the standard High profile at the planned bitrate (ce31aeb).

A fixed bitrate has its own problem

Without a quality setting, the encoder spends roughly the bitrate you ask for, on every frame. That's the wrong budget for a screen. A screen recording is often a still screen, then some scrolling text, then a video playing. A fixed bitrate pays the same for all three.

Below 1.0, AVVideoQualityKey does something more useful: it lets the encoder decide how many bits each frame needs. A still screen costs almost nothing, text stays sharp, and a playing video gets what it needs. With the quality set, the average bitrate stopped acting as a cap in my tests.

var compression: [String: Any] = [
    AVVideoAverageBitRateKey: quality.bitrate(width: format.width, height: format.height, fps: fps, codec: format.codec),
    AVVideoQualityKey: quality.encoderQuality,
    AVVideoExpectedSourceFrameRateKey: fps,
    AVVideoMaxKeyFrameIntervalKey: fps,
    AVVideoAllowFrameReorderingKey: false,
]

var encoderQuality: Double {
    switch self {
    case .balanced: 0.70
    case .high: 0.80
    case .max: 0.88
    }
}

Picking the numbers

I measured five kinds of content: text, a playing video, gradients, a dragged window and typing. Each setting was compared with a reference recording frame by frame using ffmpeg's SSIM, looking at the worst frame as well as the average.

0.8 looks the same as the old fixed bitrate, even zoomed in. 0.7 starts to soften a playing video. 0.88 is as close to the screen as the High profile gets.

So where does 4:4:4 start? I encoded the same clip at 0.88, 0.9, 0.95 and 0.99, and every one of them stayed in High. Only 1.0 itself switches profiles. The presets stay below 0.9 anyway, and the test checks the profile at each one.

Results

• A minute of scrolling text on a Retina screen: 392 MB before, 48 MB after.
• A dragged window: about 40% smaller.
• A video playing on screen: about the same size, same look.
• Gradients and typing: unchanged.

The default preset is High, 0.8. Depending on what's on screen, that's anywhere from about 2 to 10 times smaller, with the same picture.

What didn't work

With a quality set, nothing caps the bitrate. On a clip of moving noise, the worst case for any encoder, 0.7 wrote about six times as much as the same settings without a quality.

So I tried to cap it with VideoToolbox's DataRateLimits property in the compression settings. AVAssetWriter accepts it, but it starved exactly the frames that needed bits most: the worst frames of a playing video lost detail. A screen is rarely anything like noise, so I left the encoder uncapped. The bitrate plan only feeds the "up to N MB/min" estimate the app shows, which stays an honest ceiling.

Shotnix is a free, open-source screen recorder for Mac. Both changes are a few lines each: ce31aeb and 4679db0.