Picture-in-picture is harder than it looks

Leave a video call and the video should follow you, floating over whatever you do next. On iOS that is Picture-in-Picture, the system gives it to you for free, and it took us longer than almost anything else in the calls stack. Here is why.

The problem is that our video and the system's window are two incompatible worlds.

WebRTC hands us decoded frames and we render them into a Metal-backed view. That is fast, it composites well, and it is what draws the call screen. The system's PiP window cannot use it. AVPictureInPictureController drives exactly one thing, an AVSampleBufferDisplayLayer, and it will not float an arbitrary view no matter how nicely you ask.

So the feature is a bridge. A lightweight renderer attaches to the remote video track, receives each frame, converts it into a CMSampleBuffer and enqueues it into a display layer that the PiP controller owns. Two renderers of the same track: one for the in-app screen, one feeding a layer the system is allowed to float. When you swipe away mid-call, the system takes over that layer and the conversation continues in a bubble.

Two details are easy to get wrong and expensive to discover late.

The first is that PiP is not really a video feature, it is a background-execution feature. The window keeps updating only because the app is allowed to keep running, which requires the audio background mode and an audio session that stays active in the background. Both are already true during a call, which is what makes this practical for calls and would make it a much larger undertaking for anything else.

The second is that there is more than one thing worth floating, and they cannot share.

A one-to-one call and a live broadcast both want PiP. The machinery is identical, a WebRTC track feeding a sample-buffer layer. But each needs its own layer and its own controller, because a single shared instance means starting one tears down the other's window. So there are two independent bridges, one for calls and one for watching a live.

They differ in a way that has nothing to do with plumbing: the close button on the floating window has to mean the right thing. On a call it hangs up. On a live it stops watching. Same pixel, same system control, two very different consequences, and getting that backwards would be the worst kind of bug, the sort where the interface behaves exactly as designed and destroys something anyway.

There is one more thing the shared bridge buys us. Our call screen can collapse to a small floating tile inside the app, and both the full screen and the tile host the same layer. Because they share one bridge, leaving the app while the call is minimised still floats the video, rather than the PiP window quietly having been attached to a screen that no longer exists.

Free from the system, then, in the sense that a piano is free once you have carried it up the stairs.