How to Support for both local and remote video in PiP mode?

[appsmall]

We need to implement Picture-in-Picture (PiP) support for displaying both local and remote videos simultaneously.

[hiroshihorie]

I think you will need to compose 2 streams of CMSampleBuffers into a single view.

[appsmall]

Could you please provide an example?

[hiroshihorie]

Hi @appsmall , sorry but I'm occupied at the moment. I hope to make this simple for developers.

I need to check:

• Does Apple's PIP spec support multiple AVSampleBufferDisplayLayer ? If yes, this will make it easier since we just need to layout the views to your preference, then push the CMSampleBuffers from the VideoRenderer protocol.
• If not, we need to process and combine the buffers to create a single source of CMSampleBuffers to render to a single AVSampleBufferDisplayLayer.

[ladvoc]

Hi @appsmall, you might want to explore the new init(activeVideoCallSourceView:contentViewController:) API for creating your content source. This update lets you supply a UIView for the PiP window instead of being limited to a single AVSampleBufferDisplayLayer, so you can lay out both side by side. Apple’s documentation includes a guide on how to use this API.

[bennnjamin]

@ladvoc @appsmall Have either of you had success using init(activeVideoCallSourceView:contentViewController:)? The linked guide is very brief and does not outline any specific requirements about the UIView other than it must contain the content of the video call. With Apple's historical limitations on background camera access, I suspect there are some undocumented requirements about the UIView because I have been unable to find any working example out there.

If anyone has more insight into this newer PiP API please share here