Background blur video processor #682
Conversation
|
it seems like you haven't added any nanpa changeset files to this PR. if this pull request includes changes to code, make sure to add a changeset, by writing a file to refer to the manpage for more information. |
|
|
|
Looks like the |
There was a problem hiding this comment.
This is just the basic gist, without mipmaps - tried them but looks like the generation itself is slower than simple CI downscale...
| // MARK: Parameters | ||
|
|
||
| private let downscaleFactor: CGFloat = 2 // Downscale before blurring, upscale before blending | ||
| private let blurRadius: Float = 3 // Keep the kernel size small O(n^2) |
There was a problem hiding this comment.
I made a counter-intuitive assumption to keep it constant (no need to expose it) and drive the blur amount by downscale itself (the bigger, the more downscaling) as in getDownscaleTransform.
Quadratic blur * quadratically fewer pixels = constant, controllable load
There was a problem hiding this comment.
I like the idea, just want to point out that in my testing on JS overly downscaling the background led to noticeably worse blurring, it didn't have the same visual "quality" anymore. Maybe CI is smarter about it though and this is not an issue here?
|
To the testers: look for |
| } | ||
|
|
||
| public extension CIImage { | ||
| func croppedAndScaled(to rect: CGRect, highQuality: Bool = true) -> CIImage { |
There was a problem hiding this comment.
Unfortunately, we need to recreate the crop information as CVPixelBuffer is still the source one...
|
|
||
| private let segmentationRequest = { | ||
| let segmentationRequest = VNGeneratePersonSegmentationRequest() | ||
| segmentationRequest.qualityLevel = .balanced |
There was a problem hiding this comment.
maybe we'd want to expose the segmentation quality level as a user setting?
There was a problem hiding this comment.
.accurate won't work in most scenarios (including macOS), but I'm happy to do a highQuality: Bool between .fast and .balanced
There was a problem hiding this comment.
sounds good to me! yeah, accurate doesn't sound like it should be used for video in the first place
| blurFilter.radius = blurRadius | ||
|
|
||
| guard let blurredImage = blurFilter.outputImage else { return frame } | ||
| let upscaledBlurredImage = blurredImage.transformed(by: downscaleTransform.inverted(), highQualityDownsample: false) |
There was a problem hiding this comment.
do we even need to manually upscale it? would have hoped the texture simply get's mapped onto the blendFilter in the correct dimension
|
@lukasIO looks like it balances itself nicely (the cost of scaling vs sigma vs total): This is iPhone 14, Release: Now we're totally safe even >30 FPS. |
| // Skip segmentation every N frames for slower devices | ||
| private var frameCount = 0 | ||
| #if os(macOS) | ||
| private let segmentationFrameInterval = 1 |
There was a problem hiding this comment.
This could be calculated using ProcessInfo.processInfo.isLowPowerModeEnabled
|
CI errors are because of old cocoapods thing, will resolve that after finalizing comments 👍 |
Overview
This is a revisited and slightly optimized version of Vision background blur.
While the basics are fairly easy to implement, the devil lies in the details:
CIMetal-backed stuff really helps hereExample PR to test: livekit-examples/swift-example#72
Rendering
CoreImage 🟢
CIContext+ the choice of paramsMetal (added for comparison) 🟠
8will drop frames, especially on iOS)For now, my estimate is that the proper shader will be a significant investment, without any guarantee it's really optimal.