iOS WebRTC 實(shí)現(xiàn)美顏濾鏡特效

最近需要實(shí)現(xiàn)美顏功能，調(diào)研了很多相關(guān)技術(shù)文章和開(kāi)源代碼。踩了很多坑，記錄實(shí)現(xiàn)步驟，希望對(duì)讀者有所幫助。
發(fā)現(xiàn)有2種實(shí)現(xiàn)方式，基于GPUImage
方案一：替換WebRTC的原生采集，使用GPUImageVideoCamera替換WebRTC中的視頻采集，得到經(jīng)過(guò)GPUImage添加美顏處理后的圖像，發(fā)送給WebRTC的OnFrame方法。(相對(duì)比較簡(jiǎn)單)
方案二：拿到WebRTC采集的原始視頻幀數(shù)據(jù)，然后傳給GPUImage庫(kù)進(jìn)行處理，最后把經(jīng)過(guò)處理的視頻幀傳回WebRTC。
本文章采用方案二
步驟為
取到采集數(shù)據(jù)i420 CVPixelBufferRef->紋理->GPUImage處理-> BGRA CVPixelBufferRef -> i420 CVPixelBufferRef(轉(zhuǎn)為webrtc支持格式)

取到采集數(shù)據(jù)i420 CVPixelBufferRef->紋理->GPUImage處理-> BGRA CVPixelBufferRef

_capturer = [[RTCCameraVideoCapturer alloc] initWithDelegate:_filter];

自定義類 遵守RTCVideoCapturerDelegate 重寫(xiě)didCaptureVideoFrame方法

- (void)capturer:(RTCVideoCapturer*)capturer
    didCaptureVideoFrame:(RTCVideoFrame*)frame {
//  操作C 需要手動(dòng)釋放  否則內(nèi)存暴漲
  CVPixelBufferRelease(_buffer)
//    拿到pixelBuffer
    ((RTCCVPixelBuffer*)frame.buffer).pixelBuffer
}


  //  采集拿到的數(shù)據(jù)進(jìn)行處理
- (CVPixelBufferRef)renderByGPUImage:(CVPixelBufferRef)pixelBuffer {
    CVPixelBufferRetain(pixelBuffer);
    __block CVPixelBufferRef output = nil;
    runSynchronouslyOnVideoProcessingQueue(^{
        [GPUImageContext useImageProcessingContext];
        //        1.取到采集數(shù)據(jù)i420 CVPixelBufferRef->紋理
        GLuint textureID = [self.pixelBufferHelper convertYUVPixelBufferToTexture:pixelBuffer];
        CGSize size = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
                                 CVPixelBufferGetHeight(pixelBuffer));
        //        2.GPUImage濾鏡處理
        [GPUImageContext setActiveShaderProgram:nil];
        GPUImageTextureInput *textureInput = [[GPUImageTextureInput alloc] initWithTexture:textureID size:size];
        // First pass: face smoothing filter
        GPUImageBilateralFilter *bilateralFilter = [[GPUImageBilateralFilter alloc] init];
        bilateralFilter.distanceNormalizationFactor = self->_distanceNormalizationFactor;
        [textureInput addTarget:bilateralFilter];
        GPUImageTextureOutput *textureOutput = [[GPUImageTextureOutput alloc] init];
        [bilateralFilter addTarget:textureOutput];
        [textureInput processTextureWithFrameTime:kCMTimeZero];
        //       3. 處理后的紋理轉(zhuǎn)pixelBuffer BGRA 
        output = [self.pixelBufferHelper convertTextureToPixelBuffer:textureOutput.texture
                                                         textureSize:size];
        [textureOutput doneWithTexture];
        glDeleteTextures(1, &textureID);
    });
    CVPixelBufferRelease(pixelBuffer);
    
    return output;
}

/// YUV 格式的 PixelBuffer 轉(zhuǎn)化為紋理
- (GLuint)convertYUVPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer {
    if (!pixelBuffer) {
        return 0;
    }
    
    CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
                                    CVPixelBufferGetHeight(pixelBuffer));

    [EAGLContext setCurrentContext:self.context];
    
    GLuint frameBuffer;
    GLuint textureID;
    
    // FBO
    glGenFramebuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    
    // texture
    glGenTextures(1, &textureID);
    glBindTexture(GL_TEXTURE_2D, textureID);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
    
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    
    
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureID, 0);
    
    glViewport(0, 0, textureSize.width, textureSize.height);
    
    // program
    glUseProgram(self.yuvConversionProgram);
    
    // texture
    CVOpenGLESTextureRef luminanceTextureRef = nil;
    CVOpenGLESTextureRef chrominanceTextureRef = nil;

    CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                                   self.textureCache,
                                                                   pixelBuffer,
                                                                   nil,
                                                                   GL_TEXTURE_2D,
                                                                   GL_LUMINANCE,
                                                                   textureSize.width,
                                                                   textureSize.height,
                                                                   GL_LUMINANCE,
                                                                   GL_UNSIGNED_BYTE,
                                                                   0,
                                                                   &luminanceTextureRef);
    if (status != kCVReturnSuccess) {
        NSLog(@"Can't create luminanceTexture");
    }
    
    status = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                          self.textureCache,
                                                          pixelBuffer,
                                                          nil,
                                                          GL_TEXTURE_2D,
                                                          GL_LUMINANCE_ALPHA,
                                                          textureSize.width / 2,
                                                          textureSize.height / 2,
                                                          GL_LUMINANCE_ALPHA,
                                                          GL_UNSIGNED_BYTE,
                                                          1,
                                                          &chrominanceTextureRef);
    
    if (status != kCVReturnSuccess) {
        NSLog(@"Can't create chrominanceTexture");
    }
    
    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(luminanceTextureRef));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "luminanceTexture"), 0);
    
    glActiveTexture(GL_TEXTURE1);
    glBindTexture(GL_TEXTURE_2D, CVOpenGLESTextureGetName(chrominanceTextureRef));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glUniform1i(glGetUniformLocation(self.yuvConversionProgram, "chrominanceTexture"), 1);
    
    GLfloat kXDXPreViewColorConversion601FullRange[] = {
        1.0,    1.0,    1.0,
        0.0,    -0.343, 1.765,
        1.4,    -0.711, 0.0,
    };
    
    GLuint yuvConversionMatrixUniform = glGetUniformLocation(self.yuvConversionProgram, "colorConversionMatrix");
    glUniformMatrix3fv(yuvConversionMatrixUniform, 1, GL_FALSE, kXDXPreViewColorConversion601FullRange);
    
    // VBO
    glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
    
    GLuint positionSlot = glGetAttribLocation(self.yuvConversionProgram, "position");
    glEnableVertexAttribArray(positionSlot);
    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
    
    GLuint textureSlot = glGetAttribLocation(self.yuvConversionProgram, "inputTextureCoordinate");
    glEnableVertexAttribArray(textureSlot);
    glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
    
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    
    glDeleteFramebuffers(1, &frameBuffer);
    
    glBindFramebuffer(GL_FRAMEBUFFER, 0);
    glBindBuffer(GL_ARRAY_BUFFER, 0);
    
    glFlush();
    
    self.luminanceTexture = luminanceTextureRef;
    self.chrominanceTexture = chrominanceTextureRef;
    
    CFRelease(luminanceTextureRef);
    CFRelease(chrominanceTextureRef);
    
    return textureID;
}


// 紋理轉(zhuǎn)化為CVPixelBufferRef 數(shù)據(jù)
- (CVPixelBufferRef)convertTextureToPixelBuffer:(GLuint)texture
                                    textureSize:(CGSize)textureSize {
    [EAGLContext setCurrentContext:self.context];
    
    CVPixelBufferRef pixelBuffer = [self createPixelBufferWithSize:textureSize];
    GLuint targetTextureID = [self convertRGBPixelBufferToTexture:pixelBuffer];
    
    GLuint frameBuffer;
    
    // FBO
    glGenFramebuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    
    // texture
    glBindTexture(GL_TEXTURE_2D, targetTextureID);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, textureSize.width, textureSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
    
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, targetTextureID, 0);
    
    glViewport(0, 0, textureSize.width, textureSize.height);
    
    // program
    glUseProgram(self.normalProgram);
    
    // texture
    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, texture);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glUniform1i(glGetUniformLocation(self.normalProgram, "renderTexture"), 0);
    
    // VBO
    glBindBuffer(GL_ARRAY_BUFFER, self.VBO);
    
    GLuint positionSlot = glGetAttribLocation(self.normalProgram, "position");
    glEnableVertexAttribArray(positionSlot);
    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)0);
    
    GLuint textureSlot = glGetAttribLocation(self.normalProgram, "inputTextureCoordinate");
    glEnableVertexAttribArray(textureSlot);
    glVertexAttribPointer(textureSlot, 2, GL_FLOAT, GL_FALSE, 5 * sizeof(float), (void*)(3* sizeof(float)));
    
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    
    glDeleteFramebuffers(1, &frameBuffer);
    
    glBindFramebuffer(GL_FRAMEBUFFER, 0);
    glBindBuffer(GL_ARRAY_BUFFER, 0);
    
    glFlush();
    
    return pixelBuffer;
}

// RBG 格式的 PixelBuffer 轉(zhuǎn)化為紋理
- (GLuint)convertRGBPixelBufferToTexture:(CVPixelBufferRef)pixelBuffer {
    if (!pixelBuffer) {
        return 0;
    }
    
    CGSize textureSize = CGSizeMake(CVPixelBufferGetWidth(pixelBuffer),
                                    CVPixelBufferGetHeight(pixelBuffer));
    CVOpenGLESTextureRef texture = nil;
    
    CVReturn status = CVOpenGLESTextureCacheCreateTextureFromImage(nil,
                                                                   self.textureCache,
                                                                   pixelBuffer,
                                                                   nil,
                                                                   GL_TEXTURE_2D,
                                                                   GL_RGBA,
                                                                   textureSize.width,
                                                                   textureSize.height,
                                                                   GL_BGRA,
                                                                   GL_UNSIGNED_BYTE,
                                                                   0,
                                                                   &texture);
    
    if (status != kCVReturnSuccess) {
        NSLog(@"Can't create texture");
    }
    
    self.renderTexture = texture;
    CFRelease(texture);
    return CVOpenGLESTextureGetName(texture);
}

BGRA CVPixelBufferRef -> i420 CVPixelBufferRef

需要引入libyuv庫(kù)

int transfer_32bgra_to_I420_ScaleToSize(CVPixelBufferRef source_pixelBuffer,CGSize targetSize,CVPixelBufferRef dst_pixelBuffer) {
    
    CVPixelBufferLockBaseAddress(source_pixelBuffer, 0);
    CVPixelBufferLockBaseAddress(dst_pixelBuffer, 0);
    
    //source-size
    size_t width = CVPixelBufferGetWidth(source_pixelBuffer);//圖像寬度（像素）
    size_t height = CVPixelBufferGetHeight(source_pixelBuffer);//圖像高度（像素）
    uint8_t *rgbaBuffer = (uint8_t *)CVPixelBufferGetBaseAddress(source_pixelBuffer);
    
    int yuvBufSize = width * height * 3 / 2;
    
    uint8_t* yuvBuf= (uint8_t*)malloc(yuvBufSize);
    
    
    //source-stride
    int Dst_Stride_Y = width;
    const int32 uv_stride = (width+1) / 2;
    
    //source-length
    const int y_length = width * height;
    int uv_length = uv_stride * ((height+1) / 2);
    
    //source-data
    unsigned char *Y_data_Dst = yuvBuf;
    unsigned char *U_data_Dst = yuvBuf + y_length;
    unsigned char *V_data_Dst = U_data_Dst + uv_length;
    
    //BGRAToI420, 內(nèi)存順序是BGRA,所以用方法得反過(guò)來(lái)ARGB
    ARGBToI420(rgbaBuffer,
                       width * 4,
                       Y_data_Dst, Dst_Stride_Y,
                       U_data_Dst, uv_stride,
                       V_data_Dst, uv_stride,
                       width, height);

    //scale-size
    int scale_yuvBufSize = targetSize.width * targetSize.height * 3 / 2;
    
    uint8_t* scale_yuvBuf= (uint8_t*)malloc(scale_yuvBufSize);
    
    //scale-stride
    int scale_Dst_Stride_Y = targetSize.width;
    const int32 scale_uv_stride = (targetSize.width+1) / 2;
    
    //scale-length
    const int scale_y_length = targetSize.width * targetSize.height;
    int scale_uv_length = scale_uv_stride * ((targetSize.height+1) / 2);
    
    //scale-data
    unsigned char *scale_Y_data_Dst = scale_yuvBuf;
    unsigned char *scale_U_data_Dst = scale_yuvBuf + scale_y_length;
    unsigned char *scale_V_data_Dst = scale_U_data_Dst + scale_uv_length;
    
    I420Scale(Y_data_Dst, Dst_Stride_Y,
                      U_data_Dst, uv_stride,
                      V_data_Dst, uv_stride,
                      width, height,
                      scale_Y_data_Dst, scale_Dst_Stride_Y,
                      scale_U_data_Dst, scale_uv_stride,
                      scale_V_data_Dst, scale_uv_stride,
                      targetSize.width, targetSize.height,
                      kFilterNone);
    
    //final-data
    uint8_t *final_y_buffer = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(dst_pixelBuffer, 0);
    uint8_t *final_uv_buffer = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(dst_pixelBuffer, 1);
    
    I420ToNV12(scale_Y_data_Dst, scale_Dst_Stride_Y,
                       scale_U_data_Dst, scale_uv_stride,
                       scale_V_data_Dst, scale_uv_stride,
                       final_y_buffer, scale_Dst_Stride_Y,
                       final_uv_buffer, scale_uv_stride*2,  //因?yàn)閡的寬度 = y * 0.5,v的寬度 = y * 0.5
                       targetSize.width, targetSize.height);
    
    CVPixelBufferUnlockBaseAddress(source_pixelBuffer, 0);
    CVPixelBufferUnlockBaseAddress(dst_pixelBuffer, 0);
    
    free(yuvBuf);
    free(scale_yuvBuf);
    
    return yuvBufSize;
}

didCaptureVideoFrame輸出處理后的數(shù)據(jù)

    transfer_32bgra_to_I420_ScaleToSize(_buffer, size, _i420Buffer);

    RTCCVPixelBuffer *rtcPixelBuffer =
    [[RTCCVPixelBuffer alloc] initWithPixelBuffer:_i420Buffer];
    RTCVideoFrame *filteredFrame =
    [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer
                                 rotation:frame.rotation
                              timeStampNs:frame.timeStampNs];
    [_output capturer:capturer didCaptureVideoFrame:filteredFrame];

總結(jié)

這樣就完成了為WebRTC的視頻添加美顏 ，此方案也適用于添加其他濾鏡。
參考鏈接
IOS技術(shù)分享| 在iOS WebRTC 中添加美顏濾鏡
iOS WebRTC 雜談之 視頻采集添加美顏特效
 WebRTC IOS視頻硬編碼流程及其中傳遞的CVPixelBufferRef

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書(shū)系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容