platform: RK3399
OS: Android 7.1
kernel：4.4
參考：
1. KrisFei https://blog.csdn.net/kris_fei/article/details/52451409
2. Kiazhu https://blog.csdn.net/kiazhu/article/details/84652749
3. https://source.android.google.cn/devices/camera?hl=en
4.https://www.cnblogs.com/blogs-of-lxl/p/10981303.html
概述
? Android Camera Hal 是android framework和kernel聯(lián)系的重要通道。該部分一般由廠商實現(xiàn)并封裝自己的算法或者業(yè)務邏輯，進一步抽象出內核設備驅動的各項功能，并通過google實現(xiàn)的統(tǒng)一接口供android framework進行調用。
? 下面是舊版的Camera框架圖(適用于android 8.0以下,android 8.0以上請使用新的camera hal組件及框架)
在這里插入圖片描述
應用框架
應用代碼位于應用框架級別，它利用 android.hardware.Camera API 與相機硬件進行交互。在內部，此代碼會調用相應的 JNI 粘合類，以訪問與相機互動的原生代碼。
JNI
與 android.hardware.Camera 關聯(lián)的 JNI 代碼位于 frameworks/base/core/jni/android_hardware_Camera.cpp 中。此代碼會調用較低級別的原生代碼以獲取對實體相機的訪問權限，并返回用于在框架級別創(chuàng)建 android.hardware.Camera 對象的數(shù)據。
原生框架
在 frameworks/av/camera/Camera.cpp 中定義的原生框架可提供相當于 android.hardware.Camera 類的原生類。此類會調用 IPC binder 代理，以獲取對相機服務的訪問權限。
Binder IPC 代理
IPC binder 代理用于促進跨越進程邊界的通信。調用相機服務的 3 個相機 binder 類位于 frameworks/av/camera 目錄中。 ICameraService 是相機服務的接口；ICamera 是已打開的特定相機設備的接口；ICameraClient 是返回到應用框架的設備接口。
相機服務
位于 frameworks/av/services/camera/libcameraservice/CameraService.cpp 下的相機服務是與 HAL 進行互動的實際代碼。
HAL
硬件抽象層定義了由相機服務調用、且您必須實現(xiàn)以確保相機硬件正常運行的標準接口。
內核驅動程序
相機的驅動程序可與實際相機硬件以及您的 HAL 實現(xiàn)進行互動。相機和驅動程序必須支持 YV12 和 NV21 圖片格式，以便在顯示和視頻錄制時支持預覽相機圖片。
Android Camera HAL簡介
HAL1簡介
功能調用簡圖
在這里插入圖片描述
Camera hal1中的camera_module_t 接口
camera_module_t HAL_MODULE_INFO_SYM = {
common: {
tag: HARDWARE_MODULE_TAG,
version_major: ((CONFIG_CAMERAHAL_VERSION&0xffff00)>>16),
version_minor: CONFIG_CAMERAHAL_VERSION&0xff,
id: CAMERA_HARDWARE_MODULE_ID,
name: CAMERA_MODULE_NAME,
author: "RockChip",
methods: &camera_module_methods,
dso: NULL, /* remove compilation warnings */
reserved: {0}, /* remove compilation warnings */
},
get_number_of_cameras: camera_get_number_of_cameras,
get_camera_info: camera_get_camera_info,
set_callbacks:NULL,
get_vendor_tag_ops:NULL,
}
HAL3簡介
? Camera API2/HAL3架構下使用了全新的CameraMetadata結構取代了之前的SetParameter/Paramters等操作，實現(xiàn)了Java到native到HAL3的參數(shù)傳遞。引入了管道的概念將安卓設備和攝像頭之間聯(lián)系起來，系統(tǒng)向攝像頭發(fā)送 Capture 請求，而攝像頭會返回 CameraMetadata，這一切建立在一個叫作 CameraCaptureSession 的會話中。
功能調用簡圖
在這里插入圖片描述
以最常見的android.control Section為例,下圖描述了Camera Metadata對不同section以及相應section下不同tag的布局圖
在這里插入圖片描述
RK Camera HAL整體框架
在這里插入圖片描述
? canera server通過與CameraHal_Module的標準接口建立和Hal層的連接后，CmaeraHal會通過SensorListener返回消息給上層。如果上層要預覽或者拍照，要通過binder機制向Hal層發(fā)送命令，Hal層MessageQueue.cpp的消息隊列獲取到上層的命令后，會通過消息通知器通知CameraHal，CameraHal有一個CommandThread接收命令，收到命令后，最終會下達命令給對應的Adapter去執(zhí)行對于的動作。
? RK 的Camera Hal還實現(xiàn)了人臉檢測等功能以及相關的輔助函數(shù)，就沒有在上面的主體框架中畫出來。
RK Camera HAL 調用流程
開機啟動
CameraHal_Module.cpp --> camera_get_number_of_cameras
-->profiles = camera_board_profiles::getInstance(); // 解析/etc/cam_board.xml
-->camera_board_profiles::LoadSensor(profiles); //注冊攝像頭驅動
-->OpenAndRegistOneSensor(profiles->mDevieVector[profiles->mXmlDevInfo[i].index]); //最多支持雙攝
-->RegisterSensorDevice(pCamInfo); //注冊i2c設備
-->camsys_fd = open(pSensorInfo->mCamsysDevPath, O_RDWR); //打開dev/camsys_marvin1 設備節(jié)點(camsys_marvin1注冊見“RK平臺攝像頭驅動”文章)
-->ioctl(camsys_fd, CAMSYS_VERCHK, &(pCamInfo->mCamsysVersion)); //檢查驅動版本
--> ioctl(camsys_fd, CAMSYS_REGISTER_DEVIO, &extdev); //注冊攝像頭
-->rk_sensor_pwrseq(camsys_fd, pCamInfo, 1); // 啟動攝像頭
--> ioctl(camsys_fd, CAMSYS_I2CWR, &i2cinfo); //配置I2C信息
--> ioctl(camsys_fd, CAMSYS_QUREYIOMMU, &iommu_enabled); //查詢IOMMU使能
--> ioctl(camsys_fd, CAMSYS_I2CRD, &i2cinfo); //查詢設備ID
-->rk_sensor_pwrseq(camsys_fd, pCamInfo, 0); //power off
--> if(pSensorInfo->mFacing == RK_CAM_FACING_FRONT){ //配置攝像頭方向
camInfoTmp[cam_cnt&0x01].facing_info.facing = CAMERA_FACING_FRONT;
--> open(cam_path, O_RDONLY); //打開videoX
-->ioctl(fd, VIDIOC_QUERYCAP, &capability) //查詢capability
-->rk_cam_total_info* pNewCamInfo = new rk_cam_total_info(); //所有sensor信息都放在里面
-->ioctl(fd, VIDIOC_ENUM_FMT, &fmtdesc）
--> ioctl(fd, VIDIOC_ENUM_FRAMESIZES, &fsize)
-->rk_DV_info *pDVResolution = new rk_DV_info();
--> ioctl(fd, VIDIOC_ENUM_FRAMEINTERVALS, &fival)
--->camera_board_profiles::ProduceNewXml(profiles); //生成media_profiles.xml
App打開Camera
CameraHal_Module.cpp -->camera_device_open
//標準實現(xiàn)
-->camera_device = (rk_camera_device_t*)malloc(sizeof(*camera_device));
--> camera_ops = (camera_device_ops_t*)malloc(sizeof(*camera_ops));
.... //初始化camera_device camera_ops
//RK自己實現(xiàn)
//CameraHal類負責與cameraservice聯(lián)系，實現(xiàn)
//cameraservice要求實現(xiàn)的接口。此類只負責公共資源的申請，以及任務的分發(fā)。
-->android::CameraHal(cameraid)
-->commandThreadCommandQ("commandCmdQ") //創(chuàng)建commandCmdQ的message queue
-> new IonMemManager
mPreviewBuf = new PreviewBufferProvider(mCamMemManager); //previewBuffer
mVideoBuf = new BufferProvider(mCamMemManager); //video
mRawBuf = new BufferProvider(mCamMemManager); //raw
mJpegBuf = new BufferProvider(mCamMemManager); //jpeg
mUvcBuf = new BufferProvider(mCamMemManager); //uvc
//根據不同類型的camera new不同類型的adapter
--->mCameraAdapter new CameraIspAdapter(cameraId)
-->mDisplayAdapter = new DisplayAdapter(); //new Display adapter
-->displayThreadCommandQ //創(chuàng)建名為displayCmdQ的message queue.
new DisplayThread //處理顯示相關事物
-->mEventNotifier = new AppMsgNotifier(mCameraAdapter);
-->encProcessThreadCommandQ //創(chuàng)建名字為pictureEncThreadQ的message queue.
eventThreadCommandQ //創(chuàng)建名字為eventThreadQ的message queue.
create_vpu_memory_pool_allocator //創(chuàng)建vpu 內存池分配器
new CameraAppMsgThread
new EncProcessThread
new CameraAppFaceDetThread //人臉檢測線程
new CameraAppCallbackThread //回調線程
-->mCameraAdapter->initialize()
--->cameraCreate(mCamId) //創(chuàng)建一個camera
--->initDefaultParameters(mCamId); //初始化各個參數(shù)默認值
--->mCommandThread = new CommandThread(this); //command Thread loop
---->mSensorListener = new SensorListener(); //注冊Listerner
---->new SensorLooperThread
---->mLooper->pollOnce //循環(huán)查詢是否有event上報，有就會運行sensor_events_listener()獲取Orientation。
支持的ops
camera_ops->set_preview_window = camera_set_preview_window;
camera_ops->set_callbacks = camera_set_callbacks;
camera_ops->enable_msg_type = camera_enable_msg_type;
camera_ops->disable_msg_type = camera_disable_msg_type;
camera_ops->msg_type_enabled = camera_msg_type_enabled;
camera_ops->start_preview = camera_start_preview;
camera_ops->stop_preview = camera_stop_preview;
camera_ops->preview_enabled = camera_preview_enabled;
camera_ops->store_meta_data_in_buffers = camera_store_meta_data_in_buffers;
camera_ops->start_recording = camera_start_recording;
camera_ops->stop_recording = camera_stop_recording;
camera_ops->recording_enabled = camera_recording_enabled;
camera_ops->release_recording_frame = camera_release_recording_frame;
camera_ops->auto_focus = camera_auto_focus;
camera_ops->cancel_auto_focus = camera_cancel_auto_focus;
camera_ops->take_picture = camera_take_picture;
camera_ops->cancel_picture = camera_cancel_picture;
camera_ops->set_parameters = camera_set_parameters;
camera_ops->get_parameters = camera_get_parameters;
camera_ops->put_parameters = camera_put_parameters;
camera_ops->send_command = camera_send_command;
camera_ops->release = camera_release;
camera_ops->dump = camera_dump;
command Thread loop支持的cmd
CMD_PREVIEW_START
CMD_PREVIEW_STOP
CMD_SET_PREVIEW_WINDOW
CMD_SET_PARAMETERS
CMD_PREVIEW_CAPTURE_CANCEL
CMD_CONTINUOS_PICTURE
CMD_AF_START
CMD_AF_CANCEL
CMD_START_FACE_DETECTION
CMD_EXIT
設置參數(shù)
CameraHal::setParameters //參數(shù)是char*
setParameters -> //參數(shù)是CameraParameters
commandThreadCommandQ.put //cmd是 CMD_SET_PARAMETERS
commandThread ->
mCameraAdapter->setParameters
CameraSOCAdapter::setParameters -> //假設位soc camera
mRefEventNotifier->setPreviewDataCbRes
cameraConfig
ioctl(mCamFd, VIDIOC_S_CTRL, &control); //設置 白平衡
//設置zoom,color effect,scene,anti banding,white balance
//lock ,exposurelock,focus,flash mode,exposure
ioctl(mCamFd, VIDIOC_S_EXT_CTRLS, &extCtrInfos);
設置顯示窗口
CameraHal::setPreviewWindow ->
commandThreadCommandQ.put -> //cmd: CMD_SET_PREVIEW_WINDOW
commandThread ->
mDisplayAdapter->setPreviewWindow
// mDisplayAdapter->startDisplay(app_previw_w, app_preview_h); //啟動預覽
Start preview
CameraHal::startPreview
---> commandThreadCommandQ.put -> //cmd: CMD_PREVIEW_START
-->commandThread
mParameters.getPreviewSize //獲取預覽參數(shù)
mCameraAdapter->getCurPreviewState //獲取當前預覽狀態(tài)
//選擇picture size, 如果和preview 分辨率不一樣，那需要stop preview再start preview
selectPreferedDrvSize(&prefered_w,&prefered_h,false);
//如果在預覽中
mDisplayAdapter->pauseDisplay
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_PAUSE
displayThread
cameraDisplayBufferDestory();
mEventNotifier->stopReceiveFrame
flushPicture(); //EncProcessThread::CMD_ENCPROCESS_PAUSE
pausePreviewCBFrameProcess(); //CameraAppMsgThread::CMD_EVENT_PAUSE
stopFaceDection(); //CameraAppFaceDetThread::CMD_FACEDET_PAUSE
mCameraAdapter->stopPreview
cameraStream(false)
ioctl(mCamFd, cmd, &type)
mCameraPreviewThread->requestExitAndWait(); //quit preview thread
mCameraPreviewThread.clear();
cameraStop()
mPreviewBufferProvider->freeBuffer()
mCameraAdapter->startPreview
mPreviewBufProvider->createBuffer //PREVIEWBUFFER
mCamBuffer->createPreviewBuffer
createIonBuffer
cameraSetSize(w, h, mCamDriverPreviewFmt,is_capture)
ioctl(mCamFd, VIDIOC_S_FMT, &format)
cameraStart()
ioctl(mCamFd, VIDIOC_REQBUFS, &creqbuf)
ioctl(mCamFd, VIDIOC_QUERYBUF, &buffer)
ioctl(mCamFd, VIDIOC_QBUF, &buffer)
mmap
cameraStream(true)
mCameraPreviewThread = new CameraPreviewThread(this) //進入preview thread loop
mEventNotifier->startReceiveFrame
mEventNotifier->startFaceDection //如果配置了人臉檢測功能就開始人臉檢測
//否則同上，只是不用stopPreview
mDisplayAdapter->startDisplay
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_START
displayThread
cameraDisplayBufferDestory()
//準備display buffer放frame,然后thread進入休眠等待frame的到來
cameraDisplayBufferCreate()
//ioctl stream on后frame就會從kernel上來，接著調用previewthread
CameraAdapter::previewThread
getFrame
ioctl(mCamFd, VIDIOC_DQBUF, &cfilledbuffer1)
//判斷以什么形式顯示：display,video,picture,datacb
//判斷調用那個notify 這里是CMD_PREVIEWBUF_DISPING
mRefDisplayAdapter->notifyNewFrame
displayThreadCommandQ.put(&msg) //CMD_DISPLAY_FRAME
displayThread
mANativeWindow->dequeue_buffer
mANativeWindow->lock_buffer
mANativeWindow->enqueue_buffer
mFrameProvider->returnFrame
adapterReturnFrame //把使用好的buffer還給系統(tǒng)
ioctl(mCamFd, VIDIOC_QBUF, &vb)
Stop preview
camera_stop_preview
gCameraHals[rk_dev->cameraid]->stopPreview()
CameraHal::stopPreview
commandThreadCommandQ.put(&msg) //CMD_PREVIEW_STOP
commandThread:
mDisplayAdapter->pauseDisplay()
mEventNotifier->stopReceiveFrame()
mCameraAdapter->stopPreview()
cameraStream(false)
ioctl(mCamFd, cmd, &type)
mCameraPreviewThread->requestExitAndWait(); //quit preview thread
mCameraPreviewThread.clear();
cameraStop()
mPreviewBufferProvider->freeBuffer()
Take Picture
camera_take_picture
gCameraHals[rk_dev->cameraid]->takePicture()
CameraHal::commandThread //CMD_CONTINUOS_PICTURE
fillPicturInfo(picinfo)
mEventNotifier->takePicture(picinfo)
AppMsgNotifier::takePicture //設置running state為STA_RECEIVE_PIC_FRAME，等著previewThread去get frame
//previewThread接到frame之后
CameraAdapter::previewThread
mRefEventNotifier->isNeedSendToPicture
mRefEventNotifier->notifyNewPicFrame
AppMsgNotifier::notifyNewPicFrame
//cmd: EncProcessThread::CMD_ENCPROCESS_SNAPSHOT
encProcessThreadCommandQ.put()
encProcessThread
captureEncProcessPicture
copyAndSendCompressedImage
callback_compressed_image
CameraAppCallbackThread:: CMD_MSG_COMPRESSED_IMAGE
mDataCb
frame->release(frame)
mFrameProvider->returnFrame
Record Video
CameraAdapter::previewThread
getFrame
mRefEventNotifier->notifyNewVideoFrame
eventThreadCommandQ.put //cmdCameraAppMsgThread::CMD_EVENT_VIDEO_ENCING
AppMsgNotifier::eventThread
processVideoCb
AppMsgNotifier::processVideoCb
callback_video_frame
callbackThreadCommandQ.put //cmd:CameraAppCallbackThread::CMD_MSG_VIDEO_FRAME
AppMsgNotifier::callbackThread ->
mDataCbTimestamp
Preview datacallback
CameraAdapter::previewThread ->
mRefEventNotifier->notifyNewPreviewCbFrame -> //前提是設置了callback回調函數(shù)
AppMsgNotifier::notifyNewPreviewCbFrame ->
eventThreadCommandQ.put -> //cmd: CameraAppMsgThread::CMD_EVENT_PREVIEW_DATA_CB
AppMsgNotifier::eventThread ->
processPreviewDataCb ->
AppMsgNotifier::processPreviewDataCb -> //格式轉換在這里完成
cameraFormatConvert
callback_preview_frame ->
callbackThreadCommandQ.put -> //cmd: CameraAppCallbackThread::CMD_MSG_PREVIEW_FRAME
AppMsgNotifier::callbackThread ->
mDataCb //調用上層傳遞下來的callback.
frame->release //release buffer.
mFrameProvider->returnFrame
