Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Split videoHdr and photoHdr into two settings #2161

Merged
merged 8 commits into from
Nov 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/docs/guides/FORMATS.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ To get all available formats, simply use the `CameraDevice`'s [`formats` propert
- [`minFps`](/docs/api/interfaces/CameraDeviceFormat#minfps)/[`maxFps`](/docs/api/interfaces/CameraDeviceFormat#maxfps): A range of possible values for the `fps` property. For example, if your format has `minFps: 1` and `maxFps: 60`, you can either use `fps={30}`, `fps={60}` or any other value in between for recording videos.
- [`videoStabilizationModes`](/docs/api/interfaces/CameraDeviceFormat#videostabilizationmodes): All supported Video Stabilization Modes, digital and optical. If this specific format contains your desired [`VideoStabilizationMode`](/docs/api/#videostabilizationmode), you can pass it to your `<Camera>` via the [`videoStabilizationMode` property](/docs/api/interfaces/CameraProps#videoStabilizationMode).
- [`pixelFormats`](/docs/api/interfaces/CameraDeviceFormat#pixelformats): All supported Pixel Formats. If this specific format contains your desired [`PixelFormat`](/docs/api/#PixelFormat), you can pass it to your `<Camera>` via the [`pixelFormat` property](/docs/api/interfaces/CameraProps#pixelFormat).
- [`supportsVideoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsvideohdr): Whether this specific format supports true 10-bit HDR for video capture. If this is `true`, you can enable `hdr` on your `<Camera>`.
- [`supportsPhotoHDR`](/docs/api/interfaces/CameraDeviceFormat#supportsphotohdr): Whether this specific format supports HDR for photo capture. It will use multiple captures to fuse over-exposed and under-exposed Images together to form one HDR photo. If this is `true`, you can enable `hdr` on your `<Camera>`.
- [`supportsVideoHdr`](/docs/api/interfaces/CameraDeviceFormat#supportsvideohdr): Whether this specific format supports true 10-bit HDR for video capture. If this is `true`, you can enable `videoHdr` on your `<Camera>`.
- [`supportsPhotoHdr`](/docs/api/interfaces/CameraDeviceFormat#supportsphotohdr): Whether this specific format supports HDR for photo capture. It will use multiple captures to fuse over-exposed and under-exposed Images together to form one HDR photo. If this is `true`, you can enable `photoHdr` on your `<Camera>`.
- [`supportsDepthCapture`](/docs/api/interfaces/CameraDeviceFormat#supportsdepthcapture): Whether this specific format supports depth data capture. For devices like the TrueDepth/LiDAR cameras, this will always be true.
- ...and more. See the [`CameraDeviceFormat` type](/docs/api/interfaces/CameraDeviceFormat) for all supported properties.

Expand Down Expand Up @@ -201,7 +201,8 @@ function App() {
Other props that depend on the `format`:

* `fps`: Specifies the frame rate to use
* `hdr`: Enables HDR photo or video capture and preview
* `videoHdr`: Enables HDR video capture and preview
* `photoHdr`: Enables HDR photo capture
* `lowLightBoost`: Enables a night-mode/low-light-boost for photo or video capture and preview
* `videoStabilizationMode`: Specifies the video stabilization mode to use for the video pipeline
* `pixelFormat`: Specifies the pixel format to use for the video pipeline
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/FRAME_PROCESSORS.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ When running frame processors, it is often important to choose an appropriate [f

* If you are running heavy AI/ML calculations in your frame processor, make sure to [select a format](/docs/guides/formats) that has a lower resolution to optimize it's performance. You can also resize the Frame on-demand.
* Sometimes a frame processor plugin only works with specific [pixel formats](/docs/api/interfaces/CameraDeviceFormat#pixelformats). Some plugins (like Tensorflow Lite Models) don't work with `yuv`, so use a [`pixelFormat`](/docs/api/interfaces/CameraProps#pixelformat) of `rgb` instead.
* Some Frame Processor plugins don't work with HDR formats. In this case you need to disable [`hdr`](/docs/api/interfaces/CameraProps#hdr).
* Some Frame Processor plugins don't work with HDR formats. In this case you need to disable [`videoHdr`](/docs/api/interfaces/CameraProps#videohdr).

## Benchmarks

Expand Down
20 changes: 13 additions & 7 deletions docs/docs/guides/HDR.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ To enable HDR capture, you need to select a format (see ["Camera Formats"](forma

```ts
const format = useCameraFormat(device, [
{ photoHDR: true },
{ videoHDR: true },
{ photoHdr: true },
{ videoHdr: true },
])
```

Expand All @@ -59,21 +59,27 @@ const format = useCameraFormat(device, [

```ts
const format = getCameraFormat(device, [
{ photoHDR: true },
{ videoHDR: true },
{ photoHdr: true },
{ videoHdr: true },
])
```

</TabItem>
</Tabs>

Then, pass the `format` to the Camera and enable the `hdr` prop if it is supported:
Then, pass the `format` to the Camera and enable the `videoHdr`/`photoHdr` props if it is supported:

```tsx
const format = ...
const supportsHdr = format.supportsPhotoHDR && format.supportsVideoHDR

return <Camera {...props} format={format} hdr={supportsHdr} />
return (
<Camera
{...props}
format={format}
videoHdr={format.supportsVideoHdr}
photoHdr={format.supportsPhotoHdr}
/>
)
```

Now, all captures (`takePhoto(..)` and `startRecording(..)`) will be configured to use HDR.
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/guides/PERFORMANCE.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ See ["Camera Devices"](devices) for more information.

Note: By default (when not passing the options object), a simpler device is already chosen.

### No HDR
### No Video HDR

HDR uses 10-bit formats and/or additional processing steps that come with additional computation overhead. Disable HDR (don't pass `hdr` to the `<Camera>`) for higher efficiency.
Video HDR uses 10-bit formats and/or additional processing steps that come with additional computation overhead. Disable Video HDR (don't pass `videoHdr` to the `<Camera>`) for higher efficiency.

### Buffer Compression

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/RECORDING_VIDEOS.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ To calculate your target bit-rate, you can use this formula:
```ts
let bitRate = baseBitRate
bitRate = bitRate / 30 * fps // FPS
if (hdr === true) bitRate *= 1.2 // HDR
if (videoHdr === true) bitRate *= 1.2 // 10-Bit Video HDR
if (codec === 'h265') bitRate *= 0.8 // H.265
bitRate *= yourCustomFactor // e.g. 0.5x for half the bit-rate
```
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/guides/TROUBLESHOOTING.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ If you're experiencing build issues or runtime issues in VisionCamera, make sure
* For errors without messages, there's often an error code attached. Look up the error code on [osstatus.com](https://www.osstatus.com) to get more information about a specific error.
2. If your Frame Processor is not running, make sure you check the native Xcode logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong.
3. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI.
4. If you are experiencing black-screens, try removing all properties such as `fps`, `hdr` or `format` on the `<Camera>` component except for the required ones:
4. If you are experiencing black-screens, try removing all properties such as `fps`, `videoHdr` or `format` on the `<Camera>` component except for the required ones:
```tsx
<Camera device={device} isActive={true} style={{ width: 500, height: 500 }} />
```
Expand Down Expand Up @@ -112,7 +112,7 @@ If you're experiencing build issues or runtime issues in VisionCamera, make sure
2. If a camera device is not being returned by [`Camera.getAvailableCameraDevices()`](/docs/api/classes/Camera#getavailablecameradevices), make sure it is a Camera2 compatible device. See [this section in the Android docs](https://developer.android.com/reference/android/hardware/camera2/CameraDevice#reprocessing) for more information.
3. If your Frame Processor is not running, make sure you check the native Android Studio/Logcat logs. There is useful information about the Frame Processor Runtime that will tell you if something goes wrong.
4. If your Frame Processor is not running, make sure you are not using a remote JS debugger such as Google Chrome, since those don't work with JSI.
5. If you are experiencing black-screens, try removing all properties such as `fps`, `hdr` or `format` on the `<Camera>` component except for the required ones:
5. If you are experiencing black-screens, try removing all properties such as `fps`, `videoHdr` or `format` on the `<Camera>` component except for the required ones:
```tsx
<Camera device={device} isActive={true} style={{ width: 500, height: 500 }} />
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ class CameraView(context: Context) :
var format: ReadableMap? = null
var fps: Int? = null
var videoStabilizationMode: VideoStabilizationMode? = null
var hdr: Boolean? = null // nullable bool
var videoHdr = false
var photoHdr = false
var lowLightBoost: Boolean? = null // nullable bool

// other props
Expand Down Expand Up @@ -177,6 +178,10 @@ class CameraView(context: Context) :
// Orientation
config.orientation = orientation

// HDR
config.videoHdr = videoHdr
config.photoHdr = photoHdr

// Format
val format = format
if (format != null) {
Expand All @@ -188,7 +193,6 @@ class CameraView(context: Context) :
// Side-Props
config.fps = fps
config.enableLowLightBoost = lowLightBoost ?: false
config.enableHdr = hdr ?: false
config.torch = torch

// Zoom
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,17 +107,22 @@ class CameraViewManager : ViewGroupManager<CameraView>() {
view.fps = if (fps > 0) fps else null
}

@ReactProp(name = "hdr")
fun setHdr(view: CameraView, hdr: Boolean?) {
view.hdr = hdr
@ReactProp(name = "photoHdr", defaultBoolean = false)
fun setPhotoHdr(view: CameraView, photoHdr: Boolean) {
view.photoHdr = photoHdr
}

@ReactProp(name = "videoHdr", defaultBoolean = false)
fun setVideoHdr(view: CameraView, videoHdr: Boolean) {
view.videoHdr = videoHdr
}

@ReactProp(name = "lowLightBoost")
fun setLowLightBoost(view: CameraView, lowLightBoost: Boolean?) {
view.lowLightBoost = lowLightBoost
}

@ReactProp(name = "isActive")
@ReactProp(name = "isActive", defaultBoolean = false)
fun setIsActive(view: CameraView, isActive: Boolean) {
view.isActive = isActive
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ data class CameraConfiguration(
var photo: Output<Photo> = Output.Disabled.create(),
var video: Output<Video> = Output.Disabled.create(),
var codeScanner: Output<CodeScanner> = Output.Disabled.create(),
var enableHdr: Boolean = false,

// HDR
var videoHdr: Boolean = false,
var photoHdr: Boolean = false,

// Orientation
var orientation: Orientation = Orientation.PORTRAIT,
Expand Down Expand Up @@ -87,7 +90,7 @@ data class CameraConfiguration(
val outputsChanged = deviceChanged || // input device
left?.photo != right.photo || left.video != right.video || left.codeScanner != right.codeScanner ||
left.preview != right.preview || // outputs
left.enableHdr != right.enableHdr || left.format != right.format // props that affect the outputs (hdr, format, ..)
left.videoHdr != right.videoHdr || left.photoHdr != right.photoHdr || left.format != right.format // props that affect the outputs

val sidePropsChanged = outputsChanged || // depend on outputs
left?.torch != right.torch || left.enableLowLightBoost != right.enableLowLightBoost || left.fps != right.fps ||
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import android.graphics.ImageFormat
import android.hardware.camera2.CameraCharacteristics
import android.hardware.camera2.CameraManager
import android.hardware.camera2.CameraMetadata
import android.hardware.camera2.params.DynamicRangeProfiles
import android.os.Build
import android.util.Range
import android.util.Size
Expand Down Expand Up @@ -90,10 +89,8 @@ class CameraDeviceDetails(private val cameraManager: CameraManager, private val
private fun getHasVideoHdr(): Boolean {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
if (capabilities.contains(CameraMetadata.REQUEST_AVAILABLE_CAPABILITIES_DYNAMIC_RANGE_TEN_BIT)) {
val availableProfiles = characteristics.get(CameraCharacteristics.REQUEST_AVAILABLE_DYNAMIC_RANGE_PROFILES)
?: DynamicRangeProfiles(LongArray(0))
return availableProfiles.supportedProfiles.contains(DynamicRangeProfiles.HLG10) ||
availableProfiles.supportedProfiles.contains(DynamicRangeProfiles.HDR10)
val recommendedHdrProfile = characteristics.get(CameraCharacteristics.REQUEST_RECOMMENDED_TEN_BIT_DYNAMIC_RANGE_PROFILE)
return recommendedHdrProfile != null
}
}
return false
Expand Down Expand Up @@ -181,8 +178,8 @@ class CameraDeviceDetails(private val cameraManager: CameraManager, private val
map.putInt("maxFps", fpsRange.upper)
map.putDouble("maxZoom", maxZoom)
map.putDouble("fieldOfView", getMaxFieldOfView())
map.putBoolean("supportsVideoHDR", supportsVideoHdr)
map.putBoolean("supportsPhotoHDR", supportsPhotoHdr)
map.putBoolean("supportsVideoHdr", supportsVideoHdr)
map.putBoolean("supportsPhotoHdr", supportsPhotoHdr)
map.putBoolean("supportsDepthCapture", supportsDepthCapture)
map.putString("autoFocusSystem", AutoFocusSystem.CONTRAST_DETECTION.unionValue)
map.putArray("videoStabilizationModes", createStabilizationModes())
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
val image = reader.acquireLatestImage()
onPhotoCaptured(image)
}, CameraQueues.cameraQueue.handler)
val output = PhotoOutput(imageReader, configuration.enableHdr)
val output = PhotoOutput(imageReader, configuration.photoHdr)
outputs.add(output.toOutputConfiguration(characteristics))
photoOutput = output
}
Expand All @@ -305,7 +305,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
isSelfie,
video.config.enableFrameProcessor
)
val output = VideoPipelineOutput(videoPipeline, configuration.enableHdr)
val output = VideoPipelineOutput(videoPipeline, configuration.videoHdr)
outputs.add(output.toOutputConfiguration(characteristics))
videoOutput = output
}
Expand All @@ -327,7 +327,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
preview.config.surface,
size,
SurfaceOutput.OutputType.PREVIEW,
configuration.enableHdr
configuration.videoHdr
)
outputs.add(output.toOutputConfiguration(characteristics))
previewOutput = output
Expand Down Expand Up @@ -398,7 +398,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
if (fps != null) {
if (!CAN_DO_60_FPS) {
// If we can't do 60 FPS, we clamp it to 30 FPS - that's always supported.
fps = Math.min(30, fps)
fps = 30.coerceAtMost(fps)
}
captureRequest.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(fps, fps))
}
Expand Down Expand Up @@ -427,7 +427,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam

// Set HDR
// TODO: Check if that value is even supported
if (config.enableHdr) {
if (config.videoHdr) {
captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_HDR)
} else if (config.enableLowLightBoost) {
captureRequest.set(CaptureRequest.CONTROL_SCENE_MODE, CaptureRequest.CONTROL_SCENE_MODE_NIGHT)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ data class CameraDeviceFormat(
val maxZoom: Double,
val videoStabilizationModes: List<VideoStabilizationMode>,
val autoFocusSystem: AutoFocusSystem,
val supportsVideoHDR: Boolean,
val supportsPhotoHDR: Boolean,
val supportsVideoHdr: Boolean,
val supportsPhotoHdr: Boolean,
val pixelFormats: List<PixelFormat>,
val supportsDepthCapture: Boolean
) {
Expand Down Expand Up @@ -50,8 +50,8 @@ data class CameraDeviceFormat(
value.getDouble("maxZoom"),
videoStabilizationModes,
autoFocusSystem,
value.getBoolean("supportsVideoHDR"),
value.getBoolean("supportsPhotoHDR"),
value.getBoolean("supportsVideoHdr"),
value.getBoolean("supportsPhotoHdr"),
pixelFormats,
value.getBoolean("supportsDepthCapture")
)
Expand Down
5 changes: 3 additions & 2 deletions package/example/src/CameraPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
const fps = Math.min(format?.maxFps ?? 1, targetFps)

const supportsFlash = device?.hasFlash ?? false
const supportsHdr = format?.supportsPhotoHDR
const supportsHdr = format?.supportsPhotoHdr
const supports60Fps = useMemo(() => device?.formats.some((f) => f.maxFps >= 60), [device?.formats])
const canToggleNightMode = device?.supportsLowLightBoost ?? false

Expand Down Expand Up @@ -181,7 +181,8 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
device={device}
format={format}
fps={fps}
hdr={enableHdr}
photoHdr={enableHdr}
videoHdr={enableHdr}
lowLightBoost={device.supportsLowLightBoost && enableNightMode}
isActive={isActive}
onInitialized={onInitialized}
Expand Down
5 changes: 3 additions & 2 deletions package/ios/CameraView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ public final class CameraView: UIView, CameraSessionDelegate {
// props that require format reconfiguring
@objc var format: NSDictionary?
@objc var fps: NSNumber?
@objc var hdr = false
@objc var videoHdr = false
@objc var photoHdr = false
@objc var lowLightBoost = false
@objc var orientation: NSString?
// other props
Expand Down Expand Up @@ -164,7 +165,7 @@ public final class CameraView: UIView, CameraSessionDelegate {
if video || enableFrameProcessor {
config.video = .enabled(config: CameraConfiguration.Video(pixelFormat: getPixelFormat(),
enableBufferCompression: enableBufferCompression,
enableHdr: hdr,
enableHdr: videoHdr,
enableFrameProcessor: enableFrameProcessor))
} else {
config.video = .disabled
Expand Down
3 changes: 2 additions & 1 deletion package/ios/CameraViewManager.m
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ @interface RCT_EXTERN_REMAP_MODULE (CameraView, CameraViewManager, RCTViewManage
// device format
RCT_EXPORT_VIEW_PROPERTY(format, NSDictionary);
RCT_EXPORT_VIEW_PROPERTY(fps, NSNumber);
RCT_EXPORT_VIEW_PROPERTY(hdr, BOOL);
RCT_EXPORT_VIEW_PROPERTY(videoHdr, BOOL);
RCT_EXPORT_VIEW_PROPERTY(photoHdr, BOOL);
RCT_EXPORT_VIEW_PROPERTY(lowLightBoost, BOOL);
RCT_EXPORT_VIEW_PROPERTY(videoStabilizationMode, NSString);
RCT_EXPORT_VIEW_PROPERTY(pixelFormat, NSString);
Expand Down
2 changes: 1 addition & 1 deletion package/ios/Core/CameraConfiguration.swift
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ extension CameraConfiguration.Video {

// Find the best matching format
guard let format = videoOutput.findPixelFormat(firstOf: targetFormats) else {
throw CameraError.format(.invalidHdr)
throw CameraError.format(.invalidVideoHdr)
}
// YUV 4:2:0 10-bit (compressed/uncompressed)
return format
Expand Down
10 changes: 5 additions & 5 deletions package/ios/Core/CameraError.swift
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ enum DeviceError: String {

enum FormatError {
case invalidFps(fps: Int)
case invalidHdr
case invalidVideoHdr
case invalidFormat
case incompatiblePixelFormatWithHDR

Expand All @@ -118,8 +118,8 @@ enum FormatError {
return "invalid-format"
case .invalidFps:
return "invalid-fps"
case .invalidHdr:
return "invalid-hdr"
case .invalidVideoHdr:
return "invalid-video-hdr"
case .incompatiblePixelFormatWithHDR:
return "incompatible-pixel-format-with-hdr-setting"
}
Expand All @@ -131,8 +131,8 @@ enum FormatError {
return "The given format was invalid. Did you check if the current device supports the given format in `device.formats`?"
case let .invalidFps(fps):
return "The given format cannot run at \(fps) FPS! Make sure your FPS is lower than `format.maxFps` but higher than `format.minFps`."
case .invalidHdr:
return "The currently selected format does not support HDR capture! Make sure you select a format which includes `supportsPhotoHDR`/`supportsVideoHDR`!"
case .invalidVideoHdr:
return "The currently selected format does not support 10-bit Video HDR streaming! Make sure you select a format which includes `supportsVideoHdr`!"
case .incompatiblePixelFormatWithHDR:
return "The currently selected pixelFormat is not compatible with HDR! HDR only works with the `yuv` pixelFormat."
}
Expand Down
Loading
Loading