[Feat]: Support for devices whose MountAngle is not general

[maehata-fairy]

Version

2.5.2 (or commit: 3a9af52)

Environment that reproduces the issue

Pixel4 - AOSP 10 (Rooted, Custom OS)
Root is required because the MountAngle in camera_config.xml, which is the ReadOnly configuration file of the OS, needs to be rewritten.
This can be reproduced by changing the angle to 0 or 180.

Use case description

Thanks for this project, it has been very helpful as I was trying to implement SRT. Thank you very much.

After using the demo and libraries of this project, I found that
It seems that devices with camera angles where the MountAngle is 0 or 180 are not oriented correctly.
MountAngle in camera_config.xml is the value of SENSOR_ORIENTATION in CameraCharacteristics.
e.g. https://github.com/LineageOS/android_device_bq_sdm660-common/blob/lineage-16.0/configs/camera/camera_config.xml

For example, if you request Size(1280, 720), then
A camera with MountAngle=90 will preview and deliver a 1280 x 720 image with the correct proportions.
However, with a camera with MountAngle=0, the captured video with 1280 horizontal and 720 vertical will be distorted, as if it were scaled down or enlarged to 1280 vertical and 720 horizontal.

The following is a capture of the distorted image.
(Android preview screen and the image distributed by SRT)

• 

• 

After applying my patch (please check alternative solutions.)

• 

Proposed solution

No response

Alternative solutions

I have created a patch to work with the less common MountAngle devices.
Here is the Dirty code. Sry.

diff --git a/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/DeviceOrientationProvider.kt b/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/DeviceOrientationProvider.kt
index 2dc06e1..cc60558 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/DeviceOrientationProvider.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/DeviceOrientationProvider.kt
@@ -16,14 +16,22 @@
 package io.github.thibaultbee.streampack.internal.data.orientation
 
 import android.content.Context
+import android.hardware.camera2.CameraCharacteristics
 import android.util.Size
 import io.github.thibaultbee.streampack.internal.interfaces.IOrientationProvider
 import io.github.thibaultbee.streampack.internal.utils.extensions.deviceOrientation
 import io.github.thibaultbee.streampack.internal.utils.extensions.isDevicePortrait
 import io.github.thibaultbee.streampack.internal.utils.extensions.landscapize
 import io.github.thibaultbee.streampack.internal.utils.extensions.portraitize
+import io.github.thibaultbee.streampack.utils.getCameraCharacteristics
+import io.github.thibaultbee.streampack.internal.sources.camera.CameraCapture
 
 class DeviceOrientationProvider(private val context: Context) : IOrientationProvider {
+    private var cameraCapture: CameraCapture? = null
+    fun setCameraCapture(capture: CameraCapture) {
+        this.cameraCapture = capture
+    }
+
     override val orientation: Int
         get() {
             //TODO: this might not be working on all devices
@@ -31,6 +39,13 @@ class DeviceOrientationProvider(private val context: Context) : IOrientationProv
             return if (deviceOrientation == 0) 270 else deviceOrientation - 90
         }
 
+    override val cameraAngle: Int
+        get() {
+            val cameraId = this.cameraCapture?.cameraId ?: return 90
+            val characteristics = context.getCameraCharacteristics(cameraId)
+            return characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) as Int
+        }
+
     override fun orientedSize(size: Size): Size {
         return if (context.isDevicePortrait) {
             size.portraitize()
diff --git a/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/FixedOrientationProvider.kt b/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/FixedOrientationProvider.kt
index 8f5ee4f..e14c15f 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/FixedOrientationProvider.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/internal/data/orientation/FixedOrientationProvider.kt
@@ -21,7 +21,7 @@ import io.github.thibaultbee.streampack.internal.utils.extensions.landscapize
 import io.github.thibaultbee.streampack.internal.utils.extensions.portraitize
 import io.github.thibaultbee.streampack.utils.OrientationUtils
 
-class FixedOrientationProvider(override val orientation: Int) : IOrientationProvider {
+class FixedOrientationProvider(override val orientation: Int, override val cameraAngle: Int = 90) : IOrientationProvider {
     override fun orientedSize(size: Size): Size {
         return if (OrientationUtils.isPortrait(orientation)) {
             size.portraitize()
diff --git a/core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/VideoMediaCodecEncoder.kt b/core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/VideoMediaCodecEncoder.kt
index 1ed1709..1a0f2d5 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/VideoMediaCodecEncoder.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/internal/encoders/VideoMediaCodecEncoder.kt
@@ -88,8 +88,15 @@ class VideoMediaCodecEncoder(
         val videoConfig = config as VideoConfig
         orientationProvider.orientedSize(videoConfig.resolution).apply {
             // Override previous format
-            format.setInteger(MediaFormat.KEY_WIDTH, width)
-            format.setInteger(MediaFormat.KEY_HEIGHT, height)
+            val cameraAngle = orientationProvider.cameraAngle
+            
+            if (cameraAngle == 90 || cameraAngle == 270) {
+                format.setInteger(MediaFormat.KEY_WIDTH, width)
+                format.setInteger(MediaFormat.KEY_HEIGHT, height)
+            } else {
+                format.setInteger(MediaFormat.KEY_WIDTH, height)
+                format.setInteger(MediaFormat.KEY_HEIGHT, width)
+            }
         }
     }
 
diff --git a/core/src/main/java/io/github/thibaultbee/streampack/internal/interfaces/IOrientationProvider.kt b/core/src/main/java/io/github/thibaultbee/streampack/internal/interfaces/IOrientationProvider.kt
index 599d8ed..215bd72 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/internal/interfaces/IOrientationProvider.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/internal/interfaces/IOrientationProvider.kt
@@ -27,6 +27,12 @@ interface IOrientationProvider {
      */
     val orientation: Int
 
+   /**
+     * CameraSensor angle. Generally 90, 270.
+     * Expected values: 0, 90, 180, 270.
+     */
+     val cameraAngle: Int
+
     /**
      * Return the size with the correct orientation.
      */
diff --git a/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseCameraStreamer.kt b/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseCameraStreamer.kt
index d685267..f2016d1 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseCameraStreamer.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseCameraStreamer.kt
@@ -61,8 +61,13 @@ open class BaseCameraStreamer(
     initialOnErrorListener = initialOnErrorListener
 ), ICameraStreamer {
     private val cameraCapture = videoCapture as CameraCapture
+    private val deviceOrientationProvider = orientationProvider as DeviceOrientationProvider
     override val helper = CameraStreamerConfigurationHelper(muxer.helper)
 
+    init {
+        deviceOrientationProvider.setCameraCapture(cameraCapture)
+    }
+
     /**
      * Get/Set current camera id.
      */
diff --git a/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseStreamer.kt b/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseStreamer.kt
index cff625d..44df9ef 100644
--- a/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseStreamer.kt
+++ b/core/src/main/java/io/github/thibaultbee/streampack/streamers/bases/BaseStreamer.kt
@@ -59,7 +59,7 @@ abstract class BaseStreamer(
     private val context: Context,
     protected val audioCapture: IAudioCapture?,
     protected val videoCapture: IVideoCapture?,
-    orientationProvider: IOrientationProvider,
+    protected val orientationProvider: IOrientationProvider,
     private val muxer: IMuxer,
     protected val endpoint: IEndpoint,
     initialOnErrorListener: OnErrorListener? = null

[ThibaultBee]

Hi there,

I have the feeling that there was on issue with the orientation, but it is a really painfull subject 🧠
Could you check that tapping on StreamPack Preview properly set the focus on the same place as the preview (with your patch).
Do you have in mind an easy way to test this?

Best regards,
Thibault

[ThibaultBee]

As the preview and the stream are 2 different things, there might be 2 issues.
To display the preview on your device, what view do you use?

[maehata-fairy]

Hello.

The combination of device orientation and camera mount angle is a headache.

The patch I created focuses only on the stream.
The preview is not fixed and needs additional work. (sry...)
I will try more when I have time.

A simple test... This is also difficult.
Probably, you hard to get a reproducible device.
I think it would be better to do it on an android emulator, but i have to examine emulator's configuration.

[maehata-fairy]

Sorry I haven't had much time to do this....

I tried to build a simple environment.
It may be possible to build it with an "android emulator" and "external usb camera".

Some devices for auto (vehicles) like alps YT9213AJ support external usb camera and it returns 0 camera sensor orientation
android/camera-samples#361

The report says "YT9213AJ supports external usb camera and it returns 0 camera sensor orientation > ".

So,

• Connect an external usb camera (sensor orientation = 0) to the PC
• On the PC, specify the camera device when starting the emulator.
  e.g. emulator.exe -avd PixelC -camera-back webcam1.
  This may work.

However, I do not have an external usb camera at hand,
so I installed the virtual camera driver and confirmed that the emulator even uses the camera.
Unfortunately, the virtual camera driver recognized sensor orientation = 90, so it may not work.

Thank you.

[ThibaultBee]

Hi,
Sorry I missed you last messages.

The combination of device orientation and camera mount angle is a headache.

Agreed! It is not clear at all.

I will try to test with an android emulator and an external USB camera. Thanks for the information.

In the meantime, I reworked the management of the orientation. It is not going to fix your issue but it might be easier to debug.
Everything is in https://github.com/ThibaultBee/StreamPack/blob/5ae4592ecfe2e6f5ebcc1d49296bf5f1a068cbf0/core/src/main/java/io/github/thibaultbee/streampack/internal/sources/camera/CameraSource.kt#L130C14-L130C14
I guess your issue has a link with the default buffer size. But it is still unclear how to set setDefaultBufferSize.

[ThibaultBee]

I will try to test with an android emulator and an external USB camera. Thanks for the information.

Just did the test with my webcam and sensor orientation is also 90 degrees...
Looking for a way to test or for a clear explanation of https://developer.android.com/training/camera2/camera-preview

[maehata-fairy]

Hi
Thank you for taking this issue.

Just did the test with my webcam and sensor orientation is also 90 degrees...

I apologize that it did not behave as expected. I will conduct further investigation on my end as well.

From a technical point, I believe it is possible to build an emulator image with Angle other than 90 from AOSP. However, I consider that as a last resort..