add support for nvenc and accel_assist

[jsorg71]

No description provided.

[matt335672]

Fix some build issues with:-

--- a/scripts/install_xrdp_build_dependencies_with_apt.sh
+++ b/scripts/install_xrdp_build_dependencies_with_apt.sh
@@ -74,6 +74,7 @@ case "$ARCH"
 in
     amd64)
         PACKAGES_AMD64_MIN=" \
+            libepoxy-dev \
             libpam0g-dev \
             libssl-dev \
             libx11-dev \
@@ -118,6 +119,7 @@ in
         PACKAGES="$PACKAGES \
             g++-multilib \
             gcc-multilib \
+            libepoxy-dev:i386 \
             $LIBFREETYPE_DEV:i386 \
             libgl1-mesa-dev:i386 \
             libglu1-mesa-dev:i386 \

Also:-

1. some files are missing from the xrdp_accel_assist directory; xrdp_accel_assist_yami.c xrdp_accel_assist_yami.h encoder_headers/yami_inf.h
2. need to run astyle

[matt335672]

Happy to merge this as-is, when CI is fixed.

[matt335672]

Suggest adding error logging if this doesn't work, for triaging user issues.

[jsorg71]

I don't t think the log ing system is initialized at this point. Can we still log?

[matt335672]

You're quite right. Calling LOG() won't work.

Two possible solutions:-

1. Log errors to stderr in this function. This isn't ideal, but it will at least get picked up by the journal on systemd-based systems.
2. Use a separate function for this purpose (g_cu_init()?) called explicitly from sesman after logging has started. We can remove it if you ever get a better answer as to why this function is required.

[matt335672]

I don't want to hold up merging this, but I'd like to ask if we think the hack to call cuInit() is in the right place. Wouldn't it be better done as part of the session if NVidia is being used?

[jsorg71]

I don't want to hold up merging this, but I'd like to ask if we think the hack to call cuInit() is in the right place. Wouldn't it be better done as part of the session if NVidia is being used?

I tried to just call cuInit() in the session but as the user it fails. Once the call is made as root, something gets 'driver init' and then it will work as the user. I guess xrdp might not be running as root, is that right? Also, is this a security risk?
In order to reproduce this problem, you have to boot a server up without X GUI running as Xorg on console also does the 'driver init'.
I tried to use nvida_persistence or something like that as it claimed to fix this but it didn't work.
I've been meaning to ask NVidia about this but the problem is hard to describe.

[jsorg71]

I rebased this branch on current devel

[jsorg71]

Should we make xrdp_accel_assit a compile time flag because with it, epoxy and gbm are required to build xrdp. We didn't need that before? Xorg, or at least the modesetting driver requires the run time libraries but now we require the development packages and the user may never plan to use xrdp_accel_assist.

[matt335672]

As far as the cuInit() goes, either sesman or sesexec runs as root. Since #2974, xrdp may not be running as root, and in fact it won't be on Debian/Ubuntu anyway - they patched it to run as non-root a long time ago.

I think a compile-time option is the best way to go, for platforms that may not support NVidia acceleration (are there any?).

[jsorg71]

As far as the cuInit() goes, either sesman or sesexec runs as root. Since #2974, xrdp may not be running as root, and in fact it won't be on Debian/Ubuntu anyway - they patched it to run as non-root a long time ago.

I think a compile-time option is the best way to go, for platforms that may not support NVidia acceleration (are there any?).

done and done

[jsorg71]

accel_assist is using environment variables set int sesman.ini, but I think it might be better to use gfx.toml.

these are the current env vars. @metalefty can accell_assist use gfx.toml, does that make sense to use?

XRDP_NVENC_RATE_CONTROL_MODE
XRDP_NVENC_AVERAGE_BITRATE
XRDP_NVENC_QP

[metalefty]

Yes, it looks good to me.

[matt335672]

One comment on recent changes. I'm still happy to merge this sooner rather than later and work in devel if that suits.

[matt335672]

You're quite right. Calling LOG() won't work.

Two possible solutions:-

1. Log errors to stderr in this function. This isn't ideal, but it will at least get picked up by the journal on systemd-based systems.
2. Use a separate function for this purpose (g_cu_init()?) called explicitly from sesman after logging has started. We can remove it if you ever get a better answer as to why this function is required.

[jsorg71]

I'm good to merge this now and then continue work in devel.

[matt335672]

Fine by me - it will get more exposure that way.

I've got #3346 to merge once @metalefty has looked at it, but it won't affect this at all.

[Nexarian]

Agreed.