Just a best practice recommendation

[OmkarShidore]

Hi, Thanks for making this framework available.
I wanted to know how to apply filters on the decoded frame before encoding it.

Input Frame (webCam) -> PyFrameUploader(on GPU) -> Apply some filters here -> nvEnc.EncodeSingleSurface

I wanted to apply multiple filters like RGB-HSV, guassian blur, histogram equalization. What would be the best practice for such operation, once the frame is on device. I don't want to move it back to the host.
PS: I've tried torch.cuda.tensor conversion and and applying filters in torchvision there. but those filters are very very slow.
Need a throughput of 4k 60 FPS.

[rarzumanyan]

Hi @OmkarShidore

Input Frame (webCam) -> PyFrameUploader(on GPU) -> Apply some filters here -> nvEnc.EncodeSingleSurface

That's exactly how you do it.
You may think about existing classes like PySurfaceConverter or PySurfaceResizer as video filters which you apply to your decoded frames before encoding.

wanted to apply multiple filters like RGB-HSV, guassian blur, histogram equalization

That would require to have such filters (in form of high-performance CUDA kernels) at your disposal. Some of them may be implemented with the help of NPP library.

[OmkarShidore]

Thanks for the quick response.

Is there any other way than writing CUDA kernels in NPP?
Any other SDKs that might help with this?

[rarzumanyan]

You don't need to write any CUDA kernels if you're using NPP. It gives you all sorts of C-style functions which launch CUDA under the hood.
E. g. NPP is used for color conversion in NPP and it could also be used for something like Gaussian blur.
However NPP is written in C so you would have to develop a new VPF class inheriting from Task to launch in from Python.

If you don't want to mess with C/C++ I assume you could use OpenCV python bindings, it does support CUDA to certain extent.

Any other SDKs that might help with this?

Unfortunately I'm not very informed in this regard.

[OmkarShidore]

Okay, understood. Thanks for the input 👍