WPE 2.38: OOM(Out of Memory) error with HLS and DASH playback while streaming any content with WebInspector open

[hridhya-narayanan-infosys]

We are seeing Out of memory error with HLS and DASH playback when streaming any content with webinspector open. We could see the memory max usage increases and then finally WebKitBrowser Plugin gets deactivated due to reason Memory Exceeded.

Below are the observations on Sagemcom VA device while streaming the content with webinspector open:

cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
sleep 60
done
688664576
688664576
688664576
688664576
688664576
688664576
688664576
688664576
692006912
705425408
715304960
730238976
747614208
773545984
774053888
784330752
794058752
805359616
817238016
830091264
844013568
849997824
862879744
871493632
883142656
895012864
904065024
915701760
923099136
931594240
959557632
966942720
973156352
975294464
975294464

"Monitor"data: {callsign: WebKitBrowser, action: Deactivate, reason: MemoryExceeded}
"Monitor"data: {callsign: WebKitBrowser, action: Activate, reason: Automatic}
"WebKitBrowser"data: {state: deactivated, reason: MemoryExceeded}

Scenario to reproduce:

1. Open any DASH/HLS based app and start playback.
2. Open Webinspector
3. Observe memory usage.

Below are the logs from Sagemcom VA for DASH and HLS playbacks:
Pluto TV (https://app-virginmedia360.pluto.tv/) HLS playback:
wpeframework.log

ITVX App (https://app.10ft.itv.com/3.449.2/virginmedia) DASH playback:
wpeframework.log

[pgorszkowski-igalia]

Unfortunately, I don't have access to PlutoTV and ITVX App so I cannot test them, so I focus on MVT tests. I found that in the case of Shaka tests, each test creates a new "video" element, and in theory, the existing one from the previous test is removed. Nevertheless, after adding some logs, I discovered that the "video" element from the previous test was not removed completely (probably shaka keeps some reference to it), which caused it to be kept in memory.

It works as expected in the case of "DashJS tests" and "Progressive HTML5 tests".

@hridhya-narayanan-infosys : Can you run PlutoTV or/and ITVX with WEBKIT_DEBUG=Media,MemoryPressure to gather more info (and reproduce OOM with RWI) and provide me logs?

[hridhya-narayanan-infosys]

@pgorszkowski-igalia should I need to share logs with including the downstream patch changes 39f3e68 which is shared as part of #1335 ?

[pgorszkowski-igalia]

@hridhya-narayanan-infosys : yes

[hridhya-narayanan-infosys]

@pgorszkowski-igalia Below are the logs from Sagemcom VA for PlutoTV and ITVX app with enabling WEBKIT_DEBUG=Media,MemoryPressure.
I have ran playback from these apps and collected logs when OOM is observed with webinspector open.

wpeframework-Pluto.log
wpeframework-itvx.log

[pgorszkowski-igalia]

From your logs I see that you have 200MB set as threshold for memory pressure which is rather small value. Keep in mind that if you use RWI it will gather additional 200MB of data at least for network data kept for RWI.

Jan 20 08:00:53 m393-72180ZB-stb WPEFramework[4805]: MemoryPressure: Critical memory usage (PID=4805) [MB]: 226/200, video: 0/1024

As a consequence of low memory threshold value, you exceed it quickly (even more quickly with RWI).

[pgorszkowski-igalia]

@hridhya-narayanan-infosys : do you know what js library is used for adaptive media playback in plutotv and itvx?

I am investigating MVT frameworks and it seems that in case of shaka player tests, the old media element is not removed by GC because something keeps reference on the old media element (video element from the previous test). The consequence of this is that memory usage is greater because of the data from the old video elements which are not removed even after GC.

[hridhya-narayanan-infosys]

@pgorszkowski-igalia , Below are the details which I got from webinspector page.

For Pluto TV app, I could see for HLS playback it uses [email protected] and for DASH it uses dash.all.min.js

    <script defer="defer" src="/assets/js/hlsjs/[email protected]" type="text/javascript"></script>
    <script defer="defer" src="https://cdnjs.cloudflare.com/ajax/libs/dashjs/4.6.0/dash.all.min.js" type="text/javascript"></script>

For ITVX app, we could not find any player related info from webinspector, seems like it's using their own player. I have attached snapshots from webinspector page: