[meta-freescale] webrtc question

[Ed Sutter]

>                From your description it seems that the webRTC will be the
>                 'communication channel' and the processing of the packages
>                 will be a
>                 second step (audio / video / whatever). For VP8 it
>         does work,
>                 as it is
>                 supported in Chromium.
>
>
>         Unfortunately, it is not that simple. webrtc & libjingle is a
>         long going project from Google that is done as a big C++
>         framework that has everything you need: encoding, decoding
>         libs, communication libs and debug stuff. Chrome is just
>         hooked up to use that framework when webrtc actions appear in
>         users javascript. webrtc does not depend on HTML5 video
>         playback for example, where we have with patches imx-vpu in
>         action. At least that is how it did work around a year ago.
>
>     Similarly, HTML5 doesn't use VP8 unless webrtc is involved right?
>     I'm hoping I don't need to get near any HTML at all, and can use
>     libjingle along with the audio and video engines to do this.
>
>
> IIRC, no, <video> tag is handled by Chrome, but when you start "webrtc 
> context", Chrome just proxies everything to webrtc library and waits 
> for answer from it. It is due to webrtc nature of being a separate 
> library ready for integration to any application. This is how Google 
> wants it.
>
> What do you mean by not getting near HTML? If you just want to use 
> webrtc in your application it is one situation, when you want a 
> "webrtc context" inside Chrome being accelerated - that is another 
> situation. Sadly, both of these situations will need you to dig into 
> webrtc sources and add imx-vpu support. There is no "easy way" here.
My use of this will (hopefully) not require a browser; hence, no HTML.  
I just want to use webrtc libraries in my application.
I've dug into this a few years ago (pre-webrtc-availability), using VP8 
alone in a different project.  Wasn't too messy but that
was prior to the point when all this "stuff" was integrated into 
webrtc.   Also, then I was working on a PC (no acceleration needed).
The difference here is that iMX6-quad != Intel-I7-quad; so I assume I'll 
need as much acceleration as I can get.
Really not sure what burden this will put on the iMX6Q, so that is yet 
to be determined.
>
>
>
>             Yea, the first step is to make my iMX6 based device be
>         compatible
>             with the
>             communication channels (and data formats, i.e. VP8/opus/etc..)
>             used/supported
>             by webrtc.  The difference will be that (in certain modes)
>         there
>             is no user on
>             the device; it will be headless and autonomous, so I need
>         to be
>             able to feed
>             media to and extract media from the underlying webrtc base
>             platform on the iMX6.
>             Does Chromium provide any non-browser type of hooks into
>         webrtc
>             down below?
>
>
>         No, chromium links with webrtc which links with libvpx for
>         encoding/decoding. Libvpx then decides what to use.
>
>         I investigated this some time ago while working on a similar
>         device (webrtc app linked with Qt5 front end on ARM
>         linux-native device communicating other similar devices, it
>         was a robotics project). You need to integrate imx vpu
>         encoding/decoding support to webrtc/libvpx internals or use
>         Android where it should be passed through from standard video
>         acceleration frameworks.
>
>     So does that imply that there is no hardware (imx-vpu)
>     acceleration support for VP8?
>     If this is true, is there any data on how the impacts CPU usage
>     when VP8 is used?
>
>
> Chrome HTML5 VP8 video playback acceleration != WebRTC VP8 decoding 
> acceleration! (see explanation above about webrtc being separate 
> library with own encoders/decoders).
> HTML5 VP8 playback works, webrtc video decoding does not.
Hmmm...(get ready for a naive comment)...
I would have thought once you get into VP8 then regardless of who calls 
it, the acceleration would be the same.
I realize it can be tuned and run in different modes; however I would 
have thought that
the bowels of the algorithm that acceleration would be applied to would 
be below this.
>
>     BTW... did you ever complete that project?  If not, was it due to
>     iMX6/webrtc issues?
>
>
> This was a long going project on Samsung Exynos SoCs and we ended up 
> just running it there unaccelerated, just bare webrtc+libvpx, since 
> 480p was enough for robotics. We were trying to move to i.MX, but at 
> that point only LTIB existed and it was soft-float (which was a 
> showstopper).
>
> But now, you have much more chances to get webrtc running great on 
> i.MX when HTML5 acceleration works in Chrome. Just adapt that code to 
> webrtc's internals and link your Chrome with it.
>
> I remember asking Carlos (author of imx patches for Chrome) about this 
> on IRC, but, unfortunately, I don't remember his answers. Hope he will 
> explain better than me :)
>
> Other than here, you might want to ask at webrtc google group 
> (https://groups.google.com/forum/#!forum/discuss-webrtc 
> <https://groups.google.com/forum/#%21forum/discuss-webrtc>) about how 
> to integrate video acceleration to the library.
>
Thanks very much for your comments.
Regards,
Ed