[meta-freescale] webrtc question

[Dmitriy B.]

2015-01-07 1:10 GMT+03:00 Ed Sutter <ed.sutter at alcatel-lucent.com>:

> Hi Dmitry,
> Questions embedded,
> Thanks
> Ed
>
>>
>>          From your description it seems that the webRTC will be the
>>         'communication channel' and the processing of the packages
>>         will be a
>>         second step (audio / video / whatever). For VP8 it does work,
>>         as it is
>>         supported in Chromium.
>>
>>
>> Unfortunately, it is not that simple. webrtc & libjingle is a long going
>> project from Google that is done as a big C++ framework that has everything
>> you need: encoding, decoding libs, communication libs and debug stuff.
>> Chrome is just hooked up to use that framework when webrtc actions appear
>> in users javascript. webrtc does not depend on HTML5 video playback for
>> example, where we have with patches imx-vpu in action. At least that is how
>> it did work around a year ago.
>>
> Similarly, HTML5 doesn't use VP8 unless webrtc is involved right?
> I'm hoping I don't need to get near any HTML at all, and can use libjingle
> along with the audio and video engines to do this.


IIRC, no, <video> tag is handled by Chrome, but when you start "webrtc
context", Chrome just proxies everything to webrtc library and waits for
answer from it. It is due to webrtc nature of being a separate library
ready for integration to any application. This is how Google wants it.

What do you mean by not getting near HTML? If you just want to use webrtc
in your application it is one situation, when you want a "webrtc context"
inside Chrome being accelerated - that is another situation. Sadly, both of
these situations will need you to dig into webrtc sources and add imx-vpu
support. There is no "easy way" here.


>
>>     Yea, the first step is to make my iMX6 based device be compatible
>>     with the
>>     communication channels (and data formats, i.e. VP8/opus/etc..)
>>     used/supported
>>     by webrtc.  The difference will be that (in certain modes) there
>>     is no user on
>>     the device; it will be headless and autonomous, so I need to be
>>     able to feed
>>     media to and extract media from the underlying webrtc base
>>     platform on the iMX6.
>>     Does Chromium provide any non-browser type of hooks into webrtc
>>     down below?
>>
>>
>> No, chromium links with webrtc which links with libvpx for
>> encoding/decoding. Libvpx then decides what to use.
>>
>> I investigated this some time ago while working on a similar device
>> (webrtc app linked with Qt5 front end on ARM linux-native device
>> communicating other similar devices, it was a robotics project). You need
>> to integrate imx vpu encoding/decoding support to webrtc/libvpx internals
>> or use Android where it should be passed through from standard video
>> acceleration frameworks.
>>
> So does that imply that there is no hardware (imx-vpu) acceleration
> support for VP8?
> If this is true, is there any data on how the impacts CPU usage when VP8
> is used?
>

Chrome HTML5 VP8 video playback acceleration != WebRTC VP8 decoding
acceleration! (see explanation above about webrtc being separate library
with own encoders/decoders).

HTML5 VP8 playback works, webrtc video decoding does not.


> BTW... did you ever complete that project?  If not, was it due to
> iMX6/webrtc issues?


This was a long going project on Samsung Exynos SoCs and we ended up just
running it there unaccelerated, just bare webrtc+libvpx, since 480p was
enough for robotics. We were trying to move to i.MX, but at that point only
LTIB existed and it was soft-float (which was a showstopper).

But now, you have much more chances to get webrtc running great on i.MX
when HTML5 acceleration works in Chrome. Just adapt that code to webrtc's
internals and link your Chrome with it.

I remember asking Carlos (author of imx patches for Chrome) about this on
IRC, but, unfortunately, I don't remember his answers. Hope he will explain
better than me :)

Other than here, you might want to ask at webrtc google group (
https://groups.google.com/forum/#!forum/discuss-webrtc) about how to
integrate video acceleration to the library.

Best Regards,
Dmitry Beykun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/meta-freescale/attachments/20150107/37d7dfdb/attachment.html>