[meta-freescale] QtMultimedia on i.MX6

[Eric Nelson]

On 07/04/2013 02:42 PM, Otavio Salvador wrote:
> On Thu, Jul 4, 2013 at 12:14 PM, Thomas Senyk
> <thomas.senyk at pelagicore.com> wrote:
>> On Thursday, 04 July, 2013 17:03:03 Thomas Senyk wrote:
>>> Hi,
>>>
>>> I've started to investigate to see the QtMultimedia-gstreamer integration on
>>> the i.MX6 could be optimized.
>>>
>>> I'm writing this mail to sync with others who might already done some work
>>> or investigations on this matter, to get hints where to start and what's
>>> worth to investigate.
>>>
>>> State right now:
>>>
>>> The HW-decoders are used (according to gstreamer log).
>>> That's a very good start/out of the box experience! Good job..freescale I
>>> guess?
>>>
>>> There is still a lot of cpu load:
>>>     - 720p: smooth playback but~18% complete cpu load => ~70% of one core
>>>     - 1080p: playback not smooth and ~27% complete cpu load -> 100% of one
>>> core + some additional cpu load
>>>
>>> The obvious reason for this cpu load is some sort of memcopy and/or
>>> conversion.
>>>    ... gstreamer doesn't render into gpu-memory directly.
>>>    ... or (if you want it the other way around) the gstreamer (/cpu) memory
>>> isn't mapped into gpu-memory (which is possibly on the imx6)
>>>
>>> Started to do some digging:
>>>
>>> One of the first things I tried was to compile gst-plugin-gl (which is part
>>> of the meta-fsl-arm layer via a .bbappend).
>>> The current recipe is driver-version-1.1.0 based and is not building out of
>>> the box (at least not for me) ... I got gst-plugins-gl-0.10.3.1.tar.gz
>>> (based on 4.0.0) building.**
>>>
>>> So, after I got it deployed I hoped for magic to happen and everything is
>>> perfect now ... no big surprise: it's not :) nothing changed, possibly the
>>> gst-pipline build by qtmultimedia is the ame.
>>>
>>> The reason for this mail / some questions:
>>>
>>> So I though, I write this mail before I waste a lot of time.
>>> What does people think is the best way forward?
>>> Try to convince QtMultimedia to use glsink?
>>> Try to use a fakesink/memory-based-sink and try to map the cpu memory into
>>> gpu memory via vivante APIs?
>>>
>>> Does anyone have c/c++ code I could try?
>>> Ideally for direct framebuffer/EGL use (I'm not using X11)
>>> I found: https://github.com/andreluizeng/i.MX6-Video-Streaming-Texture.git
>>>   .. but it's crashing rather early and I'm not sure the if the time is well
>>> spend fix this code? Anyone used it before?
>>> Does anyone know if gst-plugins-gl is applicable for framebuffer/EGL setups?
>>>
>>>
>>> Greet
>>> Thomas
>>
>> Ok ... what is it with me and sending mails 1minute to early all the time ...
>> After I pressed "send", I had a very simple and obvious idea .. why not just
>> simply try glimagesink instead of mfw_v4lsink:
>>
>> gst-launch filesrc location=sintel_trailer-1080p.mp4 typefind=true ! aiurdemux !
>> vpudec !  glimagesink
>>
>>   ... long shot, but:
>> It's actually working!
>> It just opens /dev/fb0 and renders the video into it.
>>
>> 1080p with 0%(!!) cpu load!
>>

Nice! I stopped short of testing this out because I thought that X was
required.

A quick browse of the sources indicates that there's some really good
stuff inside, including what appears to be the ability to upload
planar YUV into a texture:
	http://bitbucket.org/Freescale/gstreamer-gst-plugins-gl/src/629e1efaf87f379a36456f4e42cdec0ca2ed7e2a/gst-libs/gst/gl/gstglvivante.c?at=fsl/0.10.3#cl-103

>> I think that pretty much answers all questions I had ;)
>> The way forward: patch QtMultimedia to use glimagesink
>>   ... I hope it's going to be trivial to just give him a active FBO or texture.
>>
>>
>> Sorry to everybody who read my way to long mail ;)
>
> It was very good read.
+1

 > By the way, Rogerio (add in Cc) started to try
> to get the gst-plugin-gl code more manageble in bitbucket
> (https://bitbucket.org/Freescale/gstreamer-gst-plugins-gl/commits/all)
> but it was not finished. Maybe you could help on this and we could try
> to merge the Vivante backend upstream? This would allow us to drop the
> bbappend completely.
>
+1

>>> ** some more info in gst-plugin-gl:
>>> It's 'detected' on the board:
>>> root at nitrogen-1:~# gst-inspect |grep gl
>>> opengl:  glupload: OpenGL upload
>>> opengl:  gldownload: OpenGL video maker
>>> opengl:  glimagesink: OpenGL video sink
>>> opengl:  glfiltercube: OpenGL cube filter
>>> opengl:  gleffects: Gstreamer OpenGL Effects
>>>
>>> ... if someone is interested on how I got it building, let me know. If I got
>>> something useful out of it (read: "it's tested") I'll start a discussion on
>>> how to upstream it properly into meta-fsl-arm (by the way: I find the
>>> current setup of how to bbappend and it's patch looks like very weird)

I'm interested in the details of how you added Qt5 to your
build. Qt4 seems to be directly accessible in Yocto, but not Qt5.

Regards,


Eric