[meta-freescale] QtMultimedia on i.MX6

[Thomas Senyk]

On Thursday, 04 July, 2013 17:03:03 Thomas Senyk wrote:
> Hi,
> 
> I've started to investigate to see the QtMultimedia-gstreamer integration on
> the i.MX6 could be optimized.
> 
> I'm writing this mail to sync with others who might already done some work
> or investigations on this matter, to get hints where to start and what's
> worth to investigate.
> 
> 
> 
> State right now:
> 
> The HW-decoders are used (according to gstreamer log).
> That's a very good start/out of the box experience! Good job..freescale I
> guess?
> 
> There is still a lot of cpu load:
>    - 720p: smooth playback but~18% complete cpu load => ~70% of one core
>    - 1080p: playback not smooth and ~27% complete cpu load -> 100% of one
> core + some additional cpu load
> 
> The obvious reason for this cpu load is some sort of memcopy and/or
> conversion.
>   ... gstreamer doesn't render into gpu-memory directly.
>   ... or (if you want it the other way around) the gstreamer (/cpu) memory
> isn't mapped into gpu-memory (which is possibly on the imx6)
> 
> 
> 
> 
> Started to do some digging:
> 
> One of the first things I tried was to compile gst-plugin-gl (which is part
> of the meta-fsl-arm layer via a .bbappend).
> The current recipe is driver-version-1.1.0 based and is not building out of
> the box (at least not for me) ... I got gst-plugins-gl-0.10.3.1.tar.gz
> (based on 4.0.0) building.**
> 
> So, after I got it deployed I hoped for magic to happen and everything is
> perfect now ... no big surprise: it's not :) nothing changed, possibly the
> gst-pipline build by qtmultimedia is the ame.
> 
> 
> 
> The reason for this mail / some questions:
> 
> So I though, I write this mail before I waste a lot of time.
> What does people think is the best way forward?
> Try to convince QtMultimedia to use glsink?
> Try to use a fakesink/memory-based-sink and try to map the cpu memory into
> gpu memory via vivante APIs?
> 
> Does anyone have c/c++ code I could try?
> Ideally for direct framebuffer/EGL use (I'm not using X11)
> I found: https://github.com/andreluizeng/i.MX6-Video-Streaming-Texture.git
>  .. but it's crashing rather early and I'm not sure the if the time is well
> spend fix this code? Anyone used it before?
> Does anyone know if gst-plugins-gl is applicable for framebuffer/EGL setups?
> 
> 
> Greet
> Thomas

Ok ... what is it with me and sending mails 1minute to early all the time ...
After I pressed "send", I had a very simple and obvious idea .. why not just 
simply try glimagesink instead of mfw_v4lsink:

gst-launch filesrc location=sintel_trailer-1080p.mp4 typefind=true ! aiurdemux ! 
vpudec !  glimagesink

 ... long shot, but:
It's actually working!
It just opens /dev/fb0 and renders the video into it.

1080p with 0%(!!) cpu load!

I think that pretty much answers all questions I had ;)
The way forward: patch QtMultimedia to use glimagesink
 ... I hope it's going to be trivial to just give him a active FBO or texture.


Sorry to everybody who read my way to long mail ;)

> 
> ** some more info in gst-plugin-gl:
> It's 'detected' on the board:
> root at nitrogen-1:~# gst-inspect |grep gl
> opengl:  glupload: OpenGL upload
> opengl:  gldownload: OpenGL video maker
> opengl:  glimagesink: OpenGL video sink
> opengl:  glfiltercube: OpenGL cube filter
> opengl:  gleffects: Gstreamer OpenGL Effects
> 
> ... if someone is interested on how I got it building, let me know. If I got
> something useful out of it (read: "it's tested") I'll start a discussion on
> how to upstream it properly into meta-fsl-arm (by the way: I find the
> current setup of how to bbappend and it's patch looks like very weird)