[meta-freescale] I.MX6 and gstreamer glvideomixer

[Jean-Michel Hautbois]

Hi,

I am trying to use openGL accelerated elements from gstreamer with my i.MX6.
It works, but this is not very impressive.
For instance :
DISPLAY=0:0 GST_GL_PLATFORM=egl gst-launch-1.0 videotestsrc !
'video/x-raw,width=320,height=240' ! glvideomixer ! ximagesink

And this consumes 45% of CPU (on a I.MX6D).
With this :
DISPLAY=0:0 GST_GL_PLATFORM=egl gst-launch-1.0 videotestsrc !
'video/x-raw,width=320,height=240' ! glvideomixer ! glimagesink
It consumes ~15% CPU which is much better.

Now, I want to have something more complicated, using a video
(640x360) displayed in two boxes (no scaling).
Here is the pipeline :
DISPLAY=0:0 GST_GL_PLATFORM=egl gst-launch-1.0 -vvv \
filesrc location=video3.mp4 ! qtdemux name=demux demux. ! queue !
h264parse ! imxvpudec \
    ! queue ! tee name=t \
glvideomixer name=mix \
      sink_0::xpos=300 sink_0::ypos=360   sink_0::zorder=1\
      sink_1::xpos=980 sink_1::ypos=360  sink_1::zorder=2\
    ! queue ! progressreport ! glimagesink \
t. ! queue ! mix.sink_0 \
t. ! queue ! mix.sink_1


I cannot have it realtime, and I get the following log messages :
[INFO]    Product Info: i.MX6Q/D/S
Pipeline is PREROLLING ...
Got context from element 'glimagesink0': gst.gl.GLDisplay=context,
gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps =
"video/x-h264\,\ stream-format\=\(string\)avc\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\
codec_data\=\(buffer\)014d4028ffe1000b274d40288b9501405ff2a001000528fe01ae20\,\
width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps =
"video/x-h264\,\ stream-format\=\(string\)avc\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\
codec_data\=\(buffer\)014d4028ffe1000b274d40288b9501405ff2a001000528fe01ae20\,\
width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps =
"video/x-h264\,\ stream-format\=\(string\)avc\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\
codec_data\=\(buffer\)014d4028ffe1000b274d40288b9501405ff2a001000528fe01ae20\,\
width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"
/GstPipeline:pipeline0/GstImxVpuDec:imxvpudec0.GstPad:sink: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"
[INFO]    bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0
/GstPipeline:pipeline0/GstImxVpuDec:imxvpudec0.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstTee:t.GstTeePad:src_0: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue3.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstTee:t.GstTeePad:src_1: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue3.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue4.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue4.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstTee:t.GstPad:sink: caps = "video/x-raw\,\
format\=\(string\)I420\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstGLVideoMixer:mix.GstGLVideoMixerPad:sink_0:
caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstGLVideoMixer:mix.GstGLVideoMixerPad:sink_1:
caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)640\,\
height\=\(int\)360\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstGLVideoMixer:mix.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:src: caps =
"video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstProgressReport:progressreport0.GstPad:src:
caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstGLImageSink:glimagesink0.GstPad:sink: caps =
"video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
/GstPipeline:pipeline0/GstProgressReport:progressreport0.GstPad:sink:
caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1620\,\
height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\
interlace-mode\=\(string\)progressive\,\
colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)7500/247"
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
WARNING: from element
/GstPipeline:pipeline0/GstGLImageSink:glimagesink0: A lot of buffers
are being dropped.
Additional debug info:
/home/marie/Projets/vodabox3/poky/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer1.0/1.4.1-r0/gstreamer-1.4.1/libs/gst/base/gstbasesink.c(2789):
gst_base_sink_is_too_late ():
/GstPipeline:pipeline0/GstGLImageSink:glimagesink0:
There may be a timestamping problem, or this computer is too slow.
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"
/GstPipeline:pipeline0/GstImxVpuDec:imxvpudec0.GstPad:sink: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"
progressreport0 (00:00:05): 4 / 27 seconds (14.8 %)
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"
/GstPipeline:pipeline0/GstImxVpuDec:imxvpudec0.GstPad:sink: caps =
"video/x-h264\,\ stream-format\=\(string\)byte-stream\,\
alignment\=\(string\)au\,\ level\=\(string\)4\,\
profile\=\(string\)main\,\ width\=\(int\)640\,\ height\=\(int\)360\,\
framerate\=\(fraction\)7500/247\,\
pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true"


Did I forget something in my package configuration (this is a custom image) ?
Is this normal ? I thought GPU would handle compositing in
glvideomixer easily but it seems to be not so simple :-).

Thanks,
JM