[yocto] Yocto Realtime tests on beaglebone black

[William Mills]

On 02/12/2015 05:05 PM, Stephen Flowers wrote:
>
> So I ran cyclictest with an idle system and loaded with multiple
> instances of cat /dev/zero > /dev/null &
>

When I suggested filesystem activity I was suggesting getting a kernel
filesystem and a physical I/O device to be active.
The load above is just two character devices so not a ton of kernel
code is active.

If you are interested in pursuing this further I would write a script
that writes multiple files to MMC and then deletes them and do this in
a loop.

Perhaps Bruce knows if there is already a test like this in the
rt-tests.

> #cyclictest -a 0 -p 99 -m -n -l 100000 -q
>
> I ran this command as shown by Toyoka at the 2014 Linuxcon Japan
> [http://events.linuxfoundation.org/sites/events/files/slides/toyooka_LCJ2014_v10.pdf]
>
> to compare against his results for the BBB.  I also threw in xenomai
> with kernel 3.8 for comparison.  For the standard kernel HR timers were
> disabled.

I believe cyclictest requires HR timers for proper operation.
This may explain the very strange numbers for standard kernel below.

>
> [idle]
> preempt_rt: min 12 avg: 20 max: 59
> standard: min: 8005 avg: 309985,955 max: 619963985
> xenomai: min: 8 avg: 16: max 803
>
> [loaded]
> preempt_rt: min 16 avg: 21 max: 47
> standard: min: 15059 avg: 67769851 max: 135530885
> xenomai: min: 10 avg: 15: max 839
>

Yes, the RT numbers now look reasonable.

The standard kernel numbers are way out.  I can't believe the average
latency on an idle system was 5 minutes. Perhaps the dependency on HR
timers is more than I expect and without it the numbers are just
bonkers. I would have expected the numbers to have a floor near the tick 
rate w/o HR.
Bruce: Is that really what that number means??

The loaded numbers are smaller for RT and std.  Strange.
It might be that the "load" is not very significant.
Its not really the CPU load that were after.  Instead we are trying to 
activate code paths that have premtption disabled due to critical
sections and locks.

I don't know if your are interested in taking this to ground, but if so 
I would enable HR in std and try a load as I suggest above or is
already included in the rt-tests.
Bruce certainly knows more about this than I do and might suggest a
load script.

> Actually the preempt_rt results tie up pretty well with Toyooka above,
> leading me to conclude theres something off in my code that could be
> optimised - what do you guys think.

Is your test code userspace or kernel space?
You can look at cyclictest to see if you missed something.
The RT wiki also has some examples for RT apps.

https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application

> Also, I ran a test with preempt_rt at 100Hz and there was maybe 10%
> improvement in latency.
>
That sounds reasonable to me.


> Steve
>
> On 12/02/2015 00:35, William Mills wrote:
>> + meta-ti
>> Please keep meta-ti in the loop.
>>
>> [Sorry for the shorting.  Thunderbird keep locking up when I tried
>> replay all in plain text to this message.]
>>
>> ~ 15-02-11, Stephen Flowers wrote:
>> > Thanks for your input.  Here are results of 1000 samples over a
>> > 10 second period:
>> >
>> > Interrupt response (microseconds)
>> > standard: min: 81, max:118, average: 84
>> > rt: min: 224, max: 289, average: 231
>> >
>> >Will share the .config later once I get on that machine.
>>
>> Steve I agree the numbers look strange.
>> There may well be something funny for RT going on for BBB.
>> TI is just starting to look into RT for BBB.
>>
>> I would like to see the cyclictest results under heavy system load for
>> standard and RT kernels.  The whole point of RT is to limit the max
>> latency when the system is doing *anything*.
>>
>> I am not surprised that the standard kernel has good latency when idle.
>> As you add load (filessystem is usually a good load) you should see
>> that max goes up a lot.
>>
>> Also, as Bruce says, some degradation of min and average and also
>> general system throughput is expected for RT.  That is the trade-off.
>> I still think the number you are getting for RT seem high but I don't
>> know what your test is doing in detail.  (I did read your explanation.)
>> cyclictest should give us a standard baseline.
>>
>>
>> On 02/11/2015 10:25 AM, Bruce Ashfield wrote:
>>> On 15-02-11 03:50 AM, Stephen Flowers wrote:
>>>>
>>>> my bad, here is the patch set.
>>>> As for load, only system idle load for the results I posted previously.
>>>> Will run some cyclic test next.
>>>
>>> One thing that did jump out was the difference in config_hz, you
>>> are taking a lot more ticks in the preempt-rt configuration. If
>>> you run both at the same hz, or with no_hz enabled, it would be
>>> interesting to see if there's a difference.
>>>
>>> Bruce
>