[yocto] [ptest-runner] Run ptests via stdbuf configured to line-buffering

[richard.purdie at linuxfoundation.org]

On Fri, 2019-04-05 at 06:54 +0000, Mikko.Rapeli at bmw.de wrote:
> On Fri, Apr 05, 2019 at 07:46:00AM +0100, 
> richard.purdie at linuxfoundation.org wrote:
> > On Fri, 2019-04-05 at 06:16 +0000, Mikko.Rapeli at bmw.de wrote:
> > > On Thu, Apr 04, 2019 at 10:48:17PM +0100, Richard Purdie wrote:
> > > > On Thu, 2019-04-04 at 18:00 +0200, Alexander Kanavin wrote:
> > > > > As ptest-runner communicates with child processes via
> > > > > pipe2(),
> > > > > the corresponding channels are not attached to a pty. In that
> > > > > situation stdio facilities like printf() or fwrite() are
> > > > > fully
> > > > > buffered. If a ptest would use them, without bothering
> > > > > to fflush() the output, ptest-runner will only receive what
> > > > > was written by the child ptest process after a buffer gets
> > > > > filled.
> > > > > If the unit tests are proceeding slowly, this may mean that
> > > > > ptest-runner will erroneously timeout due to an apparent lack
> > > > > of
> > > > > 'signs of life' from the child process.
> > > > > 
> > > > > stdbuf utility from coreutils adjusts the buffering to a
> > > > > line-
> > > > > buffered
> > > > > one, and so ptest-runner will get the lines as soon as they
> > > > > are
> > > > > written.
> > > > > 
> > > > > Signed-off-by: Alexander Kanavin <alex.kanavin at gmail.com>
> > > > > ---
> > > > >  utils.c | 7 ++-----
> > > > >  1 file changed, 2 insertions(+), 5 deletions(-)
> > > > 
> > > > I'm a little torn on this. I noticed some of the run-ptest
> > > > scripts
> > > > use
> > > > "| sed -u" whilst the one you were seeing problems with uses "|
> > > > sed"
> > > > without -u.
> > > > 
> > > > We may want to consider strongly recommending -u. I'm testing a
> > > > patch
> > > > with some tweaks like that in it...
> > > 
> > > Please no. I'm running images without sed and using busybox sed
> > > instead, and that
> > > doesn't support -u. I'd rather be compatible with sed from
> > > busybox to
> > > keep changes
> > > to images minimal (e.g. install of additional packages) before
> > > executing ptests.
> > 
> > The other alternative option being proposed is for ptest-runner to
> > depend on coreutils which is worse?
> 
> GNU sed does not come from coreutils but from sed recipe.
> 
> Your call in the end. I just provided my point of view.

The original patch in this thread from Alex needs stdbuf which would
add a coreutils depends to ptest-runner for everything as I understand
it.

> > I did test the -u option to sed in the openssh ptest runner and it
> > did
> > fix the problems we've been seeing.
> > 
> > I'm open to other alternatives but the -u option to sed is looking
> > like
> > the best one we have right now. These bugs are making our testing
> > of
> > ptests effectively useless and unpredictable so this is a serious
> > issue...
> 
> Understood. I hope you could also add 'set -eux' to all ptest shell
> scripts. Many of them seem to be missing shell script error handling
> and failures like providing -u to busybox sed may go unnoticed.

Patches would be welcome to help clean up some of these scripts. I feel
like I'm fighting a lonely battle trying to get any of this to work at
times :(

The good news is the automated monitoring of the ptest results should
catch problems like that as if a chunk of ptests "disappear", the new
testing should quickly highlight that.

(We should improve the scripts, we just also have another way to spot
problems)

Cheers,

Richard