[poky] Quick hack for profiling tasks

[Richard Purdie]

On Tue, 2011-02-01 at 09:38 -0600, Mark Hatle wrote:
> I'm still used to the RPM world where locales are included in the main package
> and "tagged" so they will or won't be included automatically.  I forget they are
> separated out in Poky by default.
> 
> So skipping them makes sense to me...

Some further numbers to backup my comments:

"time bitbake poky-image-sato -k"

Before patch:

real 129m40.482s
user 480m16.520s
sys 66m9.170s

After patch:

real 117m7.704s
user 455m40.330s
sys 55m1.390s

The patch in question:

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 856858c..fbc1463 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -607,7 +607,7 @@ python package_do_filedeps() {
 
 	# Determine dependencies
 	for pkg in packages.split():
-		if pkg.endswith('-dbg'):
+		if pkg.endswith('-dbg') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1  or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') != -1:
 			continue
 
 		# Process provides


So I think this is a sufficient improvement for us to include it unless
there are any objections from anyone.

> > FWIW, the waitpid above is from the binary locale generation, not from
> > filedeps and we know that is "slow" in the libc case although the cross
> > locale generation speeds it up a lot. The reason process_deps is showing
> > so badly on the profile is not because that python function is slow (as
> > it usually the case) but because its reading from a pipe which blocks on
> > the output of the rpmdeps process.
> 
> Is there a faster mechanism we could use -- or is this actually efficient, but
> not showing it in the trace?

I was thinking this is ok efficiency wise, it just means the function
will look to take a long time on the graphs. The result above surprises
me though as we shrunk the kernel (sys) time by over 10 minutes. This
could mean we're trashing a lot with context switching using this
approach. There is no reason we can't log to a file and then read the
file for processing rather than context switch.

> > I did take a look at the perfile_rpmdeps.sh script and rpmdeps itself. I
> > suspect we might be faster if we called "rpmdeps --requires --provides
> > -v" on each file ourselves and handle the prefix the -v option adds to
> > differentiate between the requires and provides. There is a sed
> > operation in the script I didn't 100% understand that we'd need to
> > account for too. Watching cpu usage whilst manually running the script
> > was not encouraging :/.
> 
> The sed actually accomplished two things.  The first is removes instances of the
> dependency "rpmlib(...)".  This dependency is added by RPM to track specific rpm
> versioning requirements.  Things such as symlink validation, compression
> methods, etc.  None of these are needed within the Poky environment.

Ok, this makes sense. We could always just skip these in the python
processing code.

> The second thing is does is translates the dependency equations into a Poky
> style equation.
> 
> In rpm versioned dependencies are normally like:
> 
> foo = 1.2-3
> 
> However in Poky the format is different:
> 
> foo (= 1.2-3)
> 
> So the sed finds these, and simply moves them into parens.
> 
> We may be able to move the sed out of the direct loop and make it a single call
> over the full run.  I think this will work, and it's likely more efficient as well.

Agreed, either we make it a single call, or we can easily adapt the
python to handle this conversion too.

> As for calling rpmdeps with the -v, we can switch that mode if it makes it
> quicker to process the results, it just becomes slightly more complicated to
> process.

My thoughts here are that we could well half the execution time by doing
everything in one exec call, not two. The additional overhead for
handling the prefix in python is negligible.

>   With the method I'm using now, all of the dependencies return on a
> single line prefixed by the filename.  If we switch to "-v", then we would
> likely need to do "filename:\ndep1\ndep2\n" etc..  I'm sure we could process
> that in python, but the difficulty wasn't worth it at the time.

Fair enough but I think there is something odd going on with that
function we need to get to the bottom of as it does seem to be eating a
large amount of time, possibly when it needn't.

Cheers,

Richard