[yocto] [RFC] possible solutions to bug 2421

[Richard Purdie]

On Fri, 2014-04-11 at 17:21 +0000, Eliaser, MaX wrote:
> I've been working on bug 2421. [1] The bug report notes that several -native
> and -cross packages include man pages, info pages, and other documentation.
> This is unnecessary, and build times and disk usage could be improved by
> eliminating these files.
> 
> However, with the exception of Perl's Module::Build, [2] the build systems we
> support have no built-in way to do this, and most (all?) upstream projects
> don't implement it manually in their build system configurations. Unless I'm
> missing something easy and obvious, which I don't discount at all.
> 
> There are actually two different unnecessary things going on. First is the
> generation of the man/info pages, HTML/PDF files, etc. from source markup.
> This is done using programs like pod2man, asciidoc, groff, nroff, etc. Second
> is the copying of the files to their destination, usually in /usr/share
> somewhere. There are a few different ways I can think of to attack one or both
> of these problems:
>  *  Add a --disable-docs feature to each applicable upstream project's build
>     setup. Keep in mind that this is at *least* 58 separate autotools
>     configurations to add this feature to (specifically just the ones used
>     by -native and -cross recipes.) Plus a bunch of CMake ones, plus probably
>     a few others. But it would allow you to prevent documentation files from
>     being generated *and* copied.

Doing it per recipe is going to be time consuming and prone to breakage
when we upgrade so whilst a nice ideal option, we probably need to find
something else.

There are probably some things with doc flags (gnome software using
gnome-doc-stub for example?). Where there are doc flags, adding a
PACKAGECONFIG[doc] would be nice but the core set of native recipes
aren't likely to have flags, agreed.

>  *  Come up with some patches for each project that will simply nuke the doc
>     building altogether instead of making it optional. These patches obviously
>     could not be upstreamed, but they'd be easier to write.
>  *  Add code to the recipes to re-clear /usr/share/man, /usr/share/info, etc.
>     after each install has finished. That saves disk space but actually *adds*
>     disk activity and build time.

FWIW we already do this albeit at do_populate_sysroot time. See
sysroot_stage_dir() in staging.bbclass.

> I'd say that those three options are all legitimate, solid solutions. There 
> are also some ways to cheat:
>  *  Attack the documentation formatting utilities used by the various build
>     systems. There are a finite number of programs that get used for this--
>     pod2man, asciidoc, groff, etc. Why not replace these with a simple program
>     that does nothing, or just emits a blank file? We could conditionally set
>     the PATH environment variable to use or not use these dummy programs. The
>     file copy still happens, but at least it's a blank file.

I like this idea. I would draw your attention to the
scripts/native-intercept directory and:

classes/native.bbclass:PATH_prepend = "${COREBASE}/scripts/native-intercept:"

since we already "intercept" chown for different reasons.

This approach has the advantage that its relatively straightforward to
attempt, maintains relatively easily and should let us see whether this
kind of optimisation is worth much or not.

>  *  Attack the "install" program, wrapping it in a shell script that checks
>     if the install target directory is in some blacklist (i.e. /usr/share/man, 
>     /usr/share/info) and if not, invoking the original "install" binary. That
>     saves a file copy.
>  *  Mark the bug as "won't fix."
> 
> There are probably a few other solutions I haven't thought of yet.

I think its worth seeing how much of a difference some simple intercept
scripts can have.

When looking at this kind of thing, also keep an open mind to other ways
we can speed up the system. We're always looking for new ideas :).

Cheers,

Richard