[yocto] Need for offline binary configuration

[Mark Hatle]

My apologies for the late response.  See below, I'll follow up on your second 
email as well...

On 11/20/12 9:09 AM, Venkata ramana gollamudi wrote:
> Poky allows to build custom Linux for you, but we have cases where the post
> build customization is required, like user-addition, network configuration,
> service control. Even selecting the required packages can be a post build activity.

One of the objectives that we are currently working toward is being able to 
start with the package feed (doesn't matter if it's rpm, deb or ipk files), 
generate a rootfs, configure the rootfs and then deploy the rootfs into an image.

The above should be able to be done on both a cross-configured environment or on 
the target via a traditional installer like mechanism.

> The current model requires the image to be rebuilt to support these configuration.

The current model can avoid many of the tasks associated with the build by using 
sstate-cache, but I agree that is not optimal for users who want to start with a 
known set of binaries and then tailor from that point.  (This includes users 
expecting a more traditional Linux install experience or users who need 
immutable software from an upstream vendor.)

> Offline Configuration tool (OCT), which allows a binary image customization
> before making a final target image. This case will be more evident in larger
> companies, where platform teams, product teams , application teams are
> distributed and Linux build from source will be owned and lab tested by a single
> team, like platform team. Other teams just configure to use it for product
> variants from same platform build.

This was factored into the work we are doing.  I know many companies have an 
outside OSV that provides the build environment, support, etc.. and they feed 
into an Operating Systems group (or similar).  This group then prepares the 
software according to the corporate guidelines (license restrictions, quality 
assurance, etc) and repackages it for consumption by others in the organization. 
  Often time the output of the Operating System group is either a blessed source 
based system -- or a set of binaries that are blessed as being in conformance 
with that companies guidelines.  This process repeats through the platform group 
and even individual projects -- with the eventual output being a product image 
including the blessed binaries, custom applications and configurations -- and a 
matching SDK.

> Detailed use cases can be found in enhancement bug:3252
>
> OCT should work on the binary pool of compiled packages generated from poky.
>
> The basic operations that can be supported includes

For the next section, I'm going to assume the pool of binary packages is what we 
refer to as the package feeds in OpenEmbedded.  This is simple a collection of 
binary packages that have been built from the build system that can be used to 
construct the rootfs.  Each package feed is constructed based on a single 
"distribution" configuration file which sets the policies for the packages in 
the feed and distribution wide options, such as selinux, pam, etc.

> a) Select/deselect required packages from pool of binary packages into final target image.

The above is definitely considered in the work we've been talking about.  This 
will require software that will take a list of the available items in the feed, 
present the user/developer with a way to enable the components they want on the 
target.  (Note, there is an inherit requirement here that dependencies also be 
met, so things the user/developer may not have selected will either have to be 
installed to meet dependencies or will have to generate an error that 
dependencies can't be met.)

> b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image.

As long as the external package is in the same format as the rest of the feeds, 
this should be easy to do.  And I certainly expect that it will be needed.

> c) Binary level Offline configuration can includes
>        Configure the users/passwords
>        Configure the network
>        Configure the host name
>        Select the services to be started by default

The above are things we've talked about, but I don't believe there have been any 
formal plans to implement them at this time.  The components necessary for the 
binary package based install -> rootfs generation have taken up most of our 
planning time at this point.

One thing I would love to see is a set of standard components (or a framework) 
that could be used for this type of configuration.  But I believe it's very 
important that the configuration work both within the OpenEmbedded environment 
and externally in the "installer" like environment.

>        Security related configuration

This is something that I don't believe has really been addressed at all, but 
like the above we need a way to implement it.

>        Generate initrd in ramfs/ext3/... format
>        etc..

We are already considering how to go from rootfs to deployable image.  initrd 
just happens to be one of many potential image types.  We've specified hard disk 
image (partition map, bootloader, individual partitions that comprise one or 
more rootfs).  A flash based image -- instruction for flashing, and individual 
flash 'partitions' that will work similarly to the hard drive.  A CD/DVD/USB 
based bootable system.  And finally as a component of these an initrd rootfs 
that can be used as a component of the others are separately to instantiate the 
system before handing off control to one of the other types.

We have a huge advantage in OE that we can generate multiple rootfses from a 
given distribution configuration, we just don't currently have the tooling to 
put those rootfses together into the various images.  I know Darren Hart at 
Intel is working on some of the ground work for the hard disk style images.

> Considering the methods to support these in our current yocto model, following changes can be done.

The first thing we would like to see is a -single- rootfs creation model for all 
of the modes.  Build system, cross-install, and target-based-install.  The way 
that I'm approaching this right now is by attempting to abstract out just the 
rootfs generation capabilities within the current build system.  The intention 
is that you will be able to access interfaces to get a list of whats available 
to be installed, (what may be already installed,) a way to pass what to install 
and where to the environment.  Then the same code will be executed to perform 
these actions.  The actions will be designed to be package type agnostic as well.

Early R&D work is going on there, just to see if I can come up with something 
reasonable.

> 1) HOB can be the tool which can be extended to support these
>      Poky can generate a binary package pool as one if its output and Hob can work on this package pool to select packages, configure and generate image.
> So HOB can support opening HOB in Binary(or OCT) mode i.e., without build options but only with binary package selection. Configuration GUI needs to be added to HOB.
> Note:HOB+OCT is together or separate, needs a bit more thought and overall organization as they will be intended for different users.

I'm not sure HOB itself is the right tool for this, but there are certainly 
facets of HOB that would be needed in this kind of tool.  The package selection, 
etc.

> 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure.
>      I think it is sufficient to keep the minimal binaries from sstate-cache which are required to execute image.bbclass, do_rootfs task to generate image.

There are multiple steps of binary package pool, each I think are valid.  For 
someone wanting a source based system, with prebuilt binaries that can be 
locally modified -- sstate-cache is by far the best way to do this.  For an 
applications group that needs to generate images and SDKs, but is never allowed 
to use anything but the corporate blessed binary packages, a different mechanism 
(package based) is likely what is needed.  There is a lot of area though where 
either method may be needed (or desired).

My experience shows people familiar with traditional workstation based systems 
prefer the binary RPM approach, while hard-core embedded developers prefer 
source based systems.  The reality is the world has fewer hard-code embedded 
developers and more folks used to the traditional environments, add to that the 
capabilities of today's embedded systems and it makes sense for a lot of people 
to start with blessed binaries, tailored for embedded systems, and simple add 
their integration of applications and configurations after.

> 3) Along with specific configuration UI implementation, a generic configuration model similar to kernel kconfig and menuconfig can be considered, in cases where more detailed offline configurations is required like detailed security configuration.

This is absolutely need, but I think we need to get the general rootfs 
generation working and figured out before we spend too much time on this. 
However, if this is a high priority for you, it may make sense to start working 
on configuration.  This is something that is definitely needed, and I don't 
believe anyone is actively working on.

--Mark

> Regards,
> Ramana
> _______________________________________________
> yocto mailing list
> yocto at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>