[yocto] wic- optimizing an image containing a large empty partition

Tue Jan 16 01:22:07 PST 2018

Hi,

I am trying to optimize an image having a fairly large empty partition with regards to file size - wks file:

martin at martin-Precision-5510:~/work/z7000-distro_wic/meta-z7000$ cat scripts/lib/wic/canned-wks/gs-boot-rootfs.wks
part --ondisk mmcblk --size 4
part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot --active --align 4096 --size 64 --fsoptions ro
part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root --align 4096
part /mnt/data --ondisk mmcblk0 --fstype=ext4 --label data --size 1024

Now, generating the map file indicates that bmap-tools is capable of significantly reducing the data transfer:

martin at martin-Precision-5510:~/work/z7000-distro_wic/build/tmp-glibc/deploy/images/zynq-soft-zedboard$ cat  z7000-base-image-zynq-soft-zedboard.wic.bmap
<?xml version="1.0" ?>
<!-- This file contains the block map for an image file, which is basically
     a list of useful (mapped) block numbers in the image file. In other words,
     it lists only those blocks which contain data (boot sector, partition
     table, file-system metadata, files, directories, extents, etc). These
     blocks have to be copied to the target device. The other blocks do not
     contain any useful data and do not have to be copied to the target
     device.

     The block map an optimization which allows to copy or flash the image to
     the image quicker than copying of flashing the entire image. This is
     because with bmap less data is copied: <MappedBlocksCount> blocks instead
     of <BlocksCount> blocks.

     Besides the machine-readable data, this file contains useful commentaries
     which contain human-readable information like image size, percentage of
     mapped data, etc.

     The 'version' attribute is the block map file format version in the
     'major.minor' format. The version major number is increased whenever an
     incompatible block map format change is made. The minor number changes
     in case of minor backward-compatible changes. -->

<bmap version="2.0">
    <!-- Image size in bytes: 1.2 GiB -->
    <ImageSize> 1257452544 </ImageSize>

    <!-- Size of a block in bytes -->
    <BlockSize> 4096 </BlockSize>

    <!-- Count of blocks in the image file -->
    <BlocksCount> 306996 </BlocksCount>

    <!-- Count of mapped blocks: 42.1 MiB or 3.5%    -->
    <MappedBlocksCount> 10765  </MappedBlocksCount>

    <!-- Type of checksum used in this file -->
    <ChecksumType> sha256 </ChecksumType>

    <!-- The checksum of this bmap file. When it is calculated, the value of
         the checksum has be zero (all ASCII "0" symbols).  -->
    <BmapFileChecksum> d41d210e8efdf32597e566360a2f701e706f3b04bcf2418bb0c50a1f8a8d6c72 </BmapFileChecksum>

    <!-- The block map which consists of elements which may either be a
         range of blocks or a single block. The 'chksum' attribute
         (if present) is the checksum of this blocks range. -->
    <BlockMap>
        <Range chksum="4e1c836ffaf1a23f316382f5d7eff44a0fc5a43ac681fa11e100e55b73e8bc7b"> 0-6 </Range>
        <Range chksum="b795c68a5cde6a8ab0b7b541f2094792f4746fb0f9e5b794024895b5c72a42ea"> 2048-2347 </Range>
        <Range chksum="073c303254e96b44400b7df159fdd8cfcc48cad90f77ab14cf157eaaaf94d810"> 23552-23620 </Range>
        <Range chksum="b34201ffa94456444c5bc8e546e3805fc58961403c0a444ce039f21eb894c05c"> 23622-23688 </Range>
        <Range chksum="731c02b87191f90e37683778e6c8817c03cf2b3e6ac1159c1bb7934700709e8b"> 23957-25600 </Range>
        <Range chksum="e1dd56bc7566dda92807ce28582f3f5ead589d7dfbf4beca9c6f928238f73ede"> 25664-29696 </Range>
        <Range chksum="2ccb999e7b63cad77d1891c93424d301cff5dd1c2f25813b96f243d425592770"> 29760-31744 </Range>
        <Range chksum="06986ccd4f1319028200fbfb9dccb72fabd4e18abf4195cfbe288ac09436630c"> 32768-33792 </Range>
        <Range chksum="8cdfec001877d4b6bb41d1caf814aadfc67cabef160a0a442fbb98dfc4190424"> 33856-35389 </Range>
        <Range chksum="f9ca40694ee8b587e20fcfb2a803055f16a6cb1b348a90badae9c13860cff8a0"> 37888 </Range>
        <Range chksum="b05058de59ab0182064323d297a3c6fba42772f3da4f48fa9f60dc58f4b386e3"> 41984 </Range>
        <Range chksum="2abc48348f9551bd05258553014f8c389970d30381e3dc64547a0a7b6f48af14"> 44851-44917 </Range>
        <Range chksum="71899ad8e3fa013443678f384e219f1064e74687fb34861036aa8e4567e12e9a"> 44920-44921 </Range>
        <Range chksum="cc80e24228a48b279bcd626dae16181c317b56acfb8fc56dc7b1f83463966166"> 44923-44925 </Range>
        <Range chksum="3efa6bcb59cd77722327d11d040bf51e983e004c8d826e4ddf7f9b94b0cc754b"> 44932-44933 </Range>
        <Range chksum="3c3ea8e0f3eb5a88321bde760d9328826b1cd87df92260cd3bac65d8bfa094e0"> 53124-53130 </Range>
        <Range chksum="74ad14376b2177e6fea86a2784711a8f8b78beb9766b958271b9fb1bf7cec3a5"> 77619-77621 </Range>
        <Range chksum="1196698d6f64c5e133a6f91030a245f16ee741acbf59e469d23137d502f50e22"> 143155-143157 </Range>
        <Range chksum="d5fda096a9876c5afec2744a95bb4583ce30adf58d64b47f4bf5f69309d4603e"> 175923-175924 </Range>
        <Range chksum="78019a98ed57a2c423406c7e5e97f4615e1ace27be573c0526d0e5b9221b07de"> 208691-208693 </Range>
        <Range chksum="c7e487ba8a927afdc9ea5edd468afc4875cfed37f4587636b1b253a1005b82e1"> 274227-274229 </Range>
        <Range chksum="7784ef4f0c425eb5578559102faaa99c4fba0ab2c2ff7dbe5fcc3c9e731a97a7"> 306992-306995 </Range>
    </BlockMap>
</bmap>

Unfortunately, my HW does not integrate an SD card (that could have been flashed using bmap-tools) and we currently flash the soldered eMMC using dfu-util. In this case all 1.2 GB are transferred, which I consider fairly suboptimal compared to the above reported 42 MB 😞.

In the past we maintained a custom image class that simply skipped creating the empty ext4 partition and left it to be created upon the first boot. I could probably live with this approach, but I don't really see an easy way of achieveing this using wic - perhaps I missed it? However, if possible, my preference would be to generate the partition during the build and not at runtime.

I see some history of sparse images related to wic,  but the works appears reverted?

Thanks,
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20180116/fb930a2e/attachment.html>