[yocto] oe-selftest proof of concept

[Stoicescu, CorneliuX]

I made an example of how we can use the functionality described to conduct 3 of our test cases:
https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=90
https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=93
https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=83

As mentioned, the structure is very similar to the automated runtime tests configuration files.
NOTE: ${POKYDIR} does not exist but I use it to refer to the location of the poky directory.

"
import unittest
import os
from oeqa import oeSelfTest, skipModule
from oeqa.utils.decorators import *

def setUpModule():
#             Here we set up the environment for a build directory

class bitbake_layers(oeSelfTest):

                @classmethod
                def setUpClass(self):
#                             Here we set up special settings for the build environment, like shared sstate-cache directory

                def test_show_overlayed(self):
                                oeSelfTest.execute("cp -R ${POKYDIR}/meta/recipes-graphics/clutter ${POKYDIR}/meta/recipes-core/")
                                (status, output) = oeSelfTest.execute("bitbake-layers show_overlayed | grep clutter", build_dir)
                                oeSelfTest.execute("rm -R ${POKYDIR}/meta/recipes-core/clutter")
                                self.assertEqual(status, 0, msg="Could not find overlayed recipe")

                def test_show_appends(self):
                                oeSelfTest.execute("echo "" > ${POKYDIR}/meta-yocto/recipes-core/busybox/linux-yocto_3.10.bbappend"/")
                                (status, output) = oeSelfTest.execute("bitbake_layers show_appends | grep \"/meta-yocto/recipes-core/busybox/linux-yocto_3.10.bbappend\"", build_dir)
                                self.assertEqual(status, 0, msg=" 'bitbake_layers show_appends' Could not find the bbappend file")

                def test_show_layers(self):
                                (ls_status) = oeSelfTest.execute("ls ${POKYDIR} | grep meta-intel")
                                if ls_status == 1:
                                                (git_status) = oeSelfTest.execute("git clone git://git.yoctoproject.org/meta-intel ${POKYDIR}/")
                                oeSelfTest.file.var.append("${BUILDDIR}/conf.bblayers.conf", "BBLAYERS", "${POKYDIR}/meta-intel")
                                (status, output) = oeSelfTest.execute("bitbake-layers show_layers | grep meta-intel", build_dir)
                                self.assertEqual(status, 0, msg=" 'bitbake-layers show_layers ' could not find meta-intel layer in output")

                @classmethod
                def tearDownClass(self):
#                             Even though now it's not needed, here we can reset the git repo in order to revert to default
                                oeSelfTest.execute("git reset --hard")

def tearDownModule():
#             Here we remove the build directory
"

Regards,
Corneliu
Romania Yocto QA

From: yocto-bounces at yoctoproject.org [mailto:yocto-bounces at yoctoproject.org] On Behalf Of Stoicescu, CorneliuX
Sent: Friday, October 18, 2013 5:36 PM
To: yocto
Subject: [yocto] oe-selftest proof of concept

Hello,

After a chat with Richard and Stefan, I came up with an outline of how the oe-selftest feature(https://bugzilla.yoctoproject.org/show_bug.cgi?id=4740 ) should look like. I made a summary of my proposal below. Please feel free to add your thoughts!

There will be a new script introduced(similar to bitbake-selftest) that will use python unit test to execute tests. Name has not been decided but it can be "oe-selftest".
Running this script will not compromise poky in any way.
Initially, the script does not need any preparation in order to be run. If this changes in the future, the user will be prompted upon execution with the pre-required tasks.
Oe-selftest can be used together with the automated runtime tests if necessary.

The following types of tests are targeted for the initial implementation:
- testing the functionality of scripts in poky/scripts (such as: bitbake-layers, yocto-bsp, yocto-kernel, yocto-layers)
- testing of the 'bitbake' command and its output (this includes output data validation such as the sstate-cache/ and tmp/ directories)

In order to achieve the above, the script will need to:

-          manipulate configuration files and variables

-          issue commands in a certain order

-          analyze command output

-          analyze output files and directories

-          manage shared resources such as sstate-cache

Oe-selftest will use Python Unit Testing as follows:
                - we will use what has already been done with the 'unittest' and 'oeqa' classes. This includes 'decorators' and methods like 'skipModules'.
                - the tests configuration files will be similar to those of the automated runtime tests.
                - because we are using the same framework and structure, the output will also be similar to that of the automated runtime tests
                - the build environments used will be set up in the setUpClass() or setUpModule() methods
                                * local.conf may be modified here to use shared resources like sstate-cache and downloads directory with other build environments
                - the build environments used will be removed in the tearDownClass() or tearDownModule() methods
                                * this ensures no residue is left over that we are not aware of
                - we will add a new class oeqa.oeSelfTest that contains the following base methods:

1) A few methods to manipulate variables in poky in a standardized way:
                - oeSelfTest.var_rewrite(filepath_relative_to_poky_location, variable_name, new_variable_value)
                                * resets a variable to the specified value(may be similar to how HOB does to local.conf and bblayers.conf)
                - oeSelfTest.var_append(filepath_relative_to_poky_location, variable_name, appended_variable_value)
                                * appends data to a variable
                - oeSelfTest.var_remove(filepath_relative_to_poky_location, variable_name, remove_variable_value)
                                * removes data from a variable

2) We need to be able to execute tasks in a multiple build directories environment.
                - oeSelfTest.execute(command, build_dir)
                                * this method executes the command 'command' from 'build_dir'
                                * for each execution, the environment is sourced (similar to autobuilder)
                                (if the build directory does not exist, the command will fail. This needs to be taken care of in the setUpClass() method)
                                * it returns the exit code and the output of the command

Additional information/notes:
- class and method naming and structure is not final and open for suggestions
- poky under git is required for the script to work; we will use git to reset modified files to default.
- a policy needs to be set in place(after experience has been gained from using the tool) for working with a shared sstate-cache directory and multiple build environments at a time.
- coding style is very important in order to get a reasonable execution time. At the moment this is not a priority though.
- no strategy has been put in place yet in order to deal with multiple poky environments



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/yocto/attachments/20131018/462edbc9/attachment.html>