[Automated-testing] [RFC] Analysis of the Yocto Test System

[Georgescu, Alexandru C]

Hi Claudius,
I¹ll try to present roughly our current setup for testing the BSP¹s:
- Trigger from LAVA the YP runtime testing for our BSP¹s (e.g. genericx86,
genericx86-64, qemu (all arches)).
- - we didn¹t write the test directly in LAVA since we wanted to test our
runtime testing but needed a place to store the results.

- After the tests are all finished, the logs are parsed and stored into
LAVA.

- also pTest is triggered by LAVA and run on the BSP¹s. We store the
results into LAVA too.


Since we have different type of BSP¹s  (with UEFI and without) we
approached differently the image deployment. For example:
- on UEFI boards we are using the Master Image to control the boot
sequence. UEFI has a flag that can be set to boot into default (Master
Image) or other partition where the tested image was deployed.
- on older boards (with BIOS), we used expect scripts to control the boot
sequence via serial between the Master Image and the image that was being
tested.
This approach (which we are still using) limits us for testing the BSP
deployment to tar.gz artefacts generated by our Autobuilder (in
genericx86¹s case), which was extracted into a separate partition along
with the Master Image. We wanted to mimic the user flow (testing the USB
installation using hddimg img) so we had this crazy idea to create a USB
Switch that dd¹s the hddimg to an USB  stick which installs it like a
normal user, and control the power on the board. We have achieved that,
and now we are on the final stage, testing the solution. This for example
helps us with boards that don¹t have a serial connection.


If you need more info about other parts of the project and how we are
running the tests, I¹ll be happy to keep you informed.

Regards,
--
Alexandru Georgescu
Yocto QA Engineer
SSG/SSD Open Source Technology Center Romania




On 09/02/15 11:48, "Claudius Heine" <ch at denx.de> wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Hi,
>
>in the last week I analyzed Yoctos automated Runtime testing System to
>find out if it could be used within DENX and Simens and, if not, what
>needs to be additionally implemented.
>
>We at DENX currently use DUTS[1] for our testing of u-boot, Linux
>kernel and its drivers. But DUTS is rather old and implemented with
>Tcl/expect scripts and therefore has become problematic to work with.
>
>My investigation of Yoctos Test System revealed that it is currently
>more focused on application and package testing and for example
>doesn't even provide convenient ways to test systems without network
>access (ssh required) or limited storage resources (Master Image
>Deployment).
>
>There also doesn't seems to be a generic method of controlling u-boot
>with its command line interface. That would be useful for us, because
>for many of our test cases a running Linux isn't even necessary.
>
>Also the deployment options are currently rather limited. NFS and TFTP
>would be nice to have.
>
>I uploaded a draft of my report[2] and request you comments, ideas or
>suggestions.
>
>Thanks and best regards,
>Claudius
>
>[1] http://www.denx.de/wiki/DUTS/DUTSDocs
>[2]
>https://owncloud.denx.de/public.php?service=files&t=ed2e2aa62dbc71e11e1da3
>c80fe1a77a
>
>- -- 
>DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch at denx.de
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1
>
>iQIbBAEBAgAGBQJU2IJVAAoJEBFzyxmYCLFTxHQP+LI3VTVJYVBB/G+OvAcXmaui
>kJNoWnhGGpEJDbQqY3m9Dp5olFo31vet7o6MWmzLctq8gDpSeZwvKRM0o+3L9E4E
>uDLmlrdfdrzbCcTtybT/PB9JKSL/K6WKSILJHPnLzxCBmwlo5OnugGeuMeBW5SPF
>ccBbgS1nvVSRPUhJFZPSlpYrgtwhDvkpioXcay6KJd226I4veVwkVQbxaQnD2QoL
>4RTDbB1ZfoM2KiEpHiRVFUoIibMsXYRehuyTnUgTvn5FYREsdh49dozqBob3tcCw
>90ITSTVo1RCai7RXvz0jFWeJcOu4dDt6FAKf0sxdh8UeA7O0EUKH6O83k3gWmzi4
>MODsbI1BxWP61h20o36ipnUn56TXpVkOJscXeKHBBQO+UKq9QkYZC871oWK38PL+
>FnYjP8eVkifQ/sGPTz/dad6l6iTnvkapGUfglI81rjXqIjitXuk7FsiS46lvIqDb
>YvZEkLk2BPXWYqc3O4s5UcXMqT+PvYVOFWqb4WbYif/VXGPFGsSrPl8v0fpMcVSq
>a+Y/O+Uiynu8xKnxmhKhKAvxRhF/oP0FLDUXXXbppJOjXBaLHQK+dIvGvv8+pXIn
>0bIZSpsZquEUMgL3mQWWtnTQLUfK3Sj5xC470YdJQwfEt8UuDIcKSe4UaR1U8x75
>8WGHtQ7P0xinSAtxsYE=
>=1lcd
>-----END PGP SIGNATURE-----
>-- 
>_______________________________________________
>automated-testing mailing list
>automated-testing at yoctoproject.org
>https://lists.yoctoproject.org/listinfo/automated-testing