[Automated-testing] Automated Testing Summit - Test Stack survey - 0-Day CI

[Philip Li]

On Mon, Oct 08, 2018 at 09:55:13PM +0000, Tim.Bird at sony.com wrote:
> 
> 
> > -----Original Message-----
> > From: Li, Philip 
> > 
> > == Survey Questions ==
> > * What is the name of your test framework?
> > "0-day CI"
> 
> Thanks - I created the following elinux wiki page for your survey response:
> https://elinux.org/0-day_survey_response
> 
> a few comments or questions below:
> 
> ...
> > ==== source code access ====
> > * access source code repositories for the software under test?
> > "yes"
> > * access source code repositories for the test software?
> > "yes"
> > * include the source for the test software?
> > "yes"
> > * provide interfaces for developers to perform code reviews?
> > "no"
> > * detect that the software under test has a new version?
> > "yes, through polling the repo and scanning mailing list regularly"
> > ** if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.)
> 
> Is the mail list scanning software open source? 
no, this part of code is not open sourced yet. We will self written code
to scan mailing list instead of existing ones

> 
> Is it published anywhere?
so it is not published outside

> 
> Do you use existing mail list scanning software (like patchworks or a derivative),
> or is this something unique to 0-day?
this is unique to 0-day

> 
> > * detect that the test software has a new version?
> > "yes, pack the test software regularly"
> I don't understand this.  Do you mean "check the test software regularly"?
yes, check is more meaningful here. We will check the test software and
fetch it to local to be the artifacts used later

> 
> > ==== Test scheduling/management ====
> > Does your test system:
> > * check that dependencies are met before a test is run?
> > "It checks kernel kconfig dependency now"
> 
> I was looking in lkp-tests/jobs, and I saw some need_* declarations
> in some of the yaml files.  
> 
> need_memory
> need_modules
> need_cpu
> need_x
> 
> Do you consider these to be test dependencies also?
thanks for reminder, yes, these are dependencies also

> 
> I remember seeing the kernel kconfig dependency information
> when I looked at 0-day a long time ago, but I can't find it now.
> Can you tell me where in your system this is stored, and maybe
> provide an example of this?
we use need_kconfig to detect whether a kernel can meet the test
requirement, like below. We maintain it in include dir now.

need_kconfig:
- CONFIG_RUNTIME_TESTING_MENU=y
- CONFIG_TEST_FIRMWARE
- CONFIG_TEST_USER_COPY

the full example can refer to https://github.com/intel/lkp-tests/blob/master/include/kernel_selftests

> 
> 
> Thanks very much for filling out the survey.
>  -- Tim
>