[agl-discussions] AGL image stability, stress and functionality tests proposal

Bocklage, Jens Jens_Bocklage at mentor.com
Thu Sep 17 15:08:21 UTC 2015


Hi Jeremiah,

I agree to the most things you mentioned.
I see your point concerning Kernel and applications testing.

Testing things, already tested makes no sense. I’m with you. But I guess we will do changes in the Kernel config, for example. So that a general testing (test depth to be discussed) of the Kernel should be done. The main focus of testing should be AGL functionality. Absolutely. And here, AGL is more specific (GENIVI-worldplay…) than GENIVI.

The AGL repo right now defines images. Those images define a set of functionalities. Therefore the tested environment should be well defined.

Also running automated tests in the CI environment is something that should be realized. But right now the infrastructure is not given.

Long story short:
Good ideas/suggestions that will be taken in consideration as we move on, but today we should start testing the current setup.

Therefore we need someone to tag the repo… 0.2015.38

Jens

From: Jeremiah Foster [mailto:jeremiah.foster at pelagicore.com]
Sent: Donnerstag, 17. September 2015 15:04
To: Bocklage, Jens
Cc: Streif, Rudolf; automotive-discussions at lists.linuxfoundation.org
Subject: Re: [agl-discussions] AGL image stability, stress and functionality tests proposal



On Thu, Sep 17, 2015 at 12:05 PM, Bocklage, Jens <Jens_Bocklage at mentor.com<mailto:Jens_Bocklage at mentor.com>> wrote:
Ok. Sounds like I can/should start NOW.

I strongly recommend you checkout the LTSI test tool. (http://ltsi.linuxfoundation.org/ltsi-test-project) Its built on Jenkins and may be a target for GENIVI tests, in which case, you'd get a lot of testing for free. Also, you should check that your tests are not repeating work already done. Since AGL is likely going to use the LTSI kernel, the LTSI test tool will have already run on the kernel AGL is using. Re-doing kernel stress testing may be a waste of cycles. I really think you ought to be testing AGL specific code and/or functionality.

Also, you'll likely want to determine what is available already from the build since if you have a complex set of test dependencies, it may hard for someone else to repeat unless they do the same build you do. So you'll want to likely reuse as much from the existing AGL build as possible or turn on ptest in Yocto or bring in layers, recipes, etc. specific to testing. Otherwise you're going to end up with an environment that is unique to you which doesn't really benefit the ecosystem so much.

Cheers,

Jeremiah

Who can create a tag on the AGL repos? I guess it would be a good idea to have Qt5 integrated before tagging.
There is a patch from Tanikawa-san that applies the needed changes. Can we somehow get speed into that?

Jens

From: Streif, Rudolf [mailto:rstreif at jaguarlandrover.com<mailto:rstreif at jaguarlandrover.com>]
Sent: Mittwoch, 16. September 2015 18:07
To: Bocklage, Jens; automotive-discussions at lists.linuxfoundation.org<mailto:automotive-discussions at lists.linuxfoundation.org>
Subject: Re: [agl-discussions] AGL image stability, stress and functionality tests proposal

Thanks, Jens, for your response and making me aware of that I missed to copy the ML on my first response.



A couple of suggestions and rough ideas:

  *   I think some of the testing has to be carried out on actual reference hardware, other portions we may be able to do using a virtual machine or emulator. It may be worthwhile adding a field to the test case descriptions indicating the test target options.
My idea is to test it on the porter HW, since this is available for us. Do you think we also should test the qemu-version? The “test target options” field is a good idea!

The Porter is a good board to test on. I think we might want to add another board, one that is more ubiquitously available. I don't even know how to get my hands on a Porter (the board not the beer :). I do have Wandboard and MinnowBoard Max though.

In general it would be good if the AGL distro runs on QEMU too. It is an easy way for folks to test it after they built it for the first time. Then you can move on to hardware. It's how I do and like it. Some automated tests could also be more easily carried out on an emulated system.


  *   What are your thoughts on actually performing the tests? Would that be a person or automated tests? I think for most part the latter is probably preferable. Have you looked into automated testing tools?
As a start, the tests are executed manually. Some may be executed via a script. But the goal should be to have it automated (maybe as part of the CI as a Jenkins-job?). I have not looked inte automated testing tools so far.

I think that's a reasonable start. The test cases to some extend also drive the choice of test tools. Scripting is a good thing to verify general system setup etc. For UI testing other tools that simulate user interaction are necessary. Jenkins CI is good to integrate and orchestrate them.


  *   Test report and verification of the results: ideally all expected results can be verified using automated tools and the actual result automatically collected into test reports. Is that in line what you have in mind?
Yes.

I think we are on the right path.

Cheers,
Rudi


_______________________________________________
automotive-discussions mailing list
automotive-discussions at lists.linuxfoundation.org<mailto:automotive-discussions at lists.linuxfoundation.org>
https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussions



--
Jeremiah C. Foster
GENIVI COMMUNITY MANAGER

Pelagicore AB
Ekelundsgatan 4, 6tr, SE-411 18
Gothenburg, Sweden
M: +46 (0)73 093 0506
jeremiah.foster at pelagicore.com<mailto:jeremiah.foster at pelagicore.com>
[cid:image001.png at 01D0F162.B5CCEEE0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/automotive-discussions/attachments/20150917/9ab797c9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 11841 bytes
Desc: image001.png
URL: <http://lists.linuxfoundation.org/pipermail/automotive-discussions/attachments/20150917/9ab797c9/attachment-0001.png>


More information about the automotive-discussions mailing list