[agl-discussions] AGL image stability, stress and functionality tests proposal

[Jeremiah Foster]

On Thu, Sep 17, 2015 at 12:05 PM, Bocklage, Jens <Jens_Bocklage at mentor.com>
wrote:

> Ok. Sounds like I can/should start NOW.
>

I strongly recommend you checkout the LTSI test tool. (
http://ltsi.linuxfoundation.org/ltsi-test-project) Its built on Jenkins and
may be a target for GENIVI tests, in which case, you'd get a lot of testing
for free. Also, you should check that your tests are not repeating work
already done. Since AGL is likely going to use the LTSI kernel, the LTSI
test tool will have already run on the kernel AGL is using. Re-doing kernel
stress testing may be a waste of cycles. I really think you ought to be
testing AGL specific code and/or functionality.

Also, you'll likely want to determine what is available already from the
build since if you have a complex set of test dependencies, it may hard for
someone else to repeat unless they do the same build you do. So you'll want
to likely reuse as much from the existing AGL build as possible or turn on
ptest in Yocto or bring in layers, recipes, etc. specific to testing.
Otherwise you're going to end up with an environment that is unique to you
which doesn't really benefit the ecosystem so much.

Cheers,

Jeremiah


> Who can create a tag on the AGL repos? I guess it would be a good idea to
> have Qt5 integrated before tagging.
>
> There is a patch from Tanikawa-san that applies the needed changes. Can we
> somehow get speed into that?
>
>
>
> Jens
>
>
>
> *From:* Streif, Rudolf [mailto:rstreif at jaguarlandrover.com]
> *Sent:* Mittwoch, 16. September 2015 18:07
> *To:* Bocklage, Jens; automotive-discussions at lists.linuxfoundation.org
> *Subject:* Re: [agl-discussions] AGL image stability, stress and
> functionality tests proposal
>
>
>
> Thanks, Jens, for your response and making me aware of that I missed to
> copy the ML on my first response.
>
>
>
>
>
>
>
> A couple of suggestions and rough ideas:
>
>    - I think some of the testing has to be carried out on actual
>    reference hardware, other portions we may be able to do using a virtual
>    machine or emulator. It may be worthwhile adding a field to the test case
>    descriptions indicating the test target options.
>
> My idea is to test it on the porter HW, since this is available for us. Do
> you think we also should test the qemu-version? The “test target options”
> field is a good idea!
>
>
>
> The Porter is a good board to test on. I think we might want to add
> another board, one that is more ubiquitously available. I don't even know
> how to get my hands on a Porter (the board not the beer :). I do have
> Wandboard and MinnowBoard Max though.
>
>
>
> In general it would be good if the AGL distro runs on QEMU too. It is an
> easy way for folks to test it after they built it for the first time. Then
> you can move on to hardware. It's how I do and like it. Some automated
> tests could also be more easily carried out on an emulated system.
>
>
>
>
>    - What are your thoughts on actually performing the tests? Would that
>    be a person or automated tests? I think for most part the latter is
>    probably preferable. Have you looked into automated testing tools?
>
> As a start, the tests are executed manually. Some may be executed via a
> script. But the goal should be to have it automated (maybe as part of the
> CI as a Jenkins-job?). I have not looked inte automated testing tools so
> far.
>
>
>
> I think that's a reasonable start. The test cases to some extend also
> drive the choice of test tools. Scripting is a good thing to verify general
> system setup etc. For UI testing other tools that simulate user interaction
> are necessary. Jenkins CI is good to integrate and orchestrate them.
>
>
>
>
>    - Test report and verification of the results: ideally all expected
>    results can be verified using automated tools and the actual result
>    automatically collected into test reports. Is that in line what you
>    have in mind?
>
> Yes.
>
>
>
> I think we are on the right path.
>
>
>
> Cheers,
>
> Rudi
>
>
>
> _______________________________________________
> automotive-discussions mailing list
> automotive-discussions at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussions
>
>


-- 
Jeremiah C. Foster
GENIVI COMMUNITY MANAGER

Pelagicore AB
Ekelundsgatan 4, 6tr, SE-411 18
Gothenburg, Sweden
M: +46 (0)73 093 0506
jeremiah.foster at pelagicore.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/automotive-discussions/attachments/20150917/ad71d336/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PELAGICORE_RGB_Black_horizontal.png
Type: image/png
Size: 11841 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/automotive-discussions/attachments/20150917/ad71d336/attachment-0001.png>