[Fuego] Fuego's version up and other changes

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Wed Feb 1 00:56:03 UTC 2017


Hi Jan-Simon,

> -----Original Message-----
> From: Jan-Simon Möller [mailto:dl9pf at gmx.de]
...
> Let me add a few notes and observations:
> 
> * I like the create-job / delete-job and the node scripts. They simplify
> 
> things a lot.
> 
> * A extension to this approach IMHO would be the use of jenkins-job-builder.

Wow honestly, I didn't know about that application. It looks quite mature.

> The beauty of this is that we do not have to deal with the xml ourselves.

That's great because I noticed that the XML format changes a bit every time I upgrade.

> Instead we define yaml files and let jjb create the necessary xml.

Cool, I'm ok with YAML but I see it also supports JSON.
http://docs.openstack.org/infra/jenkins-job-builder/

> Another plus is that such an approach would also allow ppl to add
> 
> the testcases to an *existing* jenkins using the jjb templates.
> 
> 
> 
> Let me illustrate this with a jump-start example:
> 
> 
> 
> Installation of jjb (in fuego container):
> 
> $ cd ~/
> 
> $ sudo apt-get install python-pip
> 
> $ sudo pip install jenkins-job-builder
> 
> # should be 1.6.x
> 
> 
> 
> $ cat <<EOF > jjb-conf.ini
> 
> [job_builder]
> 
> ignore_cache=True
> 
> keep_descriptions=False
> 
> include_path=.:scripts:~/git/
> 
> recursive=True
> 
> [jenkins]
> 
> #user=
> 
> #password=
> 
> url=http://localhost:8080/fuego
> 
> ##### This is deprecated, use job_builder section instead
> 
> #ignore_cache=True
> 
> query_plugins_info=False
> 
> EOF
> 
> 
> 
> $ mkdir jjb
> 
> $ cd jjb/
> 
> 
> 
> $ mkdir testjob
> 
> $ cat <<EOF > testjob/testjob.yaml
> 
> - project:
> 
> name: fuego-tests-smoke
> 
> testplan:
> 
> - smoke
> 
> machine:
> 
> - raspberrypi3
> 
> testname:
> 
> - Benchmark.Dhrystone
> 
> - Benchmark.IOzone
> 
> - Functional.hello_world
> 
> - Functional.ipv6connect
> 
> - Functional.stress
> 
> jobs:
> 
> - fuego-{machine}-{testplan}-batch
> 
> - fuego-{machine}-{testplan}-{testname}
> 
> 
> 
> - job-template:
> 
> name: fuego-{machine}-{testplan}-batch
> 
> project-type: freestyle
> 
> concurrent: false
> 
> node: master
> 
> 
> 
> builders:
> 
> - shell: 'echo "Run Orchestrator hook - action to enable board"'
> 
> - shell: 'echo "Next section should also be autogenerated - just dont know the kong-fu, yet."'
> 
> - trigger-builds:
> 
> - project: fuego-{machine}-{testplan}-Benchmark.Dhrystone
> 
> current-parameters: true
> 
> block: true
> 
> - project: fuego-{machine}-{testplan}-Benchmark.IOzone
> 
> current-parameters: true
> 
> block: true
> 
> - project: fuego-{machine}-{testplan}-Functional.hello_world
> 
> current-parameters: true
> 
> block: true
> 
> - project: fuego-{machine}-{testplan}-Functional.ipv6connect
> 
> current-parameters: true
> 
> block: true
> 
> - project: fuego-{machine}-{testplan}-Functional.stress
> 
> current-parameters: true
> 
> block: true
> 
> - conditional-step:
> 
> condition-kind: always
> 
> steps:
> 
> - shell: 'echo "Run orchestrator hook - action do disable board"'
> 
> 
> 
> - job-template:
> 
> name: 'fuego-{machine}-{testplan}-{testname}'
> 
> project-type: freestyle
> 
> concurrent: false
> 
> node: '{machine}'
> 
> builders:
> 
> - shell: |
> 
> export Reboot=false
> 
> export Rebuild=true
> 
> export Target_Cleanup=true
> 
> export TESTDIR={testname} # yep, mixed up testname here with testdir and vice-versa. anyway.
> 
> export TESTNAME=$(echo "{testname}" | sed -e "s#.*\.##")
> 
> export DISTRIB="{{distrib}}"
> 
> export TESTPLAN="{testplan}"
> 
> timeout --signal=9 10m /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
> 
> 
> 
> 
> 
> EOF
> 
> 
> 
> $ jenkins-jobs --conf jjb-conf.ini test -r testjob/
> 
> # to create the jobs within jenkins:
> $ jenkins-jobs --conf jjb-conf.ini update -r testjob/
> 
> 
> 
> 
> 
> The idea would be that fuego-core would provide the scripts and the yaml file in the same git.
> 

Thanks for the detailed example. A few observations:

- Normally the timeout value depends on the board's performance and the test duration. 
In my next branch, timeouts are defined for each test in the testplan_board.json file. 
Here it looks like the timeout needs to be the same for all of the tests. I was wondering
if it is possible to put a list of timeouts (same amount as the number of tests) and then 
use timeout --signal=9 {timeout}. Doing this by hand would be too tedious but we could
generate the yaml files from the testplans using a script.

- What would we do if we want to execute the same test with several test_specs? I was
thinking about creating separate jobs for each combination of test-test_spec because 
it's the simple but it wonder if that can be implemented with jjb. Another approach
would be to create a single job and then have fuego-core take care of reading the
tests specs of each test from the testplan, and repeat the test for each test spec. Then 
a modified parser would display and compare the results of the test across the
different test specs.
[Note] I think that AGL's fuego has a similar parser.

- In my next branch, test specs have the ability to define which links should be available
after the test is complete. I'm using the Description Setter plugin for that. Do you think
that would be possible with the jjb approach?.
 
> While running tests I also found:
> 
> - we need to split the build of the test and the postprocessing to be run separately
> 
> -- this is b/c it is not useful to keep to board waiting on us all the time.

I wasn't aware that the board is waiting on us while we do the post-processing.
Could you elaborate a bit more?
 
> - Integration of board up/down could happen in a few ways:
> 
> -- Either ppl should just generate a wrapper around the (blocking) batch job and trigger it
> 
> with their pre/post to their needs. Done.
> 
> -- Or we allow hooks like the current "TARGET_SETUP_LINK" and amend it with a matching "TARGET_TEARDOWN_LINK"
> 
> - If a model like the "TARGET_SETUP_LINK" is used, we add a delay here as this call must block until the board is up.
> 
> In this case our predifined timeouts are bogus as they track not just the test run, but all processing.

Good point. I wasn't using the reboot feature so I hadn't thought about it.
I guess we would need to define a timeout for the board to reboot, and then the timeout for each test.
The timeout for the board could be called BOARD_REBOOT_TIMEOUT for example and be defined
in the asdf.board file. And the timeout for each test defined in another variable such as TEST_TIMEOUT.

BOARD_REBOOT_TIMEOUT=1m
TEST_TIMEOUT=15m
/bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh

Then, internally the timeouts would be used separately.
# we could also split timeouts between execution time and post-processing time

> Thats my 0,02 - please comment.
> 
> Best,
> 
> --
> 
> --
> 
> Jan-Simon Möller
> 
> dl9pf at gmx.de

Thanks a lot.

Cheers,
Daniel

PS: sorry, my mailer converts html mails into text automatically.




More information about the Fuego mailing list