[Fuego] Stuff Tim's been working on

[Daniel Sangorrin]

> Finally, I think it's important to create a test package system.  I would like, eventually,
> for there to be an ecosystem of tests, outside of a single git repository.  Modifying
> the tests so that they do not need to be accompanied by binaries (tarfiles), will be
> important to this.  So I'm interested in the ideas we've discussed about retrieving
> test program source using git (or other fetch mechanisms).  
> 
> I think formalizing the test packages will also help us isolate the tests themselves
> from the framework.  This will help us implement update mechanisms, for when
> fundamental framework changes occur.  I want the system to be modular enough
> to handle changes to different system parts without stuff breaking.  I strongly applaud

I've given it a thought and I'd like to propose the following. Let me know your opinions:

Installing a test package
    - Officially registered test packages
        + Database stored as a single centralized file (testpackagelist.yaml) online:
            -> testpackagelist.yaml
                name: testname
                    description: this is a test for this and that.
                    categories: security, network
                    arch: arm, x86 (supported architectures)
                    stars: ** (number of stars 0-5 depending on the quality of your test package)
                    url: https://bitbucket.org/fuego/fuego-testname.git
                    sha512: safwerwr23fwefwfwsfawfwe...
                name: iozone
                    description: this is iozone for testing disks
                    url: https://www.github.com/fuego/fuego-iozone.git
                    alt-url: ftp://123.123.123.123/fuego-iozone.zip
                    sha512: asfeewrfwfsdfsdfsd...
                ...
        + Workflow
            docker# fuego-update-testpackagelist
                -> downloads the latest testpackagelist.yaml to /etc/fuego/
            docker# fuego-search-test disk
                iozone
                    this is iozone for testing disks
                bonnie++
                    this is bonnie++ for testing disks
                [Note] you can also search by category, supported arch, etc..
            docker# fuego-install-test iozone bonnie++
    - Non-official test packages
        docker# fuego-install-test https://www.gitlab.com/pepe/fuego-supertest.git
        docker# fuego-install-test /path/to/developer/test/folder/

Installing from a testplan
    docker# fuego-install-test -p testplan_raspi.json (install tests in a testplan)
        -> non-official packages: write the url/folder in the testplan

Updating a test package
    docker# fuego-update-tests [testname]
        -> if no name is provided all tests will be updated

Contents of fuego-testname.git:
    - testname.yaml: 
        [Note] used to autogenerate testpackagelist.yaml periodically
        description: this is a test for fuego
        type: Functional | Benchmark
        categories: security, network
        stars: ** (number of stars 0-5 depending on the quality of your test package)
        arch: arm, x86 (supported architectures)
            -> when a job is created (fuego-create-jobs), the board's 
               architecture is matched to the ones supported by the test. 
               If not found, an error is displayed unless -f (force) is provided
               which converts it into a warning.
        host-dep: python-pandas, .. (debian packages to install in docker, apart from default or the toolchain)
            -> a script (install.sh) can be passed instead (e.g.: pip install, yum install ..)
            -> target dependencies are checked by test_pre_check
    - test/ (contents go to fuego-core/engine/tests/Category.testname/)
        testname.sh
            + It should contain the same functions as of now:
                test_pre_check, test_build, test_deploy, test_run, test_processing
            + It should also contain a new one: test_download
                -> downloads stuff needed for the test into buildzone (source code, binaries..)
                    + Executed as jenkins (not as root), so no installing
                -> e.g.: git clone -b next https://www.gitlab.com/ramon/testname.git
                -> e.g.: tar zxvf $TESTDIR/testname.tar.gz (for tarballs)
        other files: parser.py, tarball, .. (optional)
    - testname.spec (optional)
        + Test specification (with default parameters first)
            -> Tests without parameters don't need to provide it
        + An entry will be added to testplan_default by fuego-install-test

> Also, I want to improve
> the support for using test programs already present on target because they are
> already in the target Linux distribution.

This should be easy: add a new variable "deploy" to indicate whether the test_deploy
function needs to be called or not ("rebuild" should be false).

Regards,
Daniel