[agl-discussions] [RFC] CIAT Continuous Integration and Automated Test

[Paul Sherwood]

Hi folks,
GENIVI Tools team has undertaken to work towards establishing an open 
infrastructure to provide build and some automated testing of GENIVI 
outputs (including ongoing versions of individual GENIVI projects, 
GENIVI Baselines and GENIVI Demo Platforms). We hope that some or 
possibly all of this work may be adopted by AGL too.

Below is a first draft "Statement of Work' for this - we would 
appreciate any feedback and suggestions to improve it.

br
Paul

SoW: Continuous Integration and Automated Test
==============================================

Background
----------
Various open source 'umbrella' projects including GENIVI and AGL have a 
need to establish and maintain public-facing infrastructure for 
Continuous Integration and Automated Testing of integrated sets of open 
source software.

This aims to document generalised requirements for CIAT in a 'Statement 
of Work' form, so that actual implementations can be planned and 
delivered. It is hoped that a workable solution can be found which 
satisfies requirements for all of these organisations, but it is 
expected that each may have further specific requirements, and may for 
its own reasons choose alternative approaches for some or all of the 
parts of this infrastructure to satisfy organisation-specific goals.

The content here is to be offered to all of the mentioned organisations 
(and more widely) for consideration. It is recognised that maintaining a 
CIAT infrastructure can be of significant engineering value, but 
involves ongoing work and cost. Therefore collaboration on a generally 
useful approach can offer particular benefits both in terms reduced cost 
per project, and increased quality through (for example) 
cross-validation and re-use of tests.

Scope
-----
CIAT includes

* collection of source code from upstreams including new versions as 
required on an ongoing basis
* automated instructions for building the upstream components including 
their dependencies
* automated instructions for installing the built upstream components 
into an integrated 'distro'
* Continuous Integration pipeline which executes the automated 
instructions automatically, either
   * on user demand, or
   * triggered by changes to the instructions or components themselves
* ability to include binary input artifacts (eg drivers or proprietary 
executables) into the CI pipeline
* publication of the built/integrated component and distro artefacts 
generated by the CI pipeline
* Automated Test pipeline which executes sets of tests on deployed 
instances of the integrated software
* publication of the results of the automated tests, including at least
   * pass/fail
   * complete log files
   * built artifacts
* demonstration of license compliance, including
   * ability to add licence validation to the pipeline
   * ensuring availability of source code for all published artefacts
* machanism for formal code review prior to merging of changes, either
   * before build and test or
   * after build and test as candidate (preferred)

Requirements
------------
* all of the CIAT software tooling must be FOSS
* any custom work to make the solution fit specific use-cases for 
GENIVI and/or AGL must be FOSS
* where possible the solution should align with the upstream developers 
for its components
* instructions for creation and setup of a working instance of CIAT 
infrastructure must be provided
* minimal manual maintenance - the system should be mostly automated:
   * minimal effort for license compliance (eg scripted license 
checking, publish applicable source for all artefacts)
   * developer/integrator engineers can trigger build/test using git 
push.
* CIAT must support
   * multiple automated test frameworks (eg FitNesse, Phoronix, LTSI 
Test Project)
   * testing scenarios on multiple architectures (eg ARMv7, x86_64, 
MIPS) for both virtual machines (eg QEMU) and actual hardware
   * multiple build systems (eg autotools, cmake, qmake, bitbake)

Deliverables
------------
* A working implementation of the CIAT infrastructure
* source code for any and all custom software required to achieve the 
CIAT solution, published in Git at an established Git hosting service
* Instructions for creation of further instances of the CIAT