[Fuego] [PATCH] core: Add functionality to make binary program cache

[Tim.Bird at sony.com]

I have worked out a patch, shown below, which makes a set of
tar files which contain the material that would be installed to 
the board, if a test was run.   This is the first phase of making
a test program binary cache.  Note that I'm calling the tarfile which
contains such materials a "board-package".

A second phase would consist of using the cache to avoid having to compile
the programs when running the tests.  This will involve detecting
that the board-package is available, and skipping the build and
deploy phases of a test.  Well, rather, the deploy will be altered
to copy the board-package to the target and untar it.
Note that this system will only work for tests that put all their
deployed materials into $BOARD_TESTDIR (which is almost,
but not all, tests).  If any test doesn't currently conform to this
requirement, it can be altered to stage deployed materials
in $BOARD_TESTDIR, and move files around during the run phase
with a script running in the board.

A third phase is to make it so that end users would not
need an SDK to run Fuego.  To do this we would need to put these
on a public test server (e.g. fuegotest.org).
We might add a command: ftc put-board-package to do this.

Finally, we might use this to make a  package that
could be run more easily in a standalone LAVA environment. 
That would involve writing some wrapper scripts which
implement some of the Fuego functionality in functions.sh,
and putting that and an altered version of prolog.sh into the
board-package.

Following is the patch that makes board-packages for a local system.
Comments and feedback are welcome.
 -- Tim

---------

Add a new special-purpose phase for making tarfiles.
Each tarfile has the files that would have been deployed
to the $BOARD_TESTDIR on the target.

Add a script to make tarfiles for every installed test.
The script is called "make_cache.sh"

Signed-off-by: Tim Bird <tim.bird at sony.com>
---
 engine/scripts/ftc           |  1 +
 engine/scripts/functions.sh  | 14 ++++++++++++
 engine/scripts/main.sh       |  7 ++++++
 engine/scripts/make_cache.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 74 insertions(+)
 create mode 100755 engine/scripts/make_cache.sh

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 682fd81..cc1556d 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -3125,6 +3125,7 @@ def do_run_test(conf, options):
             "c": "pre_check",
             "b": "build",
             "d": "deploy",
+            "m": "maketar",
             "r": "run",
             "t": "post_test",
             "a": "processing" # a is for "analyze" (p already used)
diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index d51340a..41dbb25 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -387,6 +387,14 @@ function post_build {
 
 function deploy {
   pre_deploy
+
+  if [[ "$FUEGO_TEST_PHASES" == *maketar* ]] ; then
+    # force deploy to local area
+    TRANSPORT=local
+    BOARD_TESTDIR=$FUEGO_RW/stage
+    mkdir -p $BOARD_TESTDIR/fuego.$TESTDIR
+  fi
+
   call_if_present test_deploy
   post_deploy
 }
@@ -399,6 +407,12 @@ function post_deploy {
   unlock_build_dir
 }
 
+function maketar {
+  mkdir -p $FUEGO_RW/cache
+  TAR_FILENAME=/fuego-rw/cache/$PLATFORM-$TESTDIR-board-package.tar.gz
+  tar czv -C $BOARD_TESTDIR/fuego.$TESTDIR -f $TAR_FILENAME ./
+}
+
 function firmware {
   ov_get_firmware
   export FWVER="$FW"
diff --git a/engine/scripts/main.sh b/engine/scripts/main.sh
index 7ce08c3..f99b909 100644
--- a/engine/scripts/main.sh
+++ b/engine/scripts/main.sh
@@ -56,12 +56,19 @@ if [[ "$FUEGO_TEST_PHASES" == *build* ]] ; then
     build
 fi
 
+
 if [[ "$FUEGO_TEST_PHASES" == *deploy* ]] ; then
     FUEGO_CUR_PHASE=deploy
     show_phase
     deploy
 fi
 
+if [[ "$FUEGO_TEST_PHASES" == *maketar* ]] ; then
+    FUEGO_CUR_PHASE=maketar
+    show_phase
+    maketar
+fi
+
 if [[ "$FUEGO_TEST_PHASES" == *run* ]] ; then
     FUEGO_CUR_PHASE=run
     show_phase
diff --git a/engine/scripts/make_cache.sh b/engine/scripts/make_cache.sh
new file mode 100755
index 0000000..7d2a313
--- /dev/null
+++ b/engine/scripts/make_cache.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+#
+# make_cache - make a binary test program cache (set of tar files) for
+# the indicated board.  This will build a cache of all tests.
+#
+# Usage: make_cache <board>
+
+usage() {
+    echo "Usage: make_cache [<options>] <board>"
+    echo
+    echo "-h = show this usage help"
+    exit 1
+}
+
+if [ -z "$1" ] ; then
+    echo "Error: Missing arguments"
+    usage
+fi
+
+# parse command line options
+while [ $(echo -- $1 | cut -b 4) = "-" ] ; do
+    case $1 in
+        -h|--help) usage;;
+        --) shift; break;;
+        -*) echo "Invalid option: $1" ; usage ;;
+    esac
+done
+
+# get board
+board=$1
+if [ -z "$board" ] ; then
+    echo "Error: Missing board"
+    usage
+fi
+
+if [ -z "$FUEGO_RW" ] ; then
+    export FUEGO_RW=/fuego-rw
+fi
+
+mkdir -p $FUEGO_RW/cache
+chown jenkins.jenkins $FUEGO_RW/cache
+CACHE_LOG=$FUEGO_RW/cache/cache.log
+
+#
+# FIXTHIS - need to mark this as NO_BOARD_CONTACT
+tests=$(ftc -q list-tests)
+echo "=============================================" | tee -a $CACHE_LOG
+date | tee -a $CACHE_LOG
+for test in $tests ; do
+    echo "Making cache package for $test" | tee -a $CACHE_LOG
+    ftc run-test -b $board -t $test -p "pbdm" >> $CACHE_LOG
+done
-- 
2.1.4