[Fuego] Using Fuego tests in LAVA

[Liu, Wenlong]

> -----Original Message-----
> From: fuego-bounces at lists.linuxfoundation.org
> [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of
> Tim.Bird at sony.com
> Sent: Thursday, January 10, 2019 8:26 AM
> To: dan.rue at linaro.org; fuego at lists.linuxfoundation.org
> Subject: Re: [Fuego] Using Fuego tests in LAVA
> 
> > -----Original Message-----
> > From: Dan Rue
> >
> > Has anyone tried to use the Fuego test definitions[1] directly in a
> > LAVA job?
> 
> Not to my knowledge, but if someone has I hope they pipe up on the list.
> Different people have tried different approaches to Fuego/LAVA integration,
> but not (I believe) the one you reference.
> 
> > I'm wondering what it would take to make the fuego tests compatible
> > with LAVA test definitions, so that they can be run directly in any
> > LAVA job. In my case, we can assume that the test files already exist
> > on the target.
> This is one of the big assumption differences between LAVA and Fuego.
> 
> There is work afoot to address this, via 2 different methods:
>  1) have Fuego create a package (build artifact) that could be included
> in the LAVA image (at image build time)
>  2) have LAVA support deployment of test materials at test runtime.
> 
> To my knowledge, I'm working on 1) and Fujitsu engineers are working on
> 2) and have some patches they've submitted to LAVA.

Yes, we're still trying to the solution 2), and there is a pending patch which will support tests from tar/url(generated in Fuego side) in LAVA side[1].

As Tim said below,
' take the test_run function from fuego_test.sh and convert it into a 
  standalone bash script (which is what it is)'.
For solutin2), actually we still need this script and another extra yaml file as the test entrance in LAVA side.

[1] https://git.lavasoftware.org/lava/lava/merge_requests/330 

Best regards
Liu

> >
> > I looked at Using Fuego with LAVA[2] and the slides (is the video
> > available?),
> I don't think it is.  I don't think ALS records their sessions.
> 
> > but it seems LAVA was used to boot the board and then Fuego ssh'd in
> > to run tests. It seems that 'Solution I' is what I'm asking about, but
> > I can't tell how tests were actually run.
> 
> Sorry - I didn't finish adding the information about the 2 different
> solutions to the wiki page.  That page is a work in progress.
> But I believe Solution 1 is as you describe.  I believe that is the one
> that AGL has experimented with, and Jan-Simon might have more information.
> 
> > [1]
> https://bitbucket.org/tbird20d/fuego-core/src/master/engine/tests/
> > [2] http://fuegotest.org/wiki/Using_Fuego_with_LAVA
> 
> I'll just start shooting out some ideas and thoughts here.
> 
> The second major impedance mismatch between Fuego and LAVA is Fuego's
> architecture of driving the test from the host.  Many Fuego tests have a
> "run" phase that consist of a single "report" line, which is the Fuego
> function that (from the host) starts a test program on the target, saves
> its output as the test log, and collects the test program return code.  For
> such tests, one could imagine a method of converting these directly into
> LAVA jobs.
> 
> However, other Fuego tests do more processing or interaction on the host.
> For example, Functional.serial_rx sets up the host side of the serial port
> for the test.
> Functional.ospfd modifies the zebra configuration file prior to the test
> (from the host).
> Functional.JAVA runs a sequence of tests as separate java commands, and
> several of the filesystem benchmarks tests (one example is Benchmark.fio)
> do mount point preparation and cleanup from the host.
> 
> 
> With regard to directly executing Fuego tests in LAVA, I think it might
> be possible to do the following:
>  - ignore the build and deploy phases of fuego_test.sh (since your test
> programs and
>   any materials they need are already on the target)
>  - take the test_run function from fuego_test.sh and convert it into a
> standalone
>    bash script (which is what it is), and
>   - run that from your LAVA job.
> 
> You would need a Fuego support library on the target to support the Fuego
> functions called by the instructions in "test_run", but you might get away
> with a significantly reduced subset of the entire Fuego function library.
> Only a few functions are called from 'test_run' in practice.
> 
> One issue that might be a problem is that fuego_test.sh is allowed to use
> bash-isms.  That is, Fuego uses a full-featured shell running on the host.
> Bash might not be available on the target.
> 
> There's also the issue of log parsing, which Fuego does on the host, and
> LAVA most often does on the target (but can do on the host).  Fuego does
> this as a post-processing step, and LAVA (to my knowledge) does this
> synchronously at test program execution time.
> Tim Orling had an interesting session at ELC Europe talking about
> regularizing the output of tests, that, if adopted, would solve a lot of
> problems (for both LAVA and Fuego).
> 
> And finally, there's the issue of dependency checking.  You are certainly
> free to take our needs_check.sh library, and apply those to your systems.
> Or, you do take our test_pre_check functions and execute them on the target,
> as part of the test.
> 
> One other approach would be to do pre-builds of the test programs, and modify
> Fuego to be able to run without it's full docker container or Jenkins (we're
> pretty close to this already).  The dependencies would be python and bash.
> And then just run that as a test execution engine on the target.
> 
> As part of the test definition standards work that I took as an action item
> from the testing summit, I think it would be good to examine the set of
> operations that most tests need to perform, and standardize on them in a
> target-side library.
> Some of the things that Fuego does host-side could be moved to be target-side
> and put into a wrapper script, if such a library existed.  It might be good
> to standardize the name and invocation rules for such a script.  This would
> help both Fuego and LAVA.
> 
> Anyway, that's more a stream-of-consciousness of ideas about possible
> approaches, than a report of success, or a firm direction to follow.
> 
> Regards,
>  -- Tim
> 
> P.S. I'm probably going to put this on the "Using Fuego with LAVA" page,
> just to save the ideas.
> _______________________________________________
> Fuego mailing list
> Fuego at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
>