[Fuego] Integration of Fuego and Linaro test-definitons
Chase Qi
chase.qi at linaro.org
Wed Feb 27 08:16:09 UTC 2019
Hello Tim,
On Wed, Feb 27, 2019 at 2:14 PM <Tim.Bird at sony.com> wrote:
> Sorry for the belated reply, but I was on vacation last week.
>
> I have a few comments I'll put inline below.
>
>
Thanks a lot for the comments. I hope you had a nice vacation.
>
> > -----Original Message-----
> > From: Chase Qi
> >
> > Hi Daniel,
> >
> > Thanks a lot for the comments.
> >
> > On Mon, Feb 25, 2019 at 1:35 PM <daniel.sangorrin at toshiba.co.jp> wrote:
> > >
> > > Hello Chase,
> > >
> > > > From: Chase Qi <chase.qi at linaro.org>
> > > [...]
> > > > > Today, I sent a patch series that allows you to install Fuego
> without
> > Jenkins on Docker. Maybe that will solve
> > > > your previous problems. I also submitted a few more changes to allow
> > users changing the port where Jenkins
> > > > listens.
> > > >
> > > > I noticed the patches. I definitely will give it a spin later on. I
> am
> > > > currently still using fuego v1.40 based docker image for prototyping
> > > > with LAVA multinode job.
> I think that using configuring Fuego support in LAVA as a multinode job is
> a good approach. But I'll comment more at your pro/con list below.
>
> > > > I built and uploaded fuego docker imge with
> > > > fuego test code included here
> > > > https://cloud.docker.com/repository/docker/chaseqi/standalone-
> > fuego/tags
> > > > . BTW, are you guys plan to publish fuego official docker image?
> > >
> > > Nice.
> > > We have talked about that but we haven't published an official image
> yet.
> With the 1.3 release, we almost released an official image. But we ran
> into
> some issues stabilizing it to run in people's environments. There always
> has
> to be a customization step (such as dealing with people's firewalls and
> proxies,
> and matching the user account on the host that is using the docker
> container),
> which we couldn't quite finish the work on for that release.
> As Daniel says, it's on our 'to-do' list.
>
>
Thanks for the info. It is very good to know.
> > >
> >
> > Ok, I will stick with the one I made for the moment.
> >
> > > > Here is the changes I made in the dockerfile.
> > > >
> > > > ```
> > > > $ git diff
> > > > diff --git a/Dockerfile b/Dockerfile
> > > > index 269e1f6..16586fa 100644
> > > > --- a/Dockerfile
> > > > +++ b/Dockerfile
> > > > @@ -114,6 +114,15 @@ RUN CHROME_DRIVER_VERSION=$(curl --silent -
> > -fail \
> > > > RUN echo "jenkins ALL = (root) NOPASSWD:ALL" >> /etc/sudoers
> > > >
> > > >
> > > > +#
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > +#Install fuego
> > > > +#
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > +RUN git clone https://bitbucket.org/tbird20d/fuego.git /fuego \
> > > > + && git clone https://bitbucket.org/tbird20d/fuego-core.git
> /fuego-
> > core \
> > > > + && ln -s /fuego/fuego-ro/ / \
> > > > + && ln -s /fuego/fuego-rw/ /
> > >
> > > The upstream repository has changed to
> > > https://bitbucket.org/fuegotest/fuego.git
> > > https://bitbucket.org/fuegotest/fuego-core.git
> > > Also, if you use the next branch you have to clone fuego-core within
> the
> > fuego folder now.
> > >
> >
> > Thanks for the pointers. I switched to the new links.
> >
> > > > +
> > > > +
> > > > #
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > # get ttc script and helpers
> > > > #
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > @@ -201,8 +210,8 @@ RUN chown -R jenkins:jenkins $JENKINS_HOME/
> > > > # Lava
> > > > #
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > >
> > > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> > > > -RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown
> /usr/local/bin
> > > > +# RUN ln -s /fuego-ro/scripts/fuego-lava-target-setup /usr/local/bin
> > > > +# RUN ln -s /fuego-ro/scripts/fuego-lava-target-teardown
> /usr/local/bin
> > > > # CONVENIENCE HACKS
> > > > # not mounted, yet
> > > > #RUN echo "fuego-create-node --board raspberrypi3" >>
> > /root/firststart.sh
> > > > @@ -218,6 +227,14 @@ RUN ln -s
> > > > /fuego-ro/scripts/fuego-lava-target-teardown /usr/local/bin
> > > > #RUN DEBIAN_FRONTEND=noninteractive apt-get update
> > > > #RUN DEBIAN_FRONTEND=noninteractive apt-get -yV install
> > > > crossbuild-essential-armhf cpp-arm-linux-gnueabihf
> > > > gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf
> > > >
> > > > +#
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > +#Install arm64 toolchain
> > > > +#
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > +RUN /fuego-ro/toolchains/install_cross_toolchain.sh arm64 \
> > > > + && apt-get clean \
> > > > + && rm -rf /tmp/* /var/tmp/*
> > > > +
> > > > +
> > > > #
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > # Setup startup command
> > > > #
> > > >
> > ==========================================================
> > =======
> > > > =============
> > > > ```
> > > >
> > > > >
> > > > > I am also preparing a native install script. Unfortunately, my
> time is up
> > today and I couldn't test it. I send it to
> > > > you attached in case you want to give it a try. But make sure you do
> it on
> > a container or VM where nothing bad
> > > > can happen ;)
> > > >
> > > > Thanks a lot. I just tried the script within lxc jessie container.
> > > > Most of it just works. The following run with ftc cmds are
> > > > problematic, which probably are expected. I guess we also need to
> > > > patch ftc to make jenkins and docker optional.
> > >
> > > Sorry about that, I am working on it. I will release a better script
> soon,
> > maybe today or tommorow. I will let you know when it's ready.
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > > > ```
> > > > root at fuego-native:/fuego/fuego-core/engine# ftc list-boards
> > > > sudo: docker: command not found
> > > > $ diff native-install-jessie.sh native-install-jessie.sh.original
> > > > 86,92c86
> > > > < # ftc list-boards reports
> > > > < # sudo: docker: command not found
> > > > < # TODO: patch ftc to make docker optional.
> > > > < curl -fsSL https://get.docker.com -o get-docker.sh
> > > > < sh get-docker.sh
> > > > <
> > > > < ln -s /fuego/fuego-core/engine/scripts/ftc /usr/local/bin/
> > > > ---
> > > > > ln -s /fuego-core/scripts/ftc /usr/local/bin/
> > > >
> > > > root at fuego-native:/fuego/fuego-core/engine/scripts# git diff
> > > > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > > > index ab2d2cb..1b09812 100755
> > > > --- a/engine/scripts/ftc
> > > > +++ b/engine/scripts/ftc
> > > > @@ -4665,7 +4665,7 @@ def main():
> > > > print "Can't do rm-jobs outside the container!
> Aborting."
> > > > sys.exit(1)
> > > > command += arg + " "
> > > > - container_command(command)
> > > > + #container_command(command)
> > > >
> > > > if len(sys.argv) < 2:
> > > > error_out('Missing command\nUse "ftc help" to get usage
> help.', 1)
> > > > @@ -4781,7 +4781,7 @@ def main():
> > > > # shows fuego boards
> > > > do_list_boards(conf)
> > > >
> > > > - import jenkins
> > > > + #import jenkins
> > > > server = jenkins.Jenkins('http://localhost:8080/fuego')
> > > >
> > > > if command.startswith("add-job"):
> > > >
> > > > # ftc run-test -b raspberrypi3 -t Benchmark.fio -s default
> > > > Traceback (most recent call last):
> > > > File "/usr/local/bin/ftc", line 4929, in <module>
> > > > main()
> > > > File "/usr/local/bin/ftc", line 4785, in main
> > > > server = jenkins.Jenkins('http://localhost:8080/fuego')
> > > > AttributeError: 'NoneType' object has no attribute 'Jenkins
> > > > ```
> > > >
> > > > Thanks,
> > > > Chase
> > > >
> > > > >
> > > > > > > > > > * as you pointed, parsing fuego's test result file in
> LAVA is easy
> > to do.
> > > > > > > > >
> > > > > > > > > The only problem is that I would need to run the Fuego
> parser
> > on the target board.
> > > > > > > > > For that, I would need to modularize the parser into a
> library
> > (e.g. import fuego-parser), and the board
> > > > > > would
> > > > > > > > need to install the python modules required by fuego-parser.
> This
> > is on my TODO list since I proposed
> > > > it
> > > > > > during
> > > > > > > > the last Fuego jamboree. I will try to do it as soon as i
> can.
> > > > > > > > >
> > > > > > > > > What alternatives do I have?
> > > > > > > > > - send the results to LAVA through a REST API instead of
> having
> > it monitor the serial cable? probably
> > > > not
> > > > > > > > possible.
> > > > > > > > > - create a simplified parser on the test (e.g. using our
> > log_compare function). Not ideal, but possible.
> > > > > > > > >
> > > > > > > > > In the end, this stems from the fact that Fuego assumes
> parsing
> > is done in the host (to use python),
> > > > while
> > > > > > > > Linaro uses grep/awk/sed directly on the target. There is a
> trade-
> > off there.
> > > > > > > > >
> > > > > > > > > > * existing way to run fuego tests in LAVA are hacks. The
> > problem is
> > > > > > > > > > they don't scale, 'scale' means remote and distributed CI
> > setup.
> > > > > > > > >
> > > > > > > > > Yes, it is a hack.
> > > > > > > > > I think Fuego is not supposed to run with LAVA, because the
> > goals are very different.
> > > > > > > > > But parts of Fuego can run with LAVA. This is what I think
> we can
> > collaborate on.
> > > > > > > >
> > > > > > > > Yes, +1. When running with LAVA, IMHO, only the backend and
> > real tests
> > > > > > > > are needed.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > * I am tring to hanld both fuego host controller and DUT
> with
> > LAVA.
> > > > > > > > > > The first part is hard part. Still tring to find a way.
> About the
> > host
> > > > > > > > > > controller part, I started with LAVA-lxc protocol, but
> hit some
> > > > > > > > > > jenkins and docker related issues. I feel build, publish
> and pull
> > a
> > > > > > > > > > fuego docker image is the way to go now.
> > > > > > > > >
> > > > > > > > > I think this approach might be too hard.
> > > > > > > >
> > > > > > > > LAVA v2 introduced lxc-protocol. With the protocol, single
> node
> > test
> > > > > > > > job can deploy and boot a lxc container to control DUT. Here
> is an
> > > > > > > > example:
> https://lkft.validation.linaro.org/scheduler/job/605270 .
> > The
> > > > > > > > example job use lxc contianer to deploy imgs to DUT. If DUT
> was
> > > > > > > > configed with static IP, the IP is known to lxc container
> with LAVA
> > > > > > > > helper lava-target-ip, then ssh connection between lxc and
> DUT is
> > > > > > > > possible. Based on these features, I thought we can run fuego
> > tests
> > > > > > > > with LAVA just like how we run it now. As mentioned above,
> > there is no
> > > > > > > > and will be no support for docker-protocol in LAVA, and
> migrating
> > > > > > > > fuego installation to lxc also is problemic. Please do let
> me know
> > > > > > > > once you have a script for fuego installation. I am having
> problem
> > to
> > > > > > > > do that, hit jenkins missing, docker missing, permission
> issues, etc.
> > > > > > > > Once I am alble to install fuego within lxc, I can propare a
> job
> > > > > > > > example. It would be one test definition for all fuego
> tests. This is
> > > > > > > > how we do it before. `automated/linux/workload-automation3
> > > > > > > > ` is a good example.
> > > > > > >
> > > > > > > I see what you want to do. Using LXC sounds doable.
> > > > > > > But I guess that having Fuego installed on the target (or an
> LXC DUT)
> > would be much easier.
> > > > > >
> > > > > > Yeah, I guess if target run Debian based distros, then
> installing on
> > > > > > DUT will be easier. Most of our targets run openembedded based
> > distro,
> > > > > > it is hard to install fuego on them. It is possible to build
> docker
> > > > > > into these OE images and run fuego on target within docker
> > container,
> > > > > > but some of the boards don't have the resource for that... In
> LAVA,
> > > > > > LXC or LXC DUT are mainly used as host to control other ARM based
> > > > > > DUTs.
> > > > >
> > > > > I see, maybe running Fuego on OE might require some work. Maybe it
> > is easier to start with Debian images.
> > > > >
> > > > > > > I am going to work on the installation of Fuego natively then.
> > > > > > > By the way, if you export the docker filesystem (docker
> export..)
> > and import it in LXC you would get a DUT
> > > > with
> > > > > > Fuego installed. Wouldn't that solve your problem? Fuego can run
> > tests on the host (see docker.board)
> > > > although
> > > > > > to run with "root" permissions you need to change jenkins
> > permissions.
> > > > > >
> > > > > > I tried to test docker.board within fuego docker container, it
> works
> > > > > > well, and yes, I hit the root permissions issue. I haven't tried
> to
> > > > > > import fuego docker filesystem in LXC, that is a new concept to
> me.
> > > > > > Does it require docker installed and running in LXC container?
> If yes,
> > > > > > that is a problem in LAVA. I think we will need to modify lxc
> > > > > > cofiguration somehow on lava-dispatcher to support docker in lxc.
> > > > >
> > > > > No, I just meant to use Docker to create the filesystem tree and
> then
> > use it in LXC.
> > > > >
> > > > > > I am getting close to get my whole setup working with LAVA
> > multinode
> > > > > > job. Here is the test definitions in case anyone interested in
> > > > > > https://github.com/chase-qi/test-
> > definitions/tree/fuego/automated/linux/fuego-multinode
> > > > > > . I will share a job example once I have it.
> >
> > Here is a LAVA job example
> > https://lava.slipslow.ml/scheduler/job/119. The test job uses:
> > * docker device as host using the customized fuego docker image, refer
> > the description
> > https://cloud.docker.com/u/chaseqi/repository/docker/chaseqi/standalone
> > -fuego
> > * raspberry pi3 as DUT, the DUT boots OE based build via tftp and
> nfsrootfs.
> > * lava multinode protocol to sync machine status between host and dut,
> > and sent IP and ssh key between them.
>
> Just to clarify, is the Fuego docker container running on the LAVA host,
> but
> treated like another DUT via LAVA multimode mechanisms? Or is it running
> on a separate (3rd) machine?
>
>
Fuego docker container runs on lava-dispatcher. It is a bit confusing. Just
like raspberry pi3, docker is a another device in LAVA. LAVA contains two
major components, lava-master and lava-dispatcher. These two components can
be installed on the same machine, which is called standalone installation.
These two components also can be installed on separate machines, we call it
distributed installation, they are connected using ZMQ. All devices(dev
boards, phones, qem, lxc, docker, ect.) are connected and controlled by
lava-dispatcher. Test jobs are scheduled by lava-master. LAVA treat docker
as a normal device, user can use it as DUT or controller or what they need
it to be. LAVA multinode mechanism is required when on test job needs two
devices. It is the case for fuego. In the example job, I use docker as
fuego host and raspberry pi3 as DUT.
> I prefer this approach to the "run Fuego tests natively on the board"
> approach
> because the latter approach requires a lot of overhead on the target board
> that we were trying to avoid (bash, python, toolchain). The last bit,
> about actually building the software, I think we'll eventually get rid of
> for
> most users, with a server-side cache of pre-built packages. However, I'd
> still like
> to require as little as possible on the board. That's the whole reason
> Fuego
> has the architecture it does - driving test commands from the host - so
> that
> any complex logic or coordination can happen on a more full-featured
> machine
> rather than the target board.
>
> >
> > From my point of view, here are the pros and cons of this approach.
> > Pros:
> > * Doesn't need local fuego host controller any more, it is native run
> > within LAVA.
> > * Easy to scale. Jobs can be submitted from any client to any devices
> > available.
> > * Pretty fast, once the docker image pulled, on the same
> > lava-dispatcher, the following test jobs will use it directly to
> > create container as fuego host.
> >
> > Cons:
> > * Complicated, typically for new LAVA users or whoever don't want to
> > touch LAVA multinode.
> That's interesting. I don't know enough about LAVA to know how big
> this issue is. Do most users not use multinode?
>
>
I am not sure about that. I would say most of our test jobs are single node
job. multinode is not a big issue, but getting familiar with it takes time,
typically for new test writers. Once test definition and job example
developed, it is pretty easy for user to copy or modify.
> One aspect of Fuego architecture that is attractive (IMHO), is that every
> test is implicitly multimode because there is always a host and a board,
> and when the host and the board are directly communicating, and the
> tests is already running on the host, it simplifies
> some of the multi-machine setup. But this is only true when the connection
> between the host and the board has the hardware configuration that applies
> to the test (e.g. they are connected via USB for USB testing, or via
> network
> for network testing, or serial for serial testing, etc.)
>
> My own opinion is that this whole area of managing off-DUT hardware and
> connections is not fully realized in either Fuego or LAVA, and I'm hoping
> to discuss ideas about this at Connect in Bangkok.
>
>
Yeah. Good to known your guys are joining. Looks like we started warming up
already :)
> > * Requires SSH access to the DUT. In the case of network isn't
> > supported, it blocks all tests.
>
> Fuego doesn't require SSH access, though that is the most common
> 'transport' used by most Fuego users. I'm not sure what you mean
> when you say "if the network isn't supported, it blocks all tests".
>
>
I realized fuego supports serial transport, but serial is occupied by LAVA
for devices in LAVA, so I am using ssh transport. I meant when network
isn't supported on DUT, then this approach cannot run any tests.
> >
> > With your effort on native installer and non-jenkins, I think it is
> > possible to do something similar with single node job. As I wrote, it
> > will be lxc protocol plus DUT(with static IP). However, when network
> > isn't available, then we are in trouble too.
> >
> > IMHO, this is hard but a faster way to get fuego tests running within
> > LAVA as it works just like how we use fuego now. I am adding it to
> > Linaro test-definitions with PR
> > https://github.com/Linaro/test-definitions/pull/22 . I believe you
> > guys are the best to review the PR. Any comments would be appreciated.
> >
> > > > >
> > > > > Great! Thanks a lot!
> > > > >
> > > > > Kind regards,
> > > > > Daniel
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Chase
> > > > > >
> > > > > > >
> > > > > > > > Alternatively, I can lunch docker device and DUT with
> multinode
> > job,
> > > > > > > > but that is complex. And fuego docker container eats a lot of
> > > > > > > > memory(blame jenkins?). The exsting docker devices in our lib
> > only
> > > > > > > > have 1G memory configured.
> > > > > > >
> > > > > > > I haven't checked the memory consumed, I guess the reason is
> > Java.
> > > > > > >
> > > > > > > > > This is my current work-in-progress approach:
> > > > > > > > > https://github.com/sangorrin/test-
> > definitions/tree/master/automated/linux/fuego
> > > > > > > > >
> > > > > > > > > - Manual usage (run locally)
> > > > > > > > > $ git clone
> https://github.com/sangorrin/test-definitions
> > > > > > > > > $ cd test-definitions
> > > > > > > > > $ . ./automated/bin/setenv.sh
> > > > > > > > > $ cd automated/linux/fuego/
> > > > > > > > > $ ./fuego.sh -d Functional.hello_world
> > > > > > > > > $ tree output/
> > > > > > > > > output/
> > > > > > > > > ├── build <- equivalent to fuego buildzone
> > > > > > > > > │ ├── hello
> > > > > > > > > │ ├── hello.c
> > > > > > > > > │ ├── Makefile
> > > > > > > > > │ └── README.md
> > > > > > > > > ├── fuego.Functional.hello_world <-
> equivalent to
> > board test folder
> > > > > > > > > │ └── hello
> > > > > > > > > └── logs <- equivalent to logdir
> > > > > > > > > └── testlog.txt
> > > > > > > > > - test-runner usage (run on remote board)
> > > > > > > > > $ cd test-definitions
> > > > > > > > > $ test-runner -g root at 192.168.1.45 -d
> > ./automated/linux/fuego/fuego.yaml -s -o ../output
> > > > > > > > > $ ls ../output
> > > > > > > > > result.csv
> > > > > > > > > result.json
> > > > > > > > >
> > > > > > > > > I have yet to add the LAVA messages and prepare result.txt
> but
> > it will be working soon.
> > > > > > > >
> > > > > > > > You don't have to. It looks like a done job to me.
> send-to-lava.sh
> > > > > > > > will take care of it. When running in LAVA, the helper uses
> > > > > > > > lava-test-case for result collecting, and when running
> without
> > LAVA,
> > > > > > > > the helper prints result lines in a fixed format for result
> parsing
> > > > > > > > within test-runner. (When I writing this, I noticed your
> next reply,
> > > > > > > > maybe I am looking at the latest code already, I will give
> it a spin
> > > > > > > > with LAVA and come back to you)
> > > > > > >
> > > > > > > Thanks again for checking. I am glad that it worked for your.
> I have a
> > LAVA setup on the CIP project so
> > > > I have
> > > > > > started to do tests there.
> > > > > > >
> > > > > > > > So basically, we are running in two different directions.
> From my
> > > > > > > > point of view, you are porting fuego tests to Linaro test-
> > definitions
> > > > > > > > natively. Although I am not yet sure how the integration
> between
> > these
> > > > > > > > two projects goes, we are happy to see this happening :)
> > > > > > >
> > > > > > > Thanks, you are right. But porting it to Fuego misses a lot of
> the
> > good features in Fuego such as the passing
> > > > > > criteria. Perhaps your approach is better.
> >
> > No really : ) As I wrote above, my approach requests ssh access to DUT
> > which isn't always the case.
>
> OK - maybe I understand your statement above better. But I'm not sure.
>
> > It also is one of the reasons we do
> > install(optional) -> run -> parsing on target. lava-dispatcher will
> > clone test repos and apply them to rootfs as overlay before
> > deployment. To some extend, it solves 'no network' issue.
>
> I presume this means that every test in LAVA requires a boot/deploy
> cycle on the target board, which is a downside in that it takes longer
> per test (but an upside in that each test starts with a clean slate).
>
> I think when we have a pre-built test package cache in Fuego, we
> might be able to support this same operational flow.
>
LAVA test job normally run the full 'deploy -> boot -> test' cycle. LAVA
Devices are shared, which means device state(software) is unknown for this
job, the last job may deployed or installed something else. The device
assigned may doesn't boot at all.
> >
> > When dependence can be installed or pre-installed, your approach will
> > work every where. We, at least me, will be very happy to see it in
> > Linaro test-definitions. It is a very good example for adding fuego
> > tests to test-definitions project.
>
> Thanks very much for working on this. We are much farther along
> at integrating Fuego and LAVA than I thought we would be at this point.
> And we are starting to shake out some interesting issues which I think
> will help each project deal with the other's idiosyncrasies and use cases.
>
>
Many thanks to you guys.
- Chase
> Regards,
> -- Tim
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20190227/08ae970a/attachment-0001.html>
More information about the Fuego
mailing list