[Fuego] RFC: Fuego's version up and other changes
Bird, Timothy
Tim.Bird at am.sony.com
Fri Jan 13 01:37:47 UTC 2017
I just pulled it. Is this right?
if [ "$(id -u)" == "0" ]; then
JENKINS_UID=$(id -u $SUDO_USER)
+ JENKINS_GID=$(id -g $SUDO_USER)
else
JENKINS_UID=$(id -u $USER)
+ JENKINS_GID=$(id -g $SUDO_USER)
fi
I'm not sure exactly what's going on, but shouldn't that second
changed line be?
JENKINS_GID=$(id -g $USER)
-- Tim
> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> Sent: Thursday, January 12, 2017 5:30 PM
> To: Bird, Timothy <Tim.Bird at am.sony.com>;
> Ibe.Kengo at bx.MitsubishiElectric.co.jp
> Cc: fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] RFC: Fuego's version up and other changes
>
> Hi Tim,
>
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at am.sony.com]
> > Sent: Friday, January 13, 2017 10:02 AM
> > To: Daniel Sangorrin; Ibe.Kengo at bx.MitsubishiElectric.co.jp
> > Cc: fuego at lists.linuxfoundation.org
> > Subject: RE: [Fuego] RFC: Fuego's version up and other changes
> >
> > Daniel,
> >
> > Thanks for all this work. I'm having some issues, which may be related to
> the user and group, so
> > I'm replying to the message rather than start a new thread.
> >
> > Basically I'm having two problems, both during ./install.sh.
> >
> > 1) I very often have the debian update and extra file installation fail (Step
> 10 in the Docker image build).
> > (noninteractive apt-get update && apt-get -yV install ...)
> >
> > There are 352 'get' operations during this secondary installation, and if even
> a single one fails,
> > the docker build fails. I kept having problems getting all the way through
> this step. I would
> > have network failures from the debian servers. It seemed to be
> intermittent, but overall it
> > required that I try to build the docker container about 10 times for it to get
> past that step.
> > Here's a sample set of messages:
> >
> > Get:95 http://httpredir.debian.org/debian/ jessie/main libatk-wrapper-
> java all 0.30.5-1 [30.3 kB]
> > Err http://httpredir.debian.org/debian/ jessie/main libavahi-glib1 amd64
> 0.6.31-5
> > Error reading from server. Remote end closed connection [IP: 5.153.231.35
> 80]
>
> Yes, I had the same problem yesterday when I installed it on a different
> machine.
> It seems that httpredir.debian.org is not as reliable as using a local mirror
> directly.
> I'm not sure if I should add retries or put a default debian server as it used to
> be.
> Could you change httpredir by the US mirror
> (http://ftp.us.debian.org/debian) and try again?
>
> > And, at the end of the failed docker build, this:
> > -------
> > Fetched 231 MB in 1min 43s (2226 kB/s)
> > E: Failed to fetch
> http://httpredir.debian.org/debian/pool/main/a/avahi/libavahi-glib1_0.6.31-
> 5_amd64.deb Error reading from
> > server. Remote end closed connection [IP: 5.153.231.35 80]
> >
> > E: Unable to fetch some archives, maybe run apt-get update or try with --
> fix-missing?
> > The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get
> update && apt-get -yV install apt-utils daemon gcc make
> > python-paramiko python-lxml python-simplejson python-matplotlib
> libtool xmlstarlet autoconf automake rsync openjdk-7-jre
> > openjdk-7-jdk iperf netperf netpipe-tcp sshpass wget git diffstat sudo
> net-tools vim curl inotify-tools python-openpyxl g++ bzip2
> > bc libaio-dev gettext pkg-config libglib2.0-dev' returned a non-zero code:
> 100
> > --------
> >
> > It was not the same package failing each time - but the times it failed some
> of the
> > same packages occurred (so, it's hard to pin it on the package or the
> server).
> >
> > Maybe this is just some bad networking at my site, but it did cause me
> problems.
> > (Or maybe the Debian servers are just having a bad day today).
> >
> > More problematical is a hang at Step 21:
> > Removing intermediate container a522905e9fc2
> > Step 21 : RUN groupadd -g ${gid} ${group} && useradd -m -d
> "${JENKINS_HOME}" -u ${uid} -g ${gid} -G sudo -s /bin/bash ${user}
> > ---> Running in 106f8bf82d61
> >
> > The build just hangs there, every time. The dockerd is running on my
> desktop,
> > and consuming 110% cpu.
> >
> > I'm not sure if this is hung in the 'groupadd' step or the 'wget -nv
> ${JENKINS_URL}' step.
> >
> > I'm on Ubuntu 14.04.5 LTS using docker version 1.12.1.
>
> The wget does take a couple of minutes. But I hadn't experienced the
> groupadd hang.
>
> > dockerd appears to be stuck waiting for a futex.
> >
> > I'm not sure that the line 'ARG uid=1000' is going to work in my case. Here's
> my setup:
> > $ id
> > uid=2060932290(1000072457) gid=2060452353(domain_users)
> >
> groups=2060452353(domain_users),4(adm),20(dialout),24(cdrom),27(sudo),
> 46(plugdev),108(lpadmin),124(sambashare),999(docker)
> >
> > Don’t ask - this is due to my Linux machine being previously administered
> by corporate types.
>
> Ok, I think I know what the problem is. I had assumed that the gid would be
> the same as the uid
> in the general case, and other users could just pass their uid/gid in the
> docker-build-image.sh
> script. I have fixed docker-build-image.sh and added the definition of gid.
> Please, could you do a git pull and try again?
>
> https://bitbucket.org/nirrognas/fuego/commits/9d8d1426c326575af8467f73
> bce7773d1f45a76c
>
> > By the way, when I kill the build, dockerd continues to dominate my CPU
> usage in top
> > (and I can hear my desktop cpu fan running).
> >
> > I think this might be a docker bug, but I'm not sure.
>
> Sorry about that!
>
> Thanks for your feedback!!
> Daniel
>
> >
> > I'll play around with the Dockerfile (maybe altering the user configuration),
> but I thought I should report my status so far.
> >
> > Any ideas?
> > -- Tim
> >
> > > -----Original Message-----
> > > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > > bounces at lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > Sent: Wednesday, January 11, 2017 10:45 PM
> > > To: Ibe.Kengo at bx.MitsubishiElectric.co.jp
> > > Cc: fuego at lists.linuxfoundation.org
> > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > >
> > > Hi Kengo-san,
> > >
> > > Thanks a lot for your feedback.
> > >
> > > > -----Original Message-----
> > > > From: Ibe.Kengo at bx.MitsubishiElectric.co.jp
> > > [mailto:Ibe.Kengo at bx.MitsubishiElectric.co.jp]
> > > > Sent: Thursday, January 12, 2017 2:52 PM
> > > > To: Daniel Sangorrin
> > > > Cc: fuego at lists.linuxfoundation.org
> > > > Subject: Re: [Fuego] RFC: Fuego's version up and other changes
> > > >
> > > > Hi, Daniel-san.
> > > >
> > > > I’m very interested in your work!
> > > > So I tried to run your build scripts to create Docker image today,
> > > > then I could run new Fuego.
> > > > And, I added Docker node and test cases to use following command.
> > > > ==
> > > > $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> core.git
> > > >
> > > > $ cd fuego/
> > > > $ ./install.sh
> > > > $ ./fuego-host-scripts/docker-create-container.sh
> > > > $ ./fuego-host-scripts/docker-start-container.sh
> > > > docker# fuego-create-node --board docker
> > > > docker# fuego-create-jobs --board docker --testplan testplan_docker
> > > > --distrib nosyslogd.dist
> > > > ==
> > > >
> > > > I can see new node and new test jobs.
> > > > But, I can’t run tests like Dhrystone on Docker.
> > > >
> > > > So far, I had one small problem that is “Failed to mkdirs: /fuego-rw/
> > > > buildzone”
> > > > This reason is the permission or owner of “fuego-rw”,
> > > > so I added the write permission to “fuego-rw” diredtory.
> > >
> > > Humm You should not need to change any permissions.
> > > The script is supposed to use your user's uid/gid for the jenkins daemon.
> > > However, I have only tested it on my pc which has a single user.
> > > Do you have more than one user? Could you check that the jenkins
> uid/gid
> > > is the same as yours? (you can check on /etc/passwd inside and outside
> > > docker)
> > > Please also check that the fuego-rw's owner is your user (e.g.: don't git
> clone
> > > with
> > > a different user or root)
> > >
> > > > I have solved this problem.
> > > >
> > > > After that, I got the error message like following in console log.
> > > > ====
> > > > Started by user anonymous
> > > > Building remotely on docker in workspace /fuego-rw/buildzone
> > > > [buildzone] $ /bin/sh -xe /tmp/hudson1255554151521556820.sh
> > > > + export Reboot=false
> > > > + Reboot=false
> > > > + export Rebuild=true
> > > > + Rebuild=true
> > > > + export Target_Cleanup=true
> > > > + Target_Cleanup=true
> > > > + export TESTDIR=Benchmark.Dhrystone
> > > > + TESTDIR=Benchmark.Dhrystone
> > > > + export TESTNAME=Dhrystone
> > > > + TESTNAME=Dhrystone
> > > > + export DISTRIB=nosyslogd.dist
> > > > + DISTRIB=nosyslogd.dist
> > > > + export TESTPLAN=testplan_docker
> > > > + TESTPLAN=testplan_docker
> > > > + timeout --signal=9 3m /bin/bash
> > > > /fuego-core/engine/tests/Benchmark.Dhrystone/Dhrystone.sh
> > > > board overlay: boards/docker.board
> > > > using boards/docker.board board overlay
> > > > using nosyslogd.dist overlay
> > > > BATCH_TESTPLAN is not set, using testplan_docker.json testplan
> > > > Traceback (most recent call last):
> > > > File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 615, in
> <module
> > > > >
> > > > run()
> > > > File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 607, in run
> > > > generateProlog(output, ofcls, classes, testPlans, testSpecs)
> > > > File "/fuego-core/engine/scripts/ovgen/ovgen.py", line 375, in
> > > > generateProlog
> > > > outfile = open(outFilePath, "w")
> > > > IOError: [Errno 2] No such file or directory: '/fuego-rw/work/docker_
> > > > prolog.sh'
> > > >
> > > > *** ABORTED ***
> > > >
> > > > Fuego error reason: Error while prolog.sh file generation
> > > >
> > > > Build step 'Execute shell' marked build as failure
> > > > [description-setter] Description set:
> > > > Finished: FAILURE
> > > > ====
> > > >
> > > > Did I miss to run test cases on Docker node?
> > > > If you know this solution, could you give me some advice?
> > >
> > > Could you check that fuego-rw is correctly mounted inside docker?
> > > outside-docker$ touch fuego-rw/hoge
> > > inside-docker$ ls /fuego-rw
> > > hoge <-- does it appear?
> > >
> > > On my PC (ubuntu 16.04, docker version 1.12.3 ) it's always succesful so I
> will
> > > try on a different one. Which OS and docker version are you using?
> > >
> > > Thanks,
> > > Daniel
> > >
> > > > Thank you.
> > > >
> > > > Best Regard.
> > > > Kengo Ibe
> > > >
> > > > ----- Original Message -----
> > > > 件名: [Fuego] RFC: Fuego's version up and other changes
> > > > 送信者: "Daniel Sangorrin" <daniel.sangorrin at toshiba.co.jp>
> > > > 送信日時: 2017年01月11日(水) 09:28:07
> > > > 宛先: "Bird, Timothy" <Tim.Bird at am.sony.com>, fuego at lists.
> > > > linuxfoundation.org
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I've been working on Fuego's upgrade for a few weeks, and in the
> > > > process I
> > > > > have made quite a few changes.
> > > > > You can check it by following these instructions. I'd really
> > > > appreciate
> > > > > your feedback:
> > > > >
> > > > > Quickstart:
> > > > > $ git clone --branch next https://bitbucket.org/nirrognas/fuego.git
> > > > > $ git clone --branch next https://bitbucket.org/nirrognas/fuego-
> core.git
> > > >
> > > > > $ cd fuego/
> > > > > $ ./install.sh
> > > > > $ ./fuego-host-scripts/docker-create-container.sh
> > > > > $ ./fuego-host-scripts/docker-start-container.sh
> > > > > docker# fuego-create-node --board docker
> > > > > docker# fuego-create-jobs --board docker --testplan
> testplan_docker
> > > > > --distrib nosyslogd.dist
> > > > > Then, start the "docker.testplan_docker.batch" job which will
> > > > trigger
> > > > > the rest of jobs.
> > > > >
> > > > > Here is a list of the most fundamental modifications I've added:
> > > > > - I automatized the version upgrade process. So far I have updated
> > > > Jenkins
> > > > > 3 times quite smoothly.
> > > > > The current version is the latest LTS version (2.31.1 LTS)
> > > > > - I reduced plugin dependencies to the minimum (only 2, and they can
> > > > be
> > > > > made optional).
> > > > > - I removed dependencies on all groovy scripts.
> > > > > - Userdata is gone. Instead we now have fuego-ro and fuego-rw,
> which
> > > > > together with fuego-core
> > > > > are mounted as external docker volumes. Only fuego-rw, which
> > > > contains
> > > > > logs and build folders,
> > > > > is mounted in RW mode. I also fixed the uid/gid of jenkins so that
> > > > it
> > > > > matches the one in the host.
> > > > > This means you can develop comfortably on your host, with GUI
> tools,
> > > > > and don't fear a buggy
> > > > > script deleting your folders.
> > > > > - I added a tool for generating jobs from testplans. Creating jobs
> > > > from
> > > > > testplans is powerful.
> > > > > It allows you, for example, to easily specify the timeout for each
> > > > test
> > > > > on your board.
> > > > > The tool also generates the corresponding batch job. For example,
> > > > you
> > > > > can easily trigger all
> > > > > tests for your board and a specific testplan periodically.
> > > > > [Note] This will also enable the creation of scripts for comparing
> > > > test
> > > > > results across
> > > > > different boards and test parameters (on my TODO list).
> > > > >
> > > > > Other less fundamental changes include:
> > > > > - I added support for using the docker container itself as a target
> > > > board
> > > > > (e.g.: for a quickstart or unit tests)
> > > > > - I fixed the flot plugin (java and javascript fixes)
> > > > > - Now you can click on a node and see which jobs (and testplans) are
> > > > > assigned to your board. This was
> > > > > one of my old feature requests and makes Fuego's GUI easier to
> use.
> > > > > - I added quite a few fixes and improvements to the core engine
> > > > scripts
> > > > > and tests (too detailed to describe here).
> > > > > - I added Excel output support for IOzone
> > > > > - Faster docker build time (ARMhf compiler installation is now
> > > > optional,
> > > > > no latex..)
> > > > > - I put a fixed name to the fuego container, instead of the
> > > > > "last_container_id" file.
> > > > > - I removed the inotify script
> > > > > - I added scripts for easily removing the docker container/image.
> > > > >
> > > > > I have to split the current "huge" patch into smaller chunks and test
> > > > it
> > > > > further but it would
> > > > > be nice to get your feedback about the approach in general.
> > > > >
> > > > > Tim: I haven't updated ftc nor your "Christmas commits" yet.
> > > > >
> > > > > Best regards
> > > > > Daniel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Fuego mailing list
> > > > > Fuego at lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> > > > >
> > > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego at lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
>
More information about the Fuego
mailing list