[Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Mon Aug 7 03:05:43 UTC 2017


> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Saturday, August 05, 2017 8:33 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> 
> Daniel,
> 
> I added this patch to my 'next' branch, but on testing it found that it removes a bit too much.
> 
> This sequence shows the problem:
>   ftc add-job -b board1 -t Functional.hello_world
>   ftc add-job -b board2 -t Functional.hello_world -s hello-fail
>   ftc build-job board1.default.Functional.hello_world
>   ftc build-job board2.hello-fail.Functional.hello_world
> At this point there are 2 jobs and 2 log directories
>   ftc rm-job --remove-logs board2.hello-fail.Functional.hello_world
> At this point the directory /fuego-rw/logs/Functional.hello_world is removed
>  (including the board1.default.1.1 log directory)

Oops sorry about that. It should be working with the new patch I just sent.
 
> Also, what happens to the results.json file when only some of the
> logs for a test are removed?  (for example, some for only one spec, or
> some for only one board).  It seems like it would need to be re-written.)

This was harder than what it sounds but it should work with the new patch.
I had to separate the results.json merging function from the common.py
parser so that I could update the results.json without having to setup lots
of environment variables.

Thanks,
Daniel

> 
> Can you please fix this?
> 
> Thanks,
>  -- Tim
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > Sent: Thursday, August 03, 2017 9:00 PM
> > To: Bird, Timothy <Tim.Bird at sony.com>; fuego at lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> >
> >
> >
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > Sent: Friday, August 04, 2017 12:14 PM
> > > To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> > > Subject: RE: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> > >
> > > See comment below.
> > >
> > > > -----Original Message-----
> > > > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > > > bounces at lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > > > Sent: Thursday, August 03, 2017 7:17 PM
> > > > To: fuego at lists.linuxfoundation.org
> > > > Subject: [Fuego] [PATCH 4/4] ftc: add remove-logs flag to rm-jobs
> > > >
> > > > When removing jobs from jenkins this flag additionally allows
> > > > deleting the corresponding log files.
> > > >
> > > > For developers, ftc rm-jobs --remove-logs, will leave Fuego
> > > > in a pristine state without jobs nor old logs.
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
> > > > ---
> > > >  engine/scripts/ftc | 45
> > +++++++++++++++++++++++++++++++++++++++---
> > > > ---
> > > >  1 file changed, 39 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > > > index e885c5f..d6cede5 100755
> > > > --- a/engine/scripts/ftc
> > > > +++ b/engine/scripts/ftc
> > > > @@ -44,7 +44,7 @@
> > > >  import os, sys, string, re
> > > >  import time
> > > >  import commands
> > > > -from shutil import copy2
> > > > +from shutil import copy2, rmtree
> > > >  from lxml import etree
> > > >  import io
> > > >  import subprocess
> > > > @@ -57,6 +57,7 @@ import shutil
> > > >  import tempfile
> > > >  import yaml
> > > >  import jenkins
> > > > +import glob
> > > >
> > > >  # MAJOR, MINOR, REVISION
> > > >  VERSION = (1,2,0)
> > > > @@ -94,15 +95,17 @@ command_help = {
> > > >    This interface may change in the future."""),
> > > >
> > > >  "rm-jobs":("Removes jobs from Jenkins.",
> > > > -    """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
> > > > +    """Usage: ftc rm-jobs [--remove-logs]
> > > > <target>.<testspec>.<testtype>.<testcase>
> > > >    Use list-jobs to see the existing jobs. A wildcard can be used to
> > > >    specify which jobs to remove (just make sure you have 4 words):
> > > >      Example: ftc rm-jobs "docker.testplan_docker.*.*"
> > > >      Example: ftc rm-jobs "docker.*.F*.*stress"
> > > >    Multiple combinations of the
> > <target>.<testspec>.<testtype>.<testcase>
> > > > -  pattern can be passed as well.
> > > > -
> > > > -  If no option is provided all existing jobs will be removed."""),
> > > > +  pattern can be passed as well. The option --remove-logs will
> > additionally
> > > > +  remove the corresponding log files in the log directory. If no job is
> > > > +  provided all existing jobs will be removed.
> > > > +    Example: ftc rm-jobs --remove-logs
> > > > +  """),
> > > >
> > > >  "add-nodes":("Adds new nodes to Jenkins.",
> > > >      """Usage: ftc add-nodes [-f] <board1> <board2> ...
> > > > @@ -1077,8 +1080,25 @@ def do_build_jobs(conf, options):
> > > >
> > > >  def do_rm_jobs(conf, options):
> > > >      count = 0
> > > > +
> > > > +    if '--remove-logs' in options:
> > > > +        remove_logs = True
> > > > +        options.remove('--remove-logs')
> > > > +        raw_input("WARNING: this will remove logs. Press ENTER to
> > > > confirm...")
> > >
> > > I don't think this is needed.  If you really want it, I can leave it,
> > > but I think the user adding '--remove-logs' is specific enough to
> > > not warrant asking for confirmation.  Let me know.  If you're
> > > OK with me taking it out, I'll do so when I add the patch to
> > > my next branch.  If you want to keep it, though, just let
> > > me know.
> >
> > Maybe I was being too conservative. Please remove the line if you want.
> >
> > > > +    else:
> > > > +        remove_logs = False
> > > > +
> > > >      if len(options) == 0:
> > > > -        # FIXTHIS: warn user that all jobs will be removed
> > > > +        if remove_logs:
> > > > +            # remove files and folders in LOGDIR
> > > > +            path = conf.FUEGO_RW + "/logs/*"
> > >
> > > I can't tell, but I assume this is in the 'remove all jobs' path?
> >
> > Yes, that's right.
> > >
> > > > +            files = glob.glob(path)
> > > > +            for f in files:
> > > > +                if os.path.isfile(f):
> > > > +                    os.remove(f)
> > > > +                else:
> > > > +                    shutil.rmtree(f, ignore_errors=True)
> > > > +        # remove jenkins jobs
> > > >          jobs = [job['name'] for job in server.get_jobs()]
> > > >          for job in jobs:
> > > >              server.delete_job(job)
> > > > @@ -1097,6 +1117,19 @@ def do_rm_jobs(conf, options):
> > > >              else:
> > > >                  for match in matches:
> > > >                      server.delete_job(match)
> > > > +                    if remove_logs:
> > > > +                        # remove the per-build_number log folders for the job
> > > > +                        path = conf.FUEGO_RW + '/logs/' +
> > '.'.join(match.split('.')[2:]) +
> > > > '/' + '.'.join(match.split('.')[:2]) + '*'
> > > > +                        folders = glob.glob(path)
> > > > +                        for folder in folders:
> > > > +                            print 'Removing folder ' + folder
> > > > +                            shutil.rmtree(folder, ignore_errors=True)
> > > > +                        # remove the log folder itself in case it became empty
> > > > +                        folders = glob.glob(path)
> > > > +                        if not folders:
> > > > +                            path = conf.FUEGO_RW + '/logs/' +
> > '.'.join(match.split('.')[2:])
> > > > +                            print 'Removing folder ' + path
> > > > +                            shutil.rmtree(path, ignore_errors=True)
> > > >                      count = count + 1
> > > >      if not quiet:
> > > >          print "Deleted %d jobs." % count
> > > > --
> > > > 2.7.4
> > >
> > > Ack.
> > >  -- Tim
> > >
> >
> > Thanks for the quick review of the patches.
> > Daniel
> >
> >
> >





More information about the Fuego mailing list