[Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
Bird, Tim
Tim.Bird at sony.com
Thu Nov 19 17:27:30 UTC 2020
> -----Original Message-----
> From: Srivatsan S <srivatsan.s at pathpartnertech.com>
>
> Tim,
>
> I hadn't tested with the change, but this was a work in progress as it created the job with empty XML config. I had the job_xml stuff in
> progress to pass the configs to Jenkins. Noticed that you've committed this change. Shall test with your latest changes.
Sorry. I didn't realize this was a work in progress. I would have given you feedback and more time
to complete it if I had known.
Please do test the changes I committed. Thanks.
-- Tim
> Thanks,
> Srivatsan
>
> On Thu, Nov 19, 2020 at 2:24 AM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
>
>
>
>
> > -----Original Message-----
> > From: Srivatsan S <srivatsan.s at pathpartnertech.com <mailto:srivatsan.s at pathpartnertech.com> >
> >
> > Tim,
> >
> > While we were looking at eliminating the usage of jenkins-cli.jar in the ftc script, we came across create_job() API in the Jenkins
> Python
> > module which can be used to create jobs in Jenkins, thus eliminating the usage of jenkins-cli.jar in the ftc script.
> >
> > The below patch works for job creation in Jenkins through ftc script. We shall get back to you on ways to programmatically
> update other
> > job properties.
>
> Thanks for sending this. It helped me work the issues with this change.
> There were a few problems with the implementation. Did you test this?
> It doesn’t seem like it would have worked, since it would have created
> jobs with no shell command data.
>
> I ended up using the basic concept, but handling other aspects required for
> this change that you-all missed.
>
> >
> > git diff ftc
> >
> > diff --git a/scripts/ftc b/scripts/ftc
> > index 30195fb..a193c4d 100755
> > --- a/scripts/ftc
> > +++ b/scripts/ftc
> > @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> > job_name = board + "." + test.spec + "." + test.name <http://test.name> <http://test.name>
> > print("Creating job " + job_name)
> > try:
> > - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> > - job_name + ' < ' + tmp, shell=True)
>
> In this case, the tmp file contains the data for config.xml for the new job. The code above this needed
> to be refactored to submit the xml data as a string rather than piping in to jenkins-cli.jar from a file.
>
> > - os.unlink(tmp)
> > + rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> This EMPTY_CONFIG_XML will end up creating a stub job, with no shell command
> to invoke the appropriate Fuego test.
>
> > return rcode
> server.create_job returns "None" on success. The caller can only handle a numeric
> return result (it does a bitwise |, in the case of creating multiple jobs).
>
> > except Exception as e:
> > print("Job already exists")
> > @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan, plan_tests):
> > print("Creating batch job ")
> > try:
> > job_name = board+'.'+testplan+'.batch'
> > - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> > - job_name + '< ' + tmp, shell=True)
> > - os.unlink(tmp)
> > + rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
> Same issues here as above.
>
> > return rcode
> > except Exception as e:
> > print("Job already exists")
> >
> >
> > Thanks,
> > Srivatsan
> >
>
> FYI - Below is the patch I ended up with, that seems to work. I tested it with
> both existing and non-existing regular jobs and batch jobs. I also tested creating
> jobs for multiple boards on the same command line (some already existing
> and some not).
>
> Thanks for the inspiration for the change. I think this is a nice improvement
> to Fuego.
> -- Tim
>
>
> commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
> Author: Tim Bird <tim.bird at sony.com <mailto:tim.bird at sony.com> >
> Date: Wed Nov 18 13:26:39 2020 -0700
>
> ftc: eliminate call to jenkins-cli.jar for creating jobs
>
> Use the python jenkins module to use the Jenkins REST API
> to perform job creation, instead of using jenkins-cli.jar.
> This eliminates all references to jenkins-cli.jar from ftc.
> Theoretically, we could remove the installation of jenkins-cli.jar
> from the Docker container, but let's hold off and make sure it's
> not needed for anything else.
>
> The call to create jobs via the jenkins module is much simpler.
> We don't need to create a temp file to hold the config xml for the
> job. We do have to convert a non-numeric return code into a
> number since callers do bitwise or on it.
>
> Signed-off-by: Tim Bird <tim.bird at sony.com <mailto:tim.bird at sony.com> >
>
> diff --git a/scripts/ftc b/scripts/ftc
> index 30195fb..d6c9033 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -1438,6 +1438,7 @@ def link_key(item):
> return "zzz+"+key
>
>
> +# return 0 on success
> def create_job(conf, board, test):
> flot_link = '<flotile.FlotPublisher plugin="flot at 1.0-SNAPSHOT"/>'
>
> @@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
> for cover, file_ref in link_tuples:
> fail_links += ' ' + template_link % (str(file_ref), str(cover))
>
> - tmp = "/tmp/fuego_tmp_job"
> - fd = open(tmp, "w+")
> - fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> + job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> <project>
> <actions/>
> <description></description>
> @@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
> precleanup=test.precleanup, postcleanup=test.postcleanup,
> testdir=test.name <http://test.name> , testspec=test.spec, timeout=test.timeout,
> - flot_link=flot_link, success_links=success_links, fail_links=fail_links))
> - fd.close()
> + flot_link=flot_link, success_links=success_links,
> + fail_links=fail_links)
>
> job_name = board + "." + test.spec + "." + test.name <http://test.name>
> print("Creating job " + job_name)
> try:
> - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> - job_name + ' < ' + tmp, shell=True)
> - os.unlink(tmp)
> + rcode = server.create_job(job_name, job_xml)
> + if not rcode:
> + return 0
> + else:
> + return 1
> return rcode
> except Exception as e:
> print("Job already exists")
> print(e)
> - sys.exit(1)
> + return 1
>
>
> def create_batch_job(conf, board, testplan, plan_tests):
> - tmp = "/tmp/fuego_tmp_batch"
> job_list = [board+'.'+test.spec+'.'+test.name <http://test.name> for test in plan_tests]
>
> - fd = open(tmp, "w+")
> - fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> + job_xml = """<?xml version='1.0' encoding='UTF-8'?>
> <project>
> <actions/>
> <description></description>
> @@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan, plan_tests):
> </publishers>
> <buildWrappers/>
> </project>
> -""".format(board=board, alljobs=','.join(job_list)))
> -
> - fd.close()
> +""".format(board=board, alljobs=','.join(job_list))
>
> print("Creating batch job ")
> try:
> job_name = board+'.'+testplan+'.batch'
> - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> - job_name + '< ' + tmp, shell=True)
> - os.unlink(tmp)
> - return rcode
> + rcode = server.create_job(job_name, job_xml)
> + if not rcode:
> + return 0
> + else:
> + return 1
> except Exception as e:
> print("Job already exists")
> print(e)
> - sys.exit(1)
> + return 1
>
>
> # parse the testplan and returns a list of test_class instances.
>
>
>
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.
More information about the Fuego
mailing list