[Fuego] Eliminating the usage of jenkins-cli.jar in ftc script
Bird, Tim
Tim.Bird at sony.com
Wed Nov 18 20:54:29 UTC 2020
> -----Original Message-----
> From: Srivatsan S <srivatsan.s at pathpartnertech.com>
>
> Tim,
>
> While we were looking at eliminating the usage of jenkins-cli.jar in the ftc script, we came across create_job() API in the Jenkins Python
> module which can be used to create jobs in Jenkins, thus eliminating the usage of jenkins-cli.jar in the ftc script.
>
> The below patch works for job creation in Jenkins through ftc script. We shall get back to you on ways to programmatically update other
> job properties.
Thanks for sending this. It helped me work the issues with this change.
There were a few problems with the implementation. Did you test this?
It doesn’t seem like it would have worked, since it would have created
jobs with no shell command data.
I ended up using the basic concept, but handling other aspects required for
this change that you-all missed.
>
> git diff ftc
>
> diff --git a/scripts/ftc b/scripts/ftc
> index 30195fb..a193c4d 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -1559,9 +1559,7 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
> job_name = board + "." + test.spec + "." + test.name <http://test.name>
> print("Creating job " + job_name)
> try:
> - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> - job_name + ' < ' + tmp, shell=True)
In this case, the tmp file contains the data for config.xml for the new job. The code above this needed
to be refactored to submit the xml data as a string rather than piping in to jenkins-cli.jar from a file.
> - os.unlink(tmp)
> + rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
This EMPTY_CONFIG_XML will end up creating a stub job, with no shell command
to invoke the appropriate Fuego test.
> return rcode
server.create_job returns "None" on success. The caller can only handle a numeric
return result (it does a bitwise |, in the case of creating multiple jobs).
> except Exception as e:
> print("Job already exists")
> @@ -1609,9 +1607,7 @@ def create_batch_job(conf, board, testplan, plan_tests):
> print("Creating batch job ")
> try:
> job_name = board+'.'+testplan+'.batch'
> - rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
> - job_name + '< ' + tmp, shell=True)
> - os.unlink(tmp)
> + rcode = server.create_job(job_name, jenkins.EMPTY_CONFIG_XML)
Same issues here as above.
> return rcode
> except Exception as e:
> print("Job already exists")
>
>
> Thanks,
> Srivatsan
>
FYI - Below is the patch I ended up with, that seems to work. I tested it with
both existing and non-existing regular jobs and batch jobs. I also tested creating
jobs for multiple boards on the same command line (some already existing
and some not).
Thanks for the inspiration for the change. I think this is a nice improvement
to Fuego.
-- Tim
commit b019ac2a2e7d16b03d24083c2fce868d736a61ec
Author: Tim Bird <tim.bird at sony.com>
Date: Wed Nov 18 13:26:39 2020 -0700
ftc: eliminate call to jenkins-cli.jar for creating jobs
Use the python jenkins module to use the Jenkins REST API
to perform job creation, instead of using jenkins-cli.jar.
This eliminates all references to jenkins-cli.jar from ftc.
Theoretically, we could remove the installation of jenkins-cli.jar
from the Docker container, but let's hold off and make sure it's
not needed for anything else.
The call to create jobs via the jenkins module is much simpler.
We don't need to create a temp file to hold the config xml for the
job. We do have to convert a non-numeric return code into a
number since callers do bitwise or on it.
Signed-off-by: Tim Bird <tim.bird at sony.com>
diff --git a/scripts/ftc b/scripts/ftc
index 30195fb..d6c9033 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -1438,6 +1438,7 @@ def link_key(item):
return "zzz+"+key
+# return 0 on success
def create_job(conf, board, test):
flot_link = '<flotile.FlotPublisher plugin="flot at 1.0-SNAPSHOT"/>'
@@ -1505,9 +1506,7 @@ def create_job(conf, board, test):
for cover, file_ref in link_tuples:
fail_links += ' ' + template_link % (str(file_ref), str(cover))
- tmp = "/tmp/fuego_tmp_job"
- fd = open(tmp, "w+")
- fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+ job_xml = """<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description></description>
@@ -1553,28 +1552,28 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
""".format(board=board, reboot=test.reboot, rebuild=test.rebuild,
precleanup=test.precleanup, postcleanup=test.postcleanup,
testdir=test.name, testspec=test.spec, timeout=test.timeout,
- flot_link=flot_link, success_links=success_links, fail_links=fail_links))
- fd.close()
+ flot_link=flot_link, success_links=success_links,
+ fail_links=fail_links)
job_name = board + "." + test.spec + "." + test.name
print("Creating job " + job_name)
try:
- rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
- job_name + ' < ' + tmp, shell=True)
- os.unlink(tmp)
+ rcode = server.create_job(job_name, job_xml)
+ if not rcode:
+ return 0
+ else:
+ return 1
return rcode
except Exception as e:
print("Job already exists")
print(e)
- sys.exit(1)
+ return 1
def create_batch_job(conf, board, testplan, plan_tests):
- tmp = "/tmp/fuego_tmp_batch"
job_list = [board+'.'+test.spec+'.'+test.name for test in plan_tests]
- fd = open(tmp, "w+")
- fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+ job_xml = """<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description></description>
@@ -1602,21 +1601,20 @@ def create_batch_job(conf, board, testplan, plan_tests):
</publishers>
<buildWrappers/>
</project>
-""".format(board=board, alljobs=','.join(job_list)))
-
- fd.close()
+""".format(board=board, alljobs=','.join(job_list))
print("Creating batch job ")
try:
job_name = board+'.'+testplan+'.batch'
- rcode = subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s '+conf.JENKINS_URL+' create-job ' +
- job_name + '< ' + tmp, shell=True)
- os.unlink(tmp)
- return rcode
+ rcode = server.create_job(job_name, job_xml)
+ if not rcode:
+ return 0
+ else:
+ return 1
except Exception as e:
print("Job already exists")
print(e)
- sys.exit(1)
+ return 1
# parse the testplan and returns a list of test_class instances.
More information about the Fuego
mailing list