[Fuego] ftc add-nodes fails
Bird, Tim
Tim.Bird at sony.com
Thu Jul 9 03:34:46 UTC 2020
This has been applied in the master branch.
Thanks for the fix!
-- Tim
> -----Original Message-----
> From: Laszlo Sitzer <laszlo.sitzer at native-instruments.de>
>
> That is so good to hear you could reproduce! Here is my patch (see attachment).
>
>
> On Fri, Jul 3, 2020 at 2:06 AM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
>
>
> Thanks. I was able to reproduce the bug - actually both bugs.
> I tried your fix for the plugin version issue, and it works. I also tried a fix
> for the Unicode exception thrown by str(e) on an exception. I got ftc to not
> throw the Unicode exception, but I'm still researching whether to use my fix
> as the long-term solution.
>
> In any event, I think I'd like to incorporate your fix, and I plan to put it in the
> master branch. I'd like to give you credit. Can you possibly send me (and CC: the list) a patch?
> (with your own Signed-off-by line)?
>
> Thanks.
> -- Tim
>
>
>
> > -----Original Message-----
> > From: Fuego <fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > On Behalf Of Laszlo
> Sitzer
> > Sent: Thursday, July 2, 2020 3:05 AM
> > To: fuego at lists.linuxfoundation.org <mailto:fuego at lists.linuxfoundation.org>
> > Subject: Re: [Fuego] ftc add-nodes fails
> >
> > Welcome!
> >
> > If you find the time could you try if you could reproduce the issue I was having?
> > Running "./install.sh && ./start.sh" (without a proxy) and then "ftc add-nodes -b bbb" in the container should trigger it.
> > I'd really like to see I am not the only one having this problem.
> >
> > Maybe then the fix could also go onto the next or even master branch.
> >
> > Regarding what you said about Jenkins ... I hear you! I actually started looking into updating Jenkins
> > and its plugins but I think I need to understand Fuego and its use of the plugins better.
> > But in the long run maintaining a stable Jenkins is a job of its own. Which makes me think about ways to
> > have a clean "seam" between Fuego and Jenkins. Maybe by having a scheduling/test execution API
> > that is implemented in ftc and in another daemon that would then talk to Jenkins or other CI systems.
> >
> > But for now ... I think I have to go back to actually using Fuego before I start thinking about how to refactor it.
> >
> > Best,
> >
> > Lazlo
> >
> > On Wed, Jul 1, 2020 at 7:25 PM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com
> <mailto:Tim.Bird at sony.com> > > wrote:
> >
> >
> > Wow - this is super helpful.
> >
> > The version of Jenkins we are currently using in Fuego has a number of issues with plugins
> > (which have gotten stale and out-of-date). We tried to address that by using a fixed list
> > of plugins and versions, but the dependencies eventually catch up with you and things
> > start to break. (See the Dockerfile - the section where we use install-plugins.sh)
> >
> > I had already started working on upgrading to a more recent version of Jenkins,
> > to address this problem, but ran into some compatibility issues that needed to
> > be addressed. It looks like I should focus back on that work. I think upgrading
> > the Jenkins version might help with a lot of these plugin problems.
> >
> > Thanks.
> > -- Tim
> >
> >
> > > -----Original Message-----
> > > From: Fuego <fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> <mailto:fuego-
> bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > > On Behalf Of
> > Laszlo Sitzer
> > > Sent: Wednesday, July 1, 2020 10:46 AM
> > > Cc: fuego at lists.linuxfoundation.org <mailto:fuego at lists.linuxfoundation.org> <mailto:fuego at lists.linuxfoundation.org
> <mailto:fuego at lists.linuxfoundation.org> >
> > > Subject: Re: [Fuego] ftc add-nodes fails
> > >
> > > By now I found that if I use a different "launcher" I can create an executor.
> > >
> > > ```
> > > import jenkins
> > > s = jenkins.Jenkins("http://localhost:8090/fuego")
> > > s.get_nodes()
> > > s.create_node('bbb', launcher='hudson.slaves.JNLPLauncher')
> > > ```
> > >
> > > This in turn makes me think I should look deeper into hudson.slaves.CommandLauncher and the command-launcher plugin
> > that is
> > > automatically installed using the Dockerfile.
> > >
> > > So I check /var/log/jenkins/jenkins.log and find the following:
> > >
> > > ```
> > > SEVERE: Failed Loading plugin Script Security Plugin v1.74 (script-security)
> > > java.io.IOException: Script Security Plugin version 1.74 failed to load.
> > > - You must update Jenkins from version 2.164.2 to version 2.176.4 or later to run this plugin.
> > > Jul 01, 2020 4:30:08 PM jenkins.InitReactorRunner$1 onTaskFailed
> > >
> > > SEVERE: Failed Loading plugin Command Agent Launcher Plugin v1.3 (command-launcher)
> > > java.io.IOException: Command Agent Launcher Plugin version 1.3 failed to load.
> > > - Script Security Plugin version 1.74 failed to load. Fix this plugin first.
> > >
> > > ```
> > >
> > > I uploaded the full log here: https://filebin.net/zu5xadhi0mdwfrpl/jenkins.log?t=44eroook
> > >
> > > I would be super curious to hear from others what "Script Security Plugin" version they have. As this plugin is not explicitly
> > installed in
> > > the Dockerfile but either comes with Jenkins or is a dependency of another plugin.
> > > Interestingly v1.74 was just released 20hours ago. See https://github.com/jenkinsci/script-security-
> > > plugin/blob/master/CHANGELOG.md
> > >
> > > Next I will try if I can somehow try older versions of the script security plugin.
> > >
> > >
> > >
> > > On Wed, Jul 1, 2020 at 6:07 PM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com
> <mailto:Tim.Bird at sony.com> > <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com>
> > <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > > > wrote:
> > >
> > >
> > > Thanks for digging into this deeper. The extra diagnostic information you provide
> > > should be helpful to narrow it down.
> > >
> > > Unfortunately, I'm busy today with a virtual conference (ELC), so I can't look at
> > > this today, but I'll try to see what I can figure out tomorrow.
> > > -- Tim
> > >
> > >
> > > > -----Original Message-----
> > > > From: Fuego <fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org>
> <mailto:fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> >
> > <mailto:fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> <mailto:fuego-
> bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > > > On Behalf Of
> > > Laszlo Sitzer
> > > >
> > > > So I manually created an executor using the Jenkins webui. That part worked.
> > > > Next I tried to run the python-jenkins code from ftc in the container and could reproduce the error without using ftc.
> > > >
> > > >
> > > > ```
> > > > import jenkins
> > > > s = jenkins.Jenkins("http://localhost:8090/fuego")
> > > > s.get_nodes()
> > > > s.create_node('bbb')
> > > >
> > > > ```
> > > > The resulting HTTP POST request that is sent then causes the exception on the Jenkins side.
> > > > From what I understand about the Java part is that the class loader fails to locate a class mentioned in the JSON of the
> > > request.
> > > >
> > > > I started to look into the HTTP POST request that is created when I manually create a executor in Jenkins webui and
> > compare
> > > that to
> > > > what python-jenkins creates.
> > > > By now I also found two more people having this issue with python-jenkins
> > > >
> > > > https://bugs.launchpad.net/python-jenkins/+bug/1845320
> > > > https://bugs.launchpad.net/python-jenkins/+bug/1853147
> > > >
> > > > So it is not a fuego issue ... but I have a hard time understanding who the docker container works for others but fails
> to
> > work
> > > for me. I
> > > > know various factors like UID/GID play into the docker image. But I doubt this is relevant to this specific problem : (
> > > >
> > > >
> > > > On Wed, Jul 1, 2020 at 3:13 PM Laszlo Sitzer <laszlo.sitzer at native-instruments.de <mailto:laszlo.sitzer at native-
> instruments.de> <mailto:laszlo.sitzer at native- <mailto:laszlo.sitzer at native->
> > instruments.de <http://instruments.de> > <mailto:laszlo.sitzer at native- <mailto:laszlo.sitzer at native->
> <mailto:laszlo.sitzer at native- <mailto:laszlo.sitzer at native-> >
> > > instruments.de <http://instruments.de> <http://instruments.de> > <mailto:laszlo.sitzer at native-instruments.de
> <mailto:laszlo.sitzer at native-instruments.de> <mailto:laszlo.sitzer at native- <mailto:laszlo.sitzer at native->
> > instruments.de <http://instruments.de> > <mailto:laszlo.sitzer at native-instruments.de <mailto:laszlo.sitzer at native-
> instruments.de> <mailto:laszlo.sitzer at native-instruments.de <mailto:laszlo.sitzer at native-instruments.de> > > > >
> > > > wrote:
> > > >
> > > >
> > > > I managed to get the Jenkins error message that was masked by the python exception.
> > > > I have uploaded it (it is a HTML page from Jenkins) to https://filebin.net/7ihevnus0vzwuoo6/err.html?t=h64q47ri
> > > >
> > > > Aside from the Java exception on the HTML page, Jenkins mentions some that the following plugins could not be
> > loaded:
> > > >
> > > > - Script Security Plugin
> > > > - Pipeline: Step API
> > > > - JUnit Plugin
> > > > - Command Agent Launcher Plugin
> > > > - Matrix Project Plugin
> > > >
> > > > Could this explain why adding an executor/node fails via ftc? I know this might be more an issue for the jenkins
> > mailing
> > > list.
> > > >
> > > > I will try to manually add a Jenkins executor to see if that part should work (without ftc).
> > > > I will also try to use Jenkins 2.176.4 as suggested by the error message to see if the above mentioned plugins will
> > start to
> > > work
> > > > again.
> > > >
> > > >
> > > > On Tue, Jun 30, 2020 at 10:09 PM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com>
> <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> >
> > <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > >
> > > <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> >
> <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > > > >
> > wrote:
> > > >
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Fuego <fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org>
> <mailto:fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> >
> > <mailto:fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> <mailto:fuego-
> bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > >
> > > <mailto:fuego-bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> <mailto:fuego-
> bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > <mailto:fuego- <mailto:fuego->
> > bounces at lists.linuxfoundation.org <mailto:bounces at lists.linuxfoundation.org> <mailto:fuego-
> bounces at lists.linuxfoundation.org <mailto:fuego-bounces at lists.linuxfoundation.org> > > > > On
> > > > Behalf Of Laszlo Sitzer
> > > > >
> > > > > Hello!
> > > > >
> > > > > I ran into this problem that "ftc add-nodes -b bbb" (or any other board) fails with a cryptic python
> > unicode/ascii
> > > > encoding/decoding
> > > > > exception. I think this is just masking the actual problem with Jenkins.
> > > > >
> > > > > ```
> > > > > # ftc add-nodes -b bbb
> > > > > Traceback (most recent call last):
> > > > > File "/usr/local/bin/ftc", line 5625, in <module>
> > > > > main()
> > > > > File "/usr/local/bin/ftc", line 5553, in main
> > > > > sys.exit(str(e) + '\n' + command_help['add-nodes'][1])
> > > > > UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 4199: ordinal not in range(128)
> > > >
> > > > First - thanks very much for the bug report!
> > > >
> > > > This looks to be an issue with the str(e). It would probably be helpful to see what that
> > > > exception is from the server. You might try moving the do_add_nodes() outside the try/except block
> > > > (or just changing the generic 'except Exception as e:' to something specific that is unlikely for this
> > > > case like: 'except ArithmeticError as e:'
> > > >
> > > > Then the exception should bubble up and maybe print something useful.
> > > >
> > > > (Independent of what the real error is, that str(e) should be changed to something
> > > > that handles Unicode. This is likely from some message from the server with a non-ascii
> > > > char in it.
> > > > >
> > > > > ```
> > > > >
> > > > >
> > > > > I am executing this inside of the fuego-container that I have built from current master branches of fuego.git
> > and
> > > > fuego-core.git repos
> > > > > and started using ./start.sh.
> > > > >
> > > > > I did some printf debugging on the python side and noticed the HTTP request is answered with HTTP 405.
> > > > > When looking at /var/log/jenkins/jenkins.log I can see warnings/errors (ambiguous) and java exceptions.
> > > > >
> > > > > ```
> > > > > WARNING: Error while serving http://localhost:8090/fuego/computer/doCreateItem
> > > > > java.lang.reflect.InvocationTargetException
> > > > > ```
> > > > >
> > > > > Anyone ran into this before?
> > > >
> > > > I haven't seen it before, but it does look like the Unicode error is masking whatever the
> > > > real issue is.
> > > > -- Tim
> > > >
> > > >
> > >
> > >
> >
> >
>
>
More information about the Fuego
mailing list