[Fuego] Similar Jenkins-based framework
daniel.sangorrin at toshiba.co.jp
daniel.sangorrin at toshiba.co.jp
Wed Feb 6 08:59:07 UTC 2019
Hello Rafael,
I tried the example in the Readme but I got an error (I put it at the end of this email).
> From: fuego-bounces at lists.linuxfoundation.org <fuego-bounces at lists.linuxfoundation.org> On Behalf Of
> Rafael Gago Castano
> You probably remember me. If you don't, I was on this list evaluating Fuego
> some time ago (Spring-Summer 2017) because we had a LAVA setup that had become
> hard to work with, but at the end we didn't switch because on Fuego the GIT
> repository was mounted on the Jenkin's instance filesystem, this prevented
> multiple users to develop on the same server.
By "the GIT repository" do you mean the Fuego core (fuego-core) repository or are you talking about the fuego-ro folder where the board files are?
> I still found Fuego superior to LAVA on a lot of aspects for our use case, so
> taking time from here and there I developed similar Jenkins-based framework that
> has a very similar philosophy: A very-thin wrapper that delegates the hard work
> to Jenkins, but adds some features from LAVA like dynamic board scheduling.
>
> Luckily my company has allowed me to open-source it, so because this framework
> is inspired in Fuego (some code is a direct copy) I thought that it would be a
> good idea to share it with you here. If you think there are cool ideas/concepts
> you have a reference implementation. If we get some criticism from experts on
> the field it is very welcome.
Thanks a lot for releasing it as open source. It looks quite clean and nicely written.
My only concern is further fragmentation in the Linux testing arena. It would be nice to share some parts to avoid redundant work.
https://elinux.org/Automated_Testing_Summit
> https://github.com/HMSAB/hottest
>
> I will try to make a list of the most important features/differences:
>
> -Like Fuego, it has small tooling coded in Python, tests are written in shell
> and uses Jenkins too.
Q: can you use it without jenkins, from the command line alone?
> -Test are generated by using shell script pieces (with json metadata for
> Jenkins consumption) that support including (copy pasting) other pieces for
> building a script and accumulating all the Jenkins metadata (e.g. each of these
> pieces can define parameters for Jenkins).
I guess that this is similar to the prolog.sh in Fuego, isn't it?
> -There is an API containing a few board functions that the user has to provide
> (dut_cmd, dut_get, dut_put), so board functions are abstracted and tests are
> shareable. All the transports on Fuego are(were) part of the core, on hottest
> they are a "module", so the user can implement them or select one of the
> transports available.
In Fuego you can choose and existing transport but you are also able to override the functions as well thanks to the overlays (ov) system.
ov_transport_cmd, ov_transport_get, ov_transport_put
> -There are predefined milestones on the test sequence. The test writer can add
> functions before or after each of the milestones dinamically, e.g. you can call
> "add_step_before_power_on" and pass a function that crosscompiles a small
> executable and then call "add_step_before_test" and pass a function that copies
> the executable to the device, then use the executable inside a test.
>
> -Allows powering on and off the boards and flashing.
Nice. Fuego also allows powering on and off the board (still a bit experimental, we want to use the pdudaemon) but not flashing it.
Apart from flashing, can it also provision (deploy the kernel and rootfs) the board using tftp and a network filesystem like LAVA?
> -The generator supports include directories, so the board, powering, lab setup
> and firmware flashing implementation can be left on separate user-private
> repositories while still using shared tests.
Is this part what enables having "multiple users developing on the same server"?
> -All the jobs(tests) are self-contained, they embed the board code into the job.
> This avoids having to mount git repositories on the server's filesystem at the
> expense of code duplication on Jenkins (handled by the tool, HDD is cheap). It
> allows tests from older versions to coexist unmodified on the same server too.
Interesting approach. What happens when you want to update the job? Do you need to remove the previous results?
> -It adds a mapping between boards and Jenkins nodes, so you can have some tests
> written for a specific board and some boards connected to the server. When you
> schedule a lot of tests for the same board Jenkins spreads the workload between
> all idle boards. This uses Jenkins labels under the hood.
Nice, I think that Jan-Simon used Jenkins labels for dynamic board scheduling in Fuego as well. But I have never tried it because I don't have the need.
> -The testplans are implemented as Groovy pipelines that schedule a series of tests
> in parallel. These pipelines can have some tests serially executed in a well defined
> order, useful for cases where e.g. there is only one hardware resource/dongle on
> the test PC that many tests use.
Sounds good.
We are planning to use a different approach. Basically it would be a fuego test calling fuego tests in order. This is possible because we can call tests from the command line.
> -For now, it has no extra Jenkins dependencies, so it works with a bare modern
> Jenkins install with default plugins.
Default plugins: do you mean the "recommended plugins" button that appears the first time you run jenkins?
> -Reporting is in a minimum-viable state now, it just generates a Junit report
> and builds a trivial gnuplot graph as a Jenkins artifact.
Another posibility is to send the results to Squad using a REST API as we can do now in Fuego. Squad has a nicer interface than Jenkins.
Thanks,
Daniel
PS: the error
host$ scripts/cli/sync.py http://localhost:8090 $USER 1187d61ba0sd7864656758c64827211a2e sync -f example-cfg/sync/localsetup.json -c chunks -I example-cfg -r test
backing up server state to folder: "hottest.bak/localsetup.json-2019-02-06_17-15-48"
backup done
node "dummyboard-4": generating
node "dummyboard-3": generating
node "dummyboard-2": generating
node "dummyboard-1": generating
node "dummyboard-4": creating
Traceback (most recent call last):
File "scripts/cli/sync.py", line 691, in <module>
main()
File "scripts/cli/sync.py", line 688, in main
args.func (args)
File "scripts/cli/sync.py", line 570, in run_sync
args.item_whitelist)
File "scripts/cli/sync.py", line 519, in jenkins_sync
gen_and_sync_nodes (srv, sync, chunk_dirs, param_dirs, whitelist)
File "scripts/cli/sync.py", line 417, in gen_and_sync_nodes
srv_sync_nodes (srv, syncnodes, whitelist)
File "scripts/cli/sync.py", line 384, in srv_sync_nodes
launcher_params = { "command": gen.JENKINS_NODE_CMD }
File "scripts/cli/sync.py", line 21, in create_node
super (JenkinsWrapper, self).create_node(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 1305, in create_node
self._build_url(CREATE_NODE, params), b''))
File "/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py", line 448, in jenkins_open
e.code, e.msg)
jenkins.JenkinsException: Error in request. Possibly authentication failed [500]: Server Error
Server
=====
Feb 06, 2019 5:15:49 PM org.eclipse.jetty.server.handler.ContextHandler$Context log
WARNING: Error while serving http://localhost:8090/computer/doCreateItem
java.lang.reflect.InvocationTargetException
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:221)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at jenkins.security.ApiCrumbExclusion.process(ApiCrumbExclusion.java:48)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:73)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Error: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"stapler-class":"hudson.slaves.CommandLauncher","command":"hottest/noderun.sh"},"numExecutors":2,"nodeProperties":{"stapler-class-bag":"true"},"name":"dummyboard-4","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/tmp/nodecreate","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":null,"labelString":null,"mode":"EXCLUSIVE"}
at hudson.model.Descriptor.newInstance(Descriptor.java:605)
at hudson.model.ComputerSet.doDoCreateItem(ComputerSet.java:300)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
... 81 more
Caused by: java.lang.IllegalArgumentException: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"stapler-class":"hudson.slaves.CommandLauncher","command":"hottest/noderun.sh"},"numExecutors":2,"nodeProperties":{"stapler-class-bag":"true"},"name":"dummyboard-4","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/tmp/nodecreate","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":null,"labelString":null,"mode":"EXCLUSIVE"}
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:681)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:478)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:474)
at hudson.model.Descriptor.newInstance(Descriptor.java:597)
... 84 more
Caused by: java.lang.IllegalArgumentException: Failed to convert the launcher parameter of the constructor public hudson.slaves.DumbSlave(java.lang.String,java.lang.String,hudson.slaves.ComputerLauncher) throws hudson.model.Descriptor$FormException,java.io.IOException
at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:780)
at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:83)
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:678)
... 87 more
Caused by: java.lang.IllegalArgumentException: Failed to instantiate class hudson.slaves.ComputerLauncher from {"stapler-class":"hudson.slaves.CommandLauncher","command":"hottest/noderun.sh"}
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:681)
at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:478)
at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:778)
... 89 more
Caused by: java.lang.IllegalArgumentException: Class hudson.slaves.CommandLauncher is specified in JSON, but no such class found in classLoader hudson.PluginManager$UberClassLoader
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:674)
... 91 more
Caused by: java.lang.ClassNotFoundException: hudson.slaves.CommandLauncher
at hudson.PluginManager$UberClassLoader.findClass(PluginManager.java:1915)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:669)
... 91 more
More information about the Fuego
mailing list