[Fuego] Please switch server to fuegotest.org (was RE: poll request scheduling)

Tim.Bird at sony.com Tim.Bird at sony.com
Tue Dec 22 23:32:48 UTC 2020


Pooja,

Thanks for the updates.

I spent quite a bit of time trying to debug this.  From what I can tell,
this is a problem on the client side.  The CGI script (fserver.py) calls
the python module 'cgi' to parse the form data for the POST operation
which uploads the run data to the server.  Inside the cgi module,
in the cgi.FieldStorage() class initializer, the code reads from the
CGI program's standard input.  In some cases, the standard input
appears truncated, and the code cannot read the full data in order
to create the file.  When this happens, the code returns the
a message about not being able to find the 'file1' attribute in
order to process the file.  In other cases, the standard input appears
to be completely empty.  This causes the cgi.FieldStorage() to block
internally on a read request. Apparently in this case the web
server eventually times out the request, and the client receives
no result data, which ends up showing the message:

Server returned message:

(with no message).

I did notice that the 'requests' module used in your setup is different from the
one I'm using.  The client string for mine is: "python-requests/2.12.4", whereas
the client string for your submissions is: "python-requests/2.25.1". I don't think
this should make a difference, but if you could try installing the specific 2.12.4
version of the 'requests' python module, it would help us eliminate that difference
between the two setups.  My other theory is that there's a firewall that's messing
up the data (somehow truncating the http request body in the POST message)
between the client and server.  I don't have
the capability to sniff the network traffic at the server.  Can you try
to run wireshark or some other packet tracer, and see if the data being sent
by ftc to the network looks correct?  The http request should include the data for
the packaged run file, in a multipart section in the body of the request.

Finally - I saw one other thing that was quite strange.  When the server CGI
script hangs (blocks reading the request data), I would have expected that 'ftc'
would wait for the response before proceeding.  But I often see the next
operation from ftc (after the put_run operation), which is an update_request,
come to the server within less than a second.  I'm not sure why this is, and
I have no explanation for it.  The calls to the 'request' module by ftc should
be completely synchronous - so I should not be seeing the update_request
operation at the server until the client times out on the previous 'put_run'
request.

Anyway - I've run out of time to work on this for now, and I don't really have
a solution.  Based on what I've seen so far, I don't think this is a server-side issue.
Please check on your side and see if the version of the request module,
something to do with the firewall, or examining the network packet data, can
provide any additional insights into the problem.

 -- Tim


> -----Original Message-----
> From: Pooja Sanjay More <pooja.sm at pathpartnertech.com>
> Hi Tim,
> 
> Run test still doesn't  update the result object on fserver.
> 
> poll request log:
> 
> root at cpu-366U:/# fuego-core/scripts/poll_requests.sh
> == Polling server "fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin> " for requests ==
> Processing requests for any board
> Waiting 60 seconds between polls of the server.
> Running forever (no specified duration)
> Type Ctrl-C to exit
> Checking
> Running request: request-2020-12-21_11:22:58.84-pptlab:rpi
> Trying to get request 'request-2020-12-21_11:22:58.84-pptlab:rpi' from server
> Executing test Functional.hello_world on board rpi (using spec default)
> Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> ===== doing fuego phase: build =====
> The test is already built
> Fuego test_build duration=0 seconds
> ===== doing fuego phase: deploy =====
> ===== doing fuego phase: snapshot =====
> Firmware revision: 5.4.83-v7-51+
> ===== doing fuego phase: run =====
> -------------------------------------------------
> Hello World!
> 
> This is a test program, to demonstrate test success and failure
> Test result: SUCCESS
> -------------------------------------------------
> ===== doing fuego phase: post_test =====
> ===== doing fuego phase: processing =====
> log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> chart config not found. Using default values.
> Fuego: requested test phases complete!
> Packaging run 'Functional.hello_world-default-104-rpi'
> run/
> run/machine-snapshot.txt
> run/syslog.before.txt
> run/devlog.txt
> run/testlog.txt
> run/run.json
> run/syslog.after.txt
> run/prolog.sh
> run/spec.json
> run/build.xml
> run/consolelog.txt
> Run packaged successfully, and is at: /tmp/run-Functional.hello_world-default-104-on-pptlab:rpi.frp
> Can't put run to server
> Server returned message:
> Run Functional.hello_world-default-104-rpi put to server
> Checking Waiting ..................................
> 
> 
> Also we have installed "ttc" outside the container and configured it for basic commands like
> 
> console and login, it's working for rpi board.
> 
> 
> 
> Thanks and regards,
> Pooja More
> 
> 
> On Wed, Dec 16, 2020 at 9:16 PM Pooja Sanjay More <pooja.sm at pathpartnertech.com <mailto:pooja.sm at pathpartnertech.com> >
> wrote:
> 
> 
> 	Hi Tim,
> 
> 	We have configured the fuego to fuegotest.org <http://fuegotest.org>  server.
> 	We tried put-request and run-request for rpi board after
> 	configuring it to fuegotest.org <http://fuegotest.org> , it's working fine.
> 	But the test run  does not create a link for test result object on fserver.
> 	We have scheduled a poll request for the rpi board.
> 
> 
> 	Thanks and Regards,
> 	Pooja More
> 
> 
> 	On Wed, Dec 16, 2020 at 8:59 AM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
> 
> 
> 		My Internet provider currently has some routing issues that prevent me
> 		from accessing birdcloud.org <http://birdcloud.org> .  People outside my network have no problems,
> 		but I can't access from my home network.  I've been working to get the
> 		issues resolved, but currently, I cannot submit jobs to the Fuego server
> 		on birdcloud.org <http://birdcloud.org> .
> 
> 		Therefore, I have transitioned my work to the main public Fuego
> 		server on fuegotest.org <http://fuegotest.org> .
> 
> 		Can you please switch your Fuego configuration to use the fuegotest.org <http://fuegotest.org>
> 		server, instead of birdcloud.org <http://birdcloud.org> ?
> 
> 		Please change the value of "server_domain" in your fuego-ro/conf/fuego.conf
> 		file to the following (which is the original default value for a Fuego installation):
> 
> 		server_domain=fuegotest.org/cgi-bin <http://fuegotest.org/cgi-bin>
> 
> 		I have added the boards humanshu_wks:rpi and pptlab:rpi to that server.
> 		There was also a board for fuegohost:rpi, which looks like it may have been
> 		related to your lab.  Can you let me know what's going on with the humanshu_wks
> 		host entry and board?  Is that someone from your team?
> 
> 		(If not, it could be time to roll out some security for the server.)
> 
> 		Let me know when you have switched your configuration, and I will send
> 		some test requests your way. Actually, I'll post a test request on fuegotest.org <http://fuegotest.org>
> 		(for pptlab:rpi) to see if your host picks it up.
> 
> 		Thanks.  Sorry about this issue.
> 		 -- Tim
> 
> 
> 		> -----Original Message-----
> 		> From: Pooja Sanjay More <pooja.sm at pathpartnertech.com <mailto:pooja.sm at pathpartnertech.com> >
> 		>
> 		> Hi Tim,
> 		>
> 		> Jenkins job has been added to schedule a poll request  to run it for 12 hrs on a daily basis and 15 minutes  wait time  is
> given. Previously
> 		> rpireboot.py  was failing through jenkins job because of permission issues. Now jenkins is added to the dialout group so
> reboot is also
> 		> working as jenkins job.
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		>
> 		> Thanks and regards,
> 		> Pooja More
> 		>
> 		>
> 		>
> 		>
> 		> This message contains confidential information and is intended only for the individual(s) named. If you are not the
> intended recipient, you
> 		> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and
> attached file/s is strictly
> 		> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be
> 		> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. The
> 		> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a
> result of e-mail
> 		> transmission.
> 
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.



More information about the Fuego mailing list