[Fuego] Need to see logs real-time on the console when target board is connected via serial port

Tue Mar 12 18:42:07 UTC 2019

> -----Original Message-----
> From: Dhinakar Kalyanasundaram 
> 
> Hi Tim,
> 
> 
> 
> Thanks a lot for the updated version of 'serio' utility.
> 
> We tried it on Functional.fuego_sleep (for 5mins) and our test which ran for
> 1.5 hrs.
> 
> It works as intended and no issues so far.

Great!  Thanks for testing it.

I'm still working with the upstream serio maintainer, and the code
may change again.  Hopefully, you can test another version of this
if it changes again.

> 
> I mainly wanted this feature to get the log up to the point in case a long
> running test (12hrs or more) failed/hanged/crashed.
>
> So far no issue up to 1.5hrs of execution, will let you know if we face any
> issues during long running tests.

Thanks.

By the way, there is one thing in the code path that worries me a little
with an extremely long test.  Internally all the data for a single run is
buffered in a single data variable.  I'm not sure this will scale well, as
the amount of data that could accumulate could be quite large.
Now that the data is output as it is read, it's unclear to me that this
internal buffer is needed any more.  So I might remove it, which would
be another refactoring of this code path.

> One question, in the current code, data is flushed out every 100 lines or
> based on time interval? (256ms?).
Right now, the serio code flushes the data on every character.  This is fairly
wasteful on the host side.  It works fine if you're only running one test on the
host, but would consume cycles unnecessarily and possible cause scaling problems
when running hundreds of (serial-based) Fuego tests.

I was thinking of backing this off to every line (when '\n' is encountered).  But I think
your idea to have it based on a time interval is good also.  I'll have to give it some thought.

It appears that the Jenkins code does not flush data through the system any faster
than once per line.  In Functional.fuego_sleep, the progress dots ('.') don't show up
in the web interface until a line feed is encountered (at least as far as I've seen on my
system).

As an aside from the discussion of the flushing issue...
Finally - did you use the Functional.fuego_sleep from the repository, or did you download
the Functional.fuego_sleep package from fuegotest.org?  If you downloaded the package,
I'm very interested to hear what your experience was.  Was it easy?  Did you encounter any
problems?  I'm considering adding some features to 'ftc' to make this easier, and it would
be nice to hear your feedback.

(By the way, if you did download the test package from the server, this would make the
first "real" use of the server and the test package system, so that's pretty neat!)

I'm thinking of adding 'ftc list-tests -r' (or --remote) to show remote tests on the server that
are available for download, and 'ftc install-test -r' (or --remote) to install a test directly from
the server (so the user doesn't have to manually download the test).
Let me know if you think these would be useful.
(...end of aside)

 -- Tim

> --------- Original Message ---------
> 
> Sender : Tim.Bird at sony.com <Tim.Bird at sony.com>
> 
> Date : 2019-03-10 00:46 (GMT+5:30)
> 
> Title : RE: Need to see logs real-time on the console when target board is
> connected via serial port
> 
> 
> 
> > -----Original Message-----
> > From: Dhinakar Kalyanasundaram >
> > Dear Fuego Users,
> >
> > Real-time logs are not displayed on Jenkins/Fuego console (in the
> > dashboard) while using boards connected via serial port to the host system.
> >
> > Target Board connected directly to fuego host through UART cable and the
> > board configuration is set to Serial with Baud rate as 115200.  When any
> script
> > triggered executes for longer duration, output does not print in real-time
> on
> > Jenkins console, instead outputs are displayed only after  execution is
> > completed.
> 
> I took at look at this, and the problem was in serio, which is located in the
> Fuego container at /usr/local/bin.
> 
> I have made some changes, but not pushed them upstream yet.  Please find
> an experimental version of 'serio' attached.  Can you please try this one
> (copy it into the docker
> container and overwrite the previous 'serio' in /usr/local/bin.  You might
> want to save a copy of the original serio, to restore back, if this one doesn't
> work)
> 
> Please test this out and let me know if it solves the problem.
> 
> Also, I've been testing this with Functional.fuego_sleep.
> There's a new version of this test available, which is useful IMHO
> for testing this particular problem.  I've uploaded a test package for
> this to the Fuego server.  You can find it on this page:
> 
> http://fuegotest.org/cgi-bin/fserver.py/tests
> 
> Please download Functional.fuego_sleep-1.5-1.ftc (not the yaml file)
> and install it in your system with: ftc install-test <test-package-name>.
> Then create a job for it, run the test, and let me know what happens.
> 
> Thanks,
>  -- Tim
> 
> P.S.  There's actually a separate problem that's embedded in your description
> below, which is gathering console information during a test run when the
> primary Fuego connection is over ssh.  For that, I'm investigating some
> a different solution.
> 
> >
> > 1. Hence if the test hangs or crashes, we are not able to get the logs until it
> > hung/crashed.
> >
> > 2. Also, if we abort or stop the test during the execution no logs are
> available.
> >
> > 3. In case of long running tests, complete logs are not available, only
> > truncated log is available (may be the size of buffer)
> >
> >
> >
> > Logs are displayed real-time output on console when board configuration is
> > set to SSH.
> >
> > Note: Instead of fuego, if we directly connect to the board via minicom or
> > teraterm we able to see logs real-time.
> >
> >
> >
> > Experimented with Serial board configuration IO_TIME_SERIAL=0.1 and still
> > real-time outputs are not observed.
> >
> >
> >
> > Snippet of Board configuration used is given below:
> >
> >
> >
> > SERIAL="/dev/ttyUSB0"
> > BAUD="115200"
> > LOGIN="root"
> > BOARD_TESTDIR="/tmp"
> > PASSWORD="root"
> > PLATFORM="aarch64-linux"
> > TRANSPORT="serial"
> > IO_TIME_SERIAL="0.1"
> > ARCHITECTURE="arm64"
> >
> >
> >
> > Any suggestions to fix this, please let me know?
> >
> >
> >
> > Regards,
> >
> > Dhinakar
> >
> >
> >
> >
> >
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> >
> kar.k&do=bWFpbElEPTIwMTkwMzA1MTEwNzE2ZXBjbXM1cDZjOGFiOWRiNTI
> >
> xMTljNjY2YzRkMzc0OThiYzY5ZjI4YyZyZWNpcGllbnRBZGRyZXNzPVRpbS5CaXJ
> > kQHNvbnkuY29t>
> 
> 
> 
> <http://ext.samsung.net/mail/ext/v1/external/status/update?userid=dhina
> kar.k&do=bWFpbElEPTIwMTkwMzExMTM1MDQ3ZXBjbXM1cDMyNjM1YWE
> 5NjkxODk5MWZjOWVkMDkyYWQwYmIyNjZjMCZyZWNpcGllbnRBZGRyZXNz
> PVRpbS5CaXJkQHNvbnkuY29t>