[Fuego] [PATCH] ftc: add a new command to interact with pdudaemon

daniel.sangorrin at toshiba.co.jp daniel.sangorrin at toshiba.co.jp
Mon Apr 15 09:03:33 UTC 2019


Hi Tim,

> From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> Sent: Friday, April 12, 2019 9:39 AM
> Daniel,
> 
> I've kind of lost track of where we are with this.  Can you update me on the
> status of new ftc commands, and their integration with pdudaemon?

OK, this is a quick update. I sent this patch in March:
https://lists.linuxfoundation.org/pipermail/fuego/2019-March/003024.html

What that patch does:
- to add a new command to ftc that can turn on/off/reboot a board
- if PDUdaemon is not running on localhost (on a separate container) the user can specify the IP address where PDUdaemon is running
- the board file contains the information that defines what PDU and port it is connected to. This information is sent to the PDUdaemon through the "requests" library.

Limitations of my patch:
- I didn't demultiplex the command request into multiple power-control tools (I was assuming that we would only need PDUdaemon).
- The overlay layer has its own power-control and demultiplexing code, but it is hard to use from ftc.
 
What to do next:
- I could fix the command to support multiple tools (ttc, PDUdaemon..), and modify the overlay layer code so that it calls ftc recursively/nested.

> Are we waiting for pdudaemon to change its API, or are we moving ahead
> with the interface to is via 'curl'?

I think that we can move ahead because PDUdaemon will support both the old and the new API in any case (nothing difficult to achieve).
I suggest that we use the interface below for Fuego and internally we use an API to connect to each power-control tool:

ftc power-on -b bbb
ftc power-off -b bbb
ftc power-cycle -b bbb
[Note] power-reset, power-reboot are also fine.

> And where are we putting the connection between board and (pdu and
> port)?

There are 2 types of variables:
- variables used by the power-control tools (e.g.: IP address of the PDUdaemon(s))
   - I suggest that we store them in the Fuego configuration file, but allow users to override them per-board file. The most common case is for Fuego users to use a single PDUdaemon on the localhost (usually on a separate container) . I suggest that we use that configuration by default.
- variables used for turning on/off the boards (e.g.: strings that identify the particular PDU and hardware port they are connected to)
  - I suggest that we put that information inside each board file. 

The lab operator will need to associate the PDU with a suitable "driver" (code that makes on/off requests to the PDU through SNMP for example) by adjusting the tool's configuration file. Perhaps it would be convenient to have a possibly optional API for users to select the driver and other parameters (delay) on the fly, without having to setup each power-control tool's configuration file.

> Do we need a new kind of test object?   I've been thinking of having test
> objects
> that are used for different things, like Monitors, Stressors, and Provisioners.
> A monitor would be executed in parallel with a test, to gather information
> from another device like a power meter.  A Stressor would also run in
> parallel, to put a known workload on a device while another test is running.
> And  a Provisioner would be run prior to a test, in order to prepare the board
> for the test by deploying the software under test and booting it.

Sounds like a "power-control" class could be used to do the multiplexing.

> So, I'm starting work on the API for starting, checking the running status of,
> and stopping these other test processes.  Let me know what you think, or if
> you have some example APIs I should look at for inspiration.

For the stressors, I think that most tests can work as Stressors. We could also support testplans as stressors (e.g.: we may want to test cyclictest under io stress first, the under scheduler stress and finally under network stress). To do that, we can add a new phase similar to cleanup (how about test_cancel) so that we can execute the test but cancel it whenever we want.

Thanks,
Daniel

> Thanks,
>  -- Tim
> 
> > -----Original Message-----
> > From: daniel.sangorrin at toshiba.co.jp
> >
> > > -----Original Message-----
> > > From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> > > Sent: Wednesday, March 27, 2019 6:09 AM
> > > To: sangorrin daniel(サンゴリン ダニエル ○SWC□OST)
> > <daniel.sangorrin at toshiba.co.jp>;
> > > fuego at lists.linuxfoundation.org
> > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact with
> > pdudaemon
> > >
> > > More feedback on this topic...  see below.
> > >
> > > > -----Original Message-----
> > > > From: daniel.sangorrin at toshiba.co.jp
> > > >
> > > > Hi Tim,
> > > >
> > > > Thanks again for reviewing in your busy time.
> > > >
> > > > > -----Original Message-----
> > > > > From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> > > > > Sent: Thursday, March 14, 2019 4:55 AM
> > > > > To: sangorrin daniel(サンゴリン ダニエル ○SWC□OST)
> > > > <daniel.sangorrin at toshiba.co.jp>;
> > > > > fuego at lists.linuxfoundation.org
> > > > > Subject: RE: [Fuego] [PATCH] ftc: add a new command to interact
> > > > > with
> > > > pdudaemon
> > > > >
> > > > > Here are some overall comments:
> > > > >
> > > > > Rather than add a new command to ftc for managing pdudaemon
> > > > operations,
> > > > > I was expecting to install support for pdudaemon at Fuego's
> > > > > board
> > > > management (/board_control)
> > > > > layer.  That is, I was expecting to see something like this:
> > > > >
> > > > > diff --git a/overlays/base/base-board.fuegoclass
> > b/overlays/base/base-
> > > > board.fuegoclass
> > > > > index ae35464..8f34922 100644
> > > > > --- a/overlays/base/base-board.fuegoclass
> > > > > +++ b/overlays/base/base-board.fuegoclass
> > > > > @@ -299,6 +299,9 @@ function ov_board_control_reboot() {
> > > > >    "ttc")
> > > > >      $TTC $TTC_TARGET reboot
> > > > >      ;;
> > > > > +  "pdudaemon"):
> > > > > +    curl
> > > > >
> > > >
> >
> "http://$PDUDAEMON_HOST:$PDUDAEMON_PORT/power/control/reboot?
> > > > hostname=$PDU_HOSTNAME&por
> > > > > t=$PDU_PORT"
> > > > > +    ;;
> > > > >    *)
> > > > >      abort_job "Error reason: unsupported BOARD_CONTROL
> > > > ${BOARD_CONTROL}"
> > > > >      ;;
> > > > > ----
> > > >
> > > > Alright, that sounds like a good idea. I will resend.
> > > >
> > > > > Note that I don't know if pdudaemon supports the 'reboot'
> > > > > control
> > option,
> > > > or if I would have
> > > > reboot is supported
> > > >
> > > > > to synthesize something by sending 'on' and 'off'.  But you get
> > > > > the idea,
> > I
> > > > think.
> > > > > I'm not sure if the hostname in the URL is supposed to be the
> > > > > board
> > name
> > > > or the pdu name,
> > > > PDU_HOSTNAME is the PDU's IP address/hostname.
> > > > PDUDAEMON_HOSTNAME is the IP address/hostname of the computer
> > > > where pdudaemon is running.
> > > > The board doesn't know that it can be "killed" at anytime by the PDU.
> > > >
> > > > > but presumably PDUDAEMON_HOST and PDUDAEMON_PORT would
> > come
> > > > from
> > > > > fuego.conf, and PDU_HOSTNAME and PDU_PORT would come from
> > the
> > > > board file?
> > > > From the board file because each board can be in a different
> > > > place. These two parameters are passed to the pdudaemon.
> Pdudaemon
> > > > then
> > internally
> > > > uses them to connect with the PDU by the supported protocol (e.g.:
> > > > SNMTP).
> > >
> > > OK. As discussed in the other thread, this means that the board file
> > > needs to be parsed by ftc to execute these, to get these
> > > board-related test
> > variables.
> > > For now I believe it's OK to ignore processing overlays and stored
> > > variables
> > for
> > > this.  However, I think the ftc internal function get_board_vars()
> > > does all
> > this processing
> > > anyway, so it might just be easier to use that.
> >
> > Yes, this works fine. I already have it implemented and it's ready for
> > demultiplexing.
> >
> > > > > If more fine-grained control is desired, than just the 'reboot'
> > > > > operation, we could add more ov_board_control() functions if
> > > > > needed, and have
> > > > Fuego use those.
> > > > > e.g. ov_board_control_on() and ov_board_control_off().
> > > >
> > > > Yes, I think we need on off.
> > > > In particular, I was thinking about Fuego testplans that would
> > > > work like
> > this:
> > > >
> > > > ftc pdu -b bbb --command on
> > > > ftc expect -b bbb --regex "u-boot>" --send SIGINT ftc send -b bbb
> > > > --command "run tftpboot"
> > > > ftc expect -b bbb --regex "mypromptlinuxprompt" --send "root"
> > > > echo ftc pdu -b bbb --command off | at now + 1 minute ftc run-test
> > > > -b bbb -t Functional.StressFilesystem sleep 60 ftc pdu -b bbb
> > > > --command on ftc expect -b bbb --regex myprompt ftc run-test -b
> > > > bbb -t Functional.ext4 ftc pdu -b bbb --command off
> > >
> > > A couple of notes here:
> > > 1) It looks like get_board_arg() looks for FUEGO_BOARD in the
> > > environment, and uses that if present.
> > > I'm pretty sure you can do: export FUEGO_BOARD=bbb, and then omit
> > > the
> > '-b bbb' on every line.
> > > From inside a test, maybe we should make this automatic, and set
> > FUEGO_BOARD from
> > > NODE_NAME when we do run-test.
> >
> > I didn't know.
> > I also had the idea to add a patch that would assume the board 'local'
> > if -b was not provided.
> >
> > > 2) 'ftc expect' looks nice, and it shouldn't be too hard to do since
> > > python
> > has an 'expect' module
> > > (famous last words... :-)
> >
> > Actually, after looking at your ttc application it looks like you
> > already have an implementation (ttc wait-for). I guess it should be quite
> similar.
> >
> > > 3) We should discuss the high-level operations we want to do, and make
> > > ftc operations for those.   For some tests, the details of how the board is
> > booted
> > > and provisioned should be opaque to the test.
> >
> > Yes, sure.
> > I think that we are arriving to what LAVA defines in yaml:
> > -power on
> > -wait for u-boot
> > -send commands to u-boot to provision the board -wait for login -run
> > the test ...
> >
> > So probably Fuego users will end up defining "test jobs" (similar to
> > LAVA
> > jobs) where they prepare a sequence of ftc instructions
> > (power-on-board, wait-for, run-test, run-test again, power-off-board,
> > et..). And we could even put a yaml layer on top of that (similar to LAVA
> syntax).
> >
> > > One issue I see with your low-level steps, is that they encode
> > > things that are board-specific (like the boot prompt regexes) into
> > > the commands I'm not opposed, to low-level operations being encoded
> > > into tests.
> > > It's important, though, to try to isolate board-specific items so
> > > that the tests don't become dependent on a specific board,
> > > bootloader, distro, board control layer, or even operating system.
> > > (e.g. I've started work on supporting Nuttx testing with Fuego).
> >
> > Sure, we can create BOARD variables as necessary, and still allow
> > users to override them, optionally, on the command line for quick testing.
> >
> > > Maybe you intend to parameterize those regexes from the board file.
> > > But this still encapsulates too much of the way a system boots and
> > > this particular example I believe requires that you have a serial port.
> > > That wouldn't be true, for example, with a mobile phone, where you
> > > have adb access, but often not a serial port.
> >
> > I think we need to consider boards that have a combination of serial
> > and ssh access.
> > Serial could be used for booting/provisioning and SSH for
> > deploying/getting results/running commands etc.
> >
> > Also, what I am talking about is not a "test" but rather a "test job"
> > with a sequence of ftc commands that would be customized by users and
> > not necessarily shared.
> >
> > > Here's how I've used ttc for board control and provisioning:
> > > ttc setenv bbb  # this allows me to omit 'bbb' on every following
> > > command
> > line
> > > ttc get_kernel
> > > ttc set_config CONFIG_SOME_KCONFIG=y;CONFIG_SOME_VALUE=255
> > > ttc kbuild
> > > ttc kinstall
> > > ttc reboot
> > > ttc wait-for "ttc run 'echo alive'"
> > > ttc cp testfiles target:/test_directory ttc run "some commands"
> > > ttc cp target:/test_directory/testlogs host_directory
> >
> > This sequence is what I am calling a "test job" (not to confuse with
> > Jenkins
> > jobs) and I think it works pretty well.
> > If we were going to integrate that with LAVA, most of the steps would
> > be done by LAVA and we would only need the "ttc run ..." part.
> >
> > > > And we can make more complicated things.
> > > > [Note] The ftc expect is something I am designing now to interact
> > > > with
> > the
> > > > board in a flexible way, and provide LAVA-like functionality.
> > > I'd like to have a discussion with you next week about the best
> > > approach in Fuego to handle provisioning, and building up complex
> > > test pipelines.  I think some sub-operations should be in ftc, and
> > > maybe some should be put into tests that call sub-tests.
> >
> > Oh "test pipelines" sounds better than "test jobs". Sure, let's discuss that.
> >
> > Thanks,
> > Daniel
> >
> > >
> > >  -- Tim
> > >
> > > >
> > > > > Once we've plugged pdudaemon into the board layer, I'd prefer to
> > have a
> > > > generic
> > > > > set of operations that someone can use from ftc, like:
> > > > >    ftc reboot-board -b <board>
> > > > >    ftc power-on-board -b <board>
> > > > >    ftc power-off-board -b <board> and maybe even
> > > > >    ftc show-status-of-board <board>
> > > >
> > > > Maybe we can use subcommands like this.
> > > > ftc power reboot -b bbb
> > > > ftc power on -b board
> > > > ftc power off -b board
> > > > ftc power status -b board
> > > >
> > > > Or put hyphens, but it would rather have the same 1st work for
> > > > bash autocomplete to be more intuitive.
> > > >
> > > > > In some labs, there may be a layer between pdudaemon and Fuego
> > (for
> > > > example libvirt,
> > > > > or labgrid or even LAVA).
> > > > >
> > > > > I think I'd rather have an abstraction layer between Fuego and
> > > > pdudaemon, rather than
> > > > > integrating it directly into our user interface.
> > > > >
> > > > > Thoughts?
> > > >
> > > > See above.
> > > > Thanks,
> > > > Daniel
> > > >
> > > > >  -- tim
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Daniel Sangorrin
> > > > > >
> > > > > > This patch adds the "pdu" command to send requests for
> > > > > > pdudaemon to power-on/off or reboot a board.
> > > > > >
> > > > > > Signed-off-by: Daniel Sangorrin
> > > > > > <daniel.sangorrin at toshiba.co.jp>
> > > > > > ---
> > > > > >  scripts/ftc               | 80
> > > > > > +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  scripts/ftc_completion.sh |  5 +++
> > > > > >  2 files changed, 85 insertions(+)
> > > > > >
> > > > > > diff --git a/scripts/ftc b/scripts/ftc index 7153c01..30ea470
> > > > > > 100755
> > > > > > --- a/scripts/ftc
> > > > > > +++ b/scripts/ftc
> > > > > > @@ -426,6 +426,29 @@ A message and the exit code indicate if
> > > > > > the
> > > > resource
> > > > > > was released.
> > > > > >
> > > > > >     ex: ftc release-resource -b beaglebone"""),
> > > > > >
> > > > > > +"pdu": ("Send a request to pdudaemon.",
> > > > > > +    """Usage: ftc pdu -b <board> --command on|off|reboot [--
> > > > > > pdudaemon_hostname localhost --pdudaemon_port 16421]
> > > > > > +
> > > > > > +This command is used to send a request for pdudaemon to
> > > > > > +power-
> > on,
> > > > > > power-off or reboot a board.
> > > > > > +
> > > > > > +The command accepts these arguments:
> > > > > > + - board file: bbb, raspberrypi3, etc (see ftc list-boards)
> > > > > > + - command: on|off|reboot
> > > > > > + - pdudaemon_hostname: hostname or IP address where
> > pdudaemon is
> > > > > > running (default: localhost)
> > > > > > + - pdudaemon_port: port where pdudaemon is listening (default:
> > 16421)
> > > > > > +   [Note] make sure that pdudaemon is configured to accept
> > > > > > + http
> > > > requests
> > > > > > +
> > > > > > +Additionally, you need to define two variables in your board file:
> > > > > > + - PDU_HOSTNAME: the PDU hostname or IP address used in the
> > > > > > pdudaemon configuration file
> > > > > > + - PDU_PORT: the PDU port number (a plug port, not a tcp
> > > > > > + port)
> > > > > > +
> > > > > > +You need to install pdudaemon by yourself
> > > > > > +  $ git clone https://github.com/pdudaemon/pdudaemon
> > > > > > +  $ cd pdudaemon
> > > > > > +  $ less README.md
> > > > > > +
> > > > > > +Example: ftc pdu -b raspberrypi3 --command on"""),
> > > > > > +
> > > > > >  "package-test": ("Package a test.",
> > > > > >      """Usage: ftc package-test <test_name> [options]
> > > > > >
> > > > > > @@ -4783,6 +4806,59 @@ def do_release_resource(conf, options):
> > > > > >
> > > > > >      return 0
> > > > > >
> > > > > > +def do_pdu(bvars, conf, options):
> > > > > > +    # pdudaemon hostname
> > > > > > +    pdudaemon_hostname = 'localhost'
> > > > > > +    if '--pdudaemon_hostname' in options:
> > > > > > +        try:
> > > > > > +            pdudaemon_hostname = options[options.index('--
> > > > > > pdudaemon_hostname')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --
> > pdudaemon_hostname')
> > > > > > +        options.remove(pdudaemon_hostname)
> > > > > > +        options.remove('--pdudaemon_hostname')
> > > > > > +
> > > > > > +    # pdudaemon port
> > > > > > +    pdudaemon_port = '16421'
> > > > > > +    if '--pdudaemon_port' in options:
> > > > > > +        try:
> > > > > > +            pdudaemon_port = options[options.index('--
> > > > pdudaemon_port')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --pdudaemon_port')
> > > > > > +        options.remove(pdudaemon_port)
> > > > > > +        options.remove('--pdudaemon_port')
> > > > > > +
> > > > > > +    # pdu hostname
> > > > > > +    if 'PDU_HOSTNAME' in bvars:
> > > > > > +        pdu_hostname = bvars['PDU_HOSTNAME']
> > > > > > +    else:
> > > > > > +        error_out('PDU_HOSTNAME not specified in board file')
> > > > > > +
> > > > > > +    # pdu port
> > > > > > +    if 'PDU_PORT' in bvars:
> > > > > > +        pdu_port = bvars['PDU_PORT']
> > > > > > +    else:
> > > > > > +        error_out('PDU_PORT not specified in board file')
> > > > > > +
> > > > > > +    # command (on|off|reboot)
> > > > > > +    if '--command' in options:
> > > > > > +        try:
> > > > > > +            command = options[options.index('--command')+1]
> > > > > > +        except IndexError:
> > > > > > +            error_out('Value not provided after --command')
> > > > > > +        options.remove(command)
> > > > > > +        options.remove('--command')
> > > > > > +        if command not in ['on', 'off', 'reboot']:
> > > > > > +            error_out('Unrecognized PDU command %s' % command)
> > > > > > +    else:
> > > > > > +        error_out('Command not supplied')
> > > > > > +
> > > > > > +    query='&'.join(['hostname='+pdu_hostname, 'port='+pdu_port])
> > > > > > +    url = 'http://%s:%s/power/control/%s?%s' %
> > > > (pdudaemon_hostname,
> > > > > > pdudaemon_port, command, query)
> > > > > > +    resp = requests.get(url)
> > > > > > +    if resp.status_code != 200:
> > > > > > +        error_out("pdudaemon did not accept the request")
> > > > > > +    return 0
> > > > > > +
> > > > > >  def main():
> > > > > >      # use global module names
> > > > > >      global re, time, copy2, subprocess, signal, fcntl,
> > > > > > requests, json @@ -5094,6 +5170,10 @@ def main():
> > > > > >          do_delete_var(bvars, options)
> > > > > >          sys.exit(0)
> > > > > >
> > > > > > +    if command == "pdu":
> > > > > > +        rcode = do_pdu(bvars, conf, options)
> > > > > > +        sys.exit(rcode)
> > > > > > +
> > > > > >      error_out("Unknown command %s" % command)
> > > > > >
> > > > > >
> > > > > > diff --git a/scripts/ftc_completion.sh
> > > > > > b/scripts/ftc_completion.sh index 984e003..2cfb77a 100755
> > > > > > --- a/scripts/ftc_completion.sh
> > > > > > +++ b/scripts/ftc_completion.sh
> > > > > > @@ -55,6 +55,7 @@ _ftc()
> > > > > >          ["list-tests"]="-q"
> > > > > >          ["package-run"]="-o -f"
> > > > > >          ["package-test"]="-o -f"
> > > > > > +        ["pdu"]="-b --command --pdudaemon_hostname --
> > > > pdudaemon_port"
> > > > > >          ["put-request"]="-R -s"
> > > > > >          ["put-run"]=""
> > > > > >          ["put-test"]=""
> > > > > > @@ -123,6 +124,10 @@ _ftc()
> > > > > >              COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
> > > > > >              return 0
> > > > > >              ;;
> > > > > > +        '--command')
> > > > > > +            COMPREPLY=( $( compgen -W "on off reboot" -- "$cur" ) )
> > > > > > +            return 0
> > > > > > +            ;;
> > > > > >          'put-run')
> > > > > >              COMPREPLY=( $( compgen -W "$(ftc list-runs -q)" -- "$cur" ) )
> > > > > >              return 0
> > > > > > --
> > > > > > 2.7.4
> > > > > >
> > > > > > _______________________________________________
> > > > > > Fuego mailing list
> > > > > > Fuego at lists.linuxfoundation.org
> > > > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego


More information about the Fuego mailing list