[cgl_discussion] RE: [Fwd: hotswap and CGL (fwd)]
John J Grana
jjg at pt.com
Thu May 1 12:14:41 PDT 2003
<rusty at linux.co.i To: John J Grana <jjg at pt.com>
ntel.com> cc: Steven Dake <sdake at mvista.com>, cgl_discussion at osdl.org,
cgl_discussion-admin at lists.osdl.org, "Bloch, Jack" <Jack.Bloch at icn.siemens.com>,
05/01/2003 01:53 "'Mika Kukkonen'" <mika at osdl.org>
PM Subject: Re: [cgl_discussion] RE: [Fwd: hotswap and CGL (fwd)]
On Thu, 2003-05-01 at 06:32, John J Grana wrote:
> Not to put too fine a point on this, but some of the subtle problems with
> the present implementation is starting to show itself.
> Last I read the code, the present CompactPCI hotswap still relied on a
> bus being present on the device being swapped. In the discussion below,
> mentions a daemon that retrieves hotswap events (good) but then talks
> a vendor/device id.
> In a cPCI backplane with the actual PCIbus supported, this is not a
> problem. Every board has it's PCI bridge that will respond with a
> vendor/device id.
> Now, fast forward to what is happening today... PICMG 2.16 Packet
> Backplane introduced switched ethernet to the backplane, but "optionally"
> allowed the PCIbus to stay in place. But, the next generation of these
> class of chassis (CompactTCA) has NO PCIbus at all. Ditto for
> I had tried to kick start a thread a while back, asking the question
> "Assuming no PCIbus, what is the best standards method of identifying
> devices in a backplane?". This includes not only hotswapping but boot
> discovery as well. There were 2 ways proposed. IPMI or Ethernet. There
> many pros and cons to each method.
> The present 2.5 cpci_hotplug code has come a long way, looks good. But,
> its present form will only support cPCI systems that have a PCIbus.
> Assuming we (CGL) leave it this way, it is safe to say that both 2.6
> and CGL do NOT support hotswap in "some" 2.16 systems and not at all on
> CompactTCA and AdvancedTCA.
> John Grana
> jjg at pt.com
Exactly what kind of hotswap is not enabled on non-PCI CompactTCA and
In some circumstances, hotswap will work fine. The point I was making was
that in the case of the present cpci_hotplug infrastructure in the 2.5
there is an assumption that there is a PCI bridge out "on the bus". In the
case of ATCA and CompactTCA, the one assumption you can make is that there
are no PCI bridges. No bus at all. So, using the present code does not work
in those systems - again in the present model.
>From what I have seen, these systems either have:
1. a complete compute nodes plugged into a slot, so as far as the kernel
running on this blade is concerned, nothing is different then if the
kernel was running on a normal rack mounted server. What does hotswap
mean in this case?
In a standalone node case, there is no issue. We have a number of cPCI
controllers that run a full 2.4 linux kernel and various
the case of hot swap - no issues. Since we are running as independent
over ethernet, all the remote clients see is the ethernet link go away for
2. some kind of special purpose device (like a line card) that will
communicate with other machines via the particular high speed bus in the
backplane (like Gigabit ethernet). From what I have seen (like with
Gigabit on the backplane), there is nothing to do in the kernel (beyond
what the bus driver is already doing) to support the notion of hotswap.
In a way. If everyone is independent (in this case the line card) and the
running on some other CPU/system understands that it's connections can get
one can come up with a variety of ways to send info. SNMP, some kind of IP
packet, maybe just an 802 ethernet frame. But, this is all proprietary.
I would prefer more standards based.
Take as a base example a chassis based system (CompactTCA or ATCA) that has
a controlling CPU card
and a series of line cards. In a traditional cPCI chassis, the present
hotplug code will work
like a charm. We have all kinds of kernel support for detecting the
for restoring the PCI configuration and a daemon or two to manage this and
let the applications
But, if we move that scenario to CompactTCA or ATCA, not all of it works,
in its present form.
I haven't spent too much time pouring over the code, but one idea is to add
a layer that
abstracts the PCI mechanics and makes it less bus specific. Then we could
add the correct
functions for PCI (like reading the config space and allocating bus
resources) and also support
something like IPMI (which doesn't really need configuration). I believe
the hotplug code would
then work as advertised!
It appears we have now pci_hotplug, cpci_hotplug. We need another - packet
switched backplane hotplug
(psb_hotplug). I believe the main question is - should it be IPMI based or
When I brought this up some months ago, my impression was that IPMI had the
Maybe you could give a specific example of a device that hotswap is not
supported with an ATCA or CompactTCA?
More information about the cgl_discussion