[cgl_discussion] Project Review: Panic Handler Enhancements
Eric.Chacron at alcatel.fr
Eric.Chacron at alcatel.fr
Tue Oct 1 07:12:48 PDT 2002
Some comments on the panic handler enhancements (sorry for delay).
1) the length of the message to be stored in SEL is limited tio 16 bytes.
Is it possible to store not ASCII text in the SEL but only an error code
and to manage a table of panic codes inside the kernel ?
2) page 6 "LKCD" / in order to have a crash dump on diskless node saved
do you think panic handler should be dependant on MCORE / LKCD method ?
3) dumping SEL content (or only last events) in crash dump can be an added
4) sending a SNMP message to a pre-configured IP address: i think this
feature or another similar one based on a lower network layer (MAC
multicast for instance) could be used by the clustering model in order to
enable a faster switchover from the failed node to a standby node. So i
suggest trying to link the "clustering model" with that componant (called
panic handler enhancements) .
"Cress, Andrew R" <andrew.r.cress at intel.com>@lists.osdl.org on 11/09/2002
Sent by: cgl_discussion-admin at lists.osdl.org
To: Eric CHACRON/FR/ALCATEL at ALCATEL
cc: cgl_discussion at osdl.org
Subject: RE: [cgl_discussion] Project Review: Panic Handler Enhancements
RE: crash dump.
Good point, and I didn't include that in the short description.
All Linux panic handling features: bmc_panic, crash dump, and kernel
debugger need to function smoothly together. This is included in the
document (attached), and this bmc_panic kernel feature has been tested with
LKCD, KDB, and KGDB to ensure that these features work smoothly together,
and that a reboot occurs at the specified timeout after all of the panic
RE: Alarms Panel
The alarms panel, in this case, is accessed via the IPMB using the IPMI
MasterWriteRead command. The subroutine that does this would be
differently on non-IPMI platforms. I don't know of any standard API that
covers the alarms panel, so each non-IPMI platform would thus have a
separate get/set subroutine for the alarms panel LED(s). In the current
code, if the alarms panel is not detected, these subroutines are skipped.
RE: SNMP alerting
This BMC firmware capability is defined in the IPMI specification, although
not all systems may implement it. This feature enables this optimal form
SNMP notification even if the OS stays down. It has a configurable alert
destination, and its OID is specified in the IPMI spec (enterprises.3183),
but I still need to add a MIB file for network administrators. If the
system does not have this capability in firmware, one of the panicsel
utilities (showsel) does add the ability to update the Linux syslog with
firmware log entries (after Linux comes back up), and there are many tools
to send SNMP alerts in Linux, based on patterns from syslog. Also, if the
OS does not come back up (God forbid :-), remote access directly to the
firmware log (via serial or LAN) provides not only the panic entry, but any
other sensor events that occurred leading up to it.
RE: SA Forum meta-standard API for system management
www.saforum.org does not show this yet, but it is expected to be published
in October. It should have an API call for each of the defined
sub-functions, so that a library underneath could be built for each
platform, a common one for IPMI, and individual ones for specific
From: Eric.Chacron at alcatel.fr [mailto:Eric.Chacron at alcatel.fr]
Sent: Wednesday, September 11, 2002 8:33 AM
To: Cress, Andrew R
Cc: cgl_discussion at osdl.org
Subject: Re: [cgl_discussion] Project Review: Panic Handler Enhancements
Please let me add some comments on this feature.
>>>4.11 Linux Panic Handler Enhancement
>>>OSDL CGL shall support enriched capabilities on system panic. Currently
>>>default system panic behavior is to print a short message to the console
>>>halt the system. OSDL CGL shall provide a set of configurable functions
>>>including log panic event to system event log as well as the options to
>>>reboot, power down, or power cycle when panic event occurs.
I think that this feature should be compatible with
crash dump generation.
Today panic behavior is not only to print and halt the system but also to
crash dump before rebooting if the option for that is choosen.
I think the requirement is to have the following ordering for an external
- 1)user defined panic handler called.
- 2)log in BMC SEL / SNMP notification ...
- 3)crash dump generation
- 4)hardware reset / power cycle / ...
Is there any problem to synchronise 3) and 4) ?
>>>- write OS Critical Stop message to firmware System Event Log (SEL)
>>>- turn on the Critical Alarm LED on the Telco Alarms Panel
How can this module be able to control the Alarms Panel
( which is architecture and application dependant i think) ?
>>>- send SNMP trap via BMC LAN Alerting mechanism
This supposes the BMC implements a SNMP MIB as an SNMP agent.
Is the format/schema of this MIB generic
and specified somewhere ?
Why not maintaining a sensor indicating the status of the OS on the
machine , that can be polled through IPMB / ICMB ?
>>>The Service Availability Forum is working on an
>>>meta-standard API that could be used to group IPMI and other system
>>>management interfaces under one meta-standard.
I there any link on this meta-standard working group ?
Alcatel - Carrier Network Group
10 rue Latecoere
78140 Velizy France
"Cress, Andrew R" <andrew.r.cress at intel.com>@lists.osdl.org on 10/09/2002
Sent by: cgl_discussion-admin at lists.osdl.org
To: cgl_discussion at osdl.org
Subject: [cgl_discussion] Project Review: Panic Handler Enhancements
1. Requirements related to the Panic Handler Enhancements project:
4.11 Linux Panic Handler Enhancement
OSDL CGL shall support enriched capabilities on system panic. Currently the
default system panic behavior is to print a short message to the console
halt the system. OSDL CGL shall provide a set of configurable functions
including log panic event to system event log as well as the options to
reboot, power down, or power cycle when panic event occurs.
2. How Panic Handler Enhancements meet the CGL requirements:
Panic Handler Enhancements includes coverage for the most widely adopted
standard for platform firmware APIs, which is IPMI. Other platforms can
currently be added to this feature by coding other specific handling for
certain subroutines. Integration of other platforms will be made easier
when a meta-standard API is published to encompass both IPMI and non-IPMI
systems. See design information below for functional descriptions.
3. Project design information:
This feature contains both a kernel module (bmc_panic) and a component
rpm (panicsel) for various utilities.
The bmc_panic kernel module adds additional features to the Linux Panic
Handler so that more information can be saved and passed along if a Linux
panic condition occurs. This package enables the bmc_panic kernel module
handle these additional features.
- write OS Critical Stop message to firmware System Event Log (SEL)
- turn on the Critical Alarm LED on the Telco Alarms Panel
- send SNMP trap via BMC LAN Alerting mechanism
The Panic Handler module (bmc_panic) inserts itself in the panic_notifier
list, then, if a panic occurs, bmc_panic is notified, and it performs
certain functions. It writes an "OS Critical Stop" message to the firmware
System Event Log, turns on the Critical Alarm LED on the Telco Alarms
and sends a BMC LAN Alert via the firmware SNMP capability, even after the
OS is unavailable. This module contains a portion of the valinux IPMI
driver in order to communicate with the BMC via IPMI, but none of the IPMI
interfaces used by bmc_panic are exposed so that it will not conflict with
any other IPMI driver module that may be loaded by the kernel.
The panicsel utilities below allow the user to access the firmware System
Event Log and configure the Platform Event Filter table for the new OS
Critical Stop records.
showsel - show the System Event Log records
pefconfig - show and configure the Platform Event Filter table
to allow BMC LAN alerts from OS Critical Stop messages,
also shows and sets the BMC LAN configuration parameters
hwreset - to cause the BMC to hard reset the system
tmconfig - to set up the BMC Serial port for various modes, such as
Terminal Mode (not yet supported in this release).
The Panic Handler Enhancements currently work with platforms that
support the IPMI standard. If the platform does not support IPMI, these
changes are inert, but the code could be modified for another system
management interface. The Service Availability Forum is working on an
meta-standard API that could be used to group IPMI and other system
management interfaces under one meta-standard. When this becomes
these Panic Handler Enhancements will conform to that API so that non-IPMI
platforms can be integrated more easily.
The Panic Handler enhancements depend on the CONFIG_BMCPANIC kernel
parameter being set in the kernel config file (/usr/src/linux/.config),
in order to export two key variables, and include the bmc_panic module.
The panicsel utilities require an IPMI Driver, either the Intel IPMI
(ipmidrvr, /dev/imb) or the valinux IPMI Driver (/dev/ipmikcs).
4. Code location:
The bmc_panic kernel patch and the panicsel utilities are located at:
cgl_discussion mailing list
cgl_discussion at lists.osdl.org
More information about the cgl_discussion