[cgl_discussion] Enhancements to mpstat utility

Mon Dec 1 05:31:28 PST 2003

David,

>Another item for release 3, from a customer who is moving multi-threaded
APPs to Linux.
>
>They reported using mpstat to help tune the locking in the original
environment, but were
>
>dismayed that there is no Linux equivalent or other tools for doing the
same.
>
>
>
>After looking at the open source mpstat that is available for Red Hat
Linux, it just displays
>
>the minimal level of processor utilization statistics
(processor/%user/%system/%idle/intr).
>
>Other versions for Unix report additional information:
>
>- Minor/Major faults
>
This is already reported by mpstat 0.1.1 ( i have used your mpstat URL ) .
What does it mean ?

>- Interprocessor cross calls
>
>- Context switches
This is already reported my mpstat and by vmstat too.

>
>- System calls
>
>- Process migrations from this processor
>
>- Kernel lock statistics (Mutex stalls)
>
>- Wait time
>
>

I think this is a first set of measurements we'd like to have.
But these are very general ( for instance intr is the sum of every
interrupt events)
 and it could be good to have also more detailed
event summary like per event en per processor interrupts
( as you have in /proc/interrupt on a UP machine).

>
>This customer said that the inability to use these tools on Linux was
hindering their ability
>
>to tune their application effectively for Linux.
>
>
>
>My guess is that the current Linux mpstat is exposing everything that is
available from the
>
>kernel, and that to do better will require additional statistics gathering
in the kernel as well
>
>as the hooks in the mpstat utility to display them. This may be hard to
sell to LKML.
>
>
I think that using  LTT model for instance ( a hook based tool )  could be
enough to
add features in mpstat or other tool .

>
>While having all of this under on hood is likely best for the customer,
i.e. in mpstat, it would
>
>seem reasonable if the functionality were offered through several tools; I
was thinking of the
>
>lockmeter framework and utilities for the kernel locking statistics.
Having a complete set of
>
>tools for determining bottlenecks would be very useful. Documentation that
describes how
>
>to use the utilities, and how to identify and debug real problems is also
needed.
>

Thanks for the link to lockmeter . I have glanced and i think it's a quite
good tool
for tracing spin lock contention but with a quite significant probe effect
too.
Yes we nned some tool for having knowledge of key mechnisms like the
SMP lock contentions, and others: scheduling , ...

>
>
>For customers who are used to using these tools for tuning on other
Unixes, keeping as
>
>much of the presentation of the stats similar would help, but may be
difficult or impossible
>
>for Linux.

Why do you think it could be impossible ?

Eric

"Howell, David P" <david.p.howell at intel.com>@lists.osdl.org on 10/20/2003
07:19:12 PM

Sent by:    cgl_discussion-bounces at lists.osdl.org

To:    "Hu, Boris" <boris.hu at intel.com>, "cgl_discussion"
       <cgl_discussion at lists.osdl.org>
cc:    "Smith, Stanley W" <stanley.w.smith at intel.com>, "Wenjen,   Wendell"
       <wendell.wenjen at intel.com>
Subject:    RE: [cgl_discussion] Enhancements to mpstat utility

This was one of the tools that I thought could be used to make up the full
requirement.

In conjunction with lockmeter we'd have enough there for users to begin
doing the tuning

that the white papers call out using mpstat.

There are some practical considerations, like the performance impact of
having a lockmeter

enabled kernel to deal with. Any of these performance counters will have a
cost.

Thanks,

Dave Howell

-----Original Message-----
From: Hu, Boris
Sent: Saturday, October 18, 2003 5:41 AM
To: Howell, David P; cgl_discussion
Cc: Smith, Stanley W; Wenjen, Wendell
Subject: RE: [cgl_discussion] Enhancements to mpstat utility

In the linux, vmstat could provide some of the lacking information, such as
minor faults, context switches, system calls.

boris

There are my thoughts, not my employer's.

-----Original Message-----
From: cgl_discussion-bounces at lists.osdl.org
[mailto:cgl_discussion-bounces at lists.osdl.org] On Behalf Of Howell, David P
Sent: Friday, October 10, 20033:34 AM
To: cgl_discussion
Cc: Smith, Stanley W; Wenjen, Wendell
Subject: [cgl_discussion] Enhancements to mpstat utility

Another item for release 3, from a customer who is moving multi-threaded
APPs to Linux.

They reported using mpstat to help tune the locking in the original
environment, but were

dismayed that there is no Linux equivalent or other tools for doing the
same.

After looking at the open source mpstat that is available for Red Hat
Linux, it just displays

the minimal level of processor utilization statistics
(processor/%user/%system/%idle/intr).

Other versions for Unix report additional information:

- Minor/Major faults

- Interprocessor cross calls

- Context switches

- System calls

- Process migrations from this processor

- Kernel lock statistics (Mutex stalls)

- Wait time

This customer said that the inability to use these tools on Linux was
hindering their ability

to tune their application effectively for Linux.

My guess is that the current Linux mpstat is exposing everything that is
available from the

kernel, and that to do better will require additional statistics gathering
in the kernel as well

as the hooks in the mpstat utility to display them. This may be hard to
sell to LKML.

While having all of this under on hood is likely best for the customer,
i.e. in mpstat, it would

seem reasonable if the functionality were offered through several tools; I
was thinking of the

lockmeter framework and utilities for the kernel locking statistics. Having
a complete set of

tools for determining bottlenecks would be very useful. Documentation that
describes how

to use the utilities, and how to identify and debug real problems is also
needed.

For customers who are used to using these tools for tuning on other Unixes,
keeping as

much of the presentation of the stats similar would help, but may be
difficult or impossible

for Linux.

Reference projects:

            http://freshmeat.net/projects/mpstat/?topic_id=862  - Utility
version used on Linux (?)

            LKML, for kernel changes to enable additional statistics

Documentation:

            - Using the mpstat load displayer (SCO)

            - http://docs.sun.com/db/doc/806-0625/6j9vfilr4?q=mpstat&a=view

            -
http://sunsite.uakom.sk/sunworldonline/swol-08-1998/swol-08-perf.html

These are my opinions and not official opinions of Intel Corp.

David Howell

Intel Corporation

Telco Server Development

Server Products Division

Voice: (803) 216-2359  Fax: (803) 216-2178

Intel Corporation

Columbia Design Center, CBA-1

100 Center Point Circle, Suite 210

Columbia, SC 29210

david.p.howell at intel.com

_______________________________________________
cgl_discussion mailing list
cgl_discussion at lists.osdl.org
http://lists.osdl.org/mailman/listinfo/cgl_discussion