[cgl_discussion] CGL benchmark

[Eric.Chacron at alcatel.fr]

During last Paris f2f meeting we have discussed a little bit the advantages
and cons to include ( and design before ) a benchmark in CGL Linux
distribution.

As a TEM member i see a lot of advantages:
- better visibility on performances from release to release and from
architecture to architecture, from vendor to vendor, from distro to
distro....

- better control of product behavior using benchmark to detect
defects caused by wrong configuration , options, interaction with drivers,
priority inversion problems ...

For a more pragmatic view of the problem , we'd need to define the
benchmark content with
non trivial measurements to be included. Let's try to have simple metrics
but more
basic than specint for instance.

For example, i'd like to be able to measure maximum latency, and this is
not trivial to have a bench
measuring it as it could happen in telco application context...


Eric




____________________________________________________________________
OSDL CGL specifies that carrier grade Linux shall be delivered with a
standard benchmark enabling to measure the target product performances in a
glance. Both hardware and Linux system software capacities should be
reflected.

The benchmark shall provide applications with different metrics based on
application profiles.

It shall include following subsets:
-     Processing: CPU, memory access, thread context switch, system call
overhead, available memory, ?
-     Real-time: interrupt and scheduling latency, timers resolution/ jiffy
, ?
-     Communication/network: local socket communication, IP forwarding,
physical network latency, communication service ( shall be configurable)
latency / bandwidth and CPU overhead.
-     Storage / file system / mirroring: local disk access, file system
local access, file system NFS access, mirroring overhead.

UP and SMP configurations should be specifically handled.

___________________________________________________________________


OSDL CGL specifies that carrier grade Linux shall enable benchmark
publication and performance comparison of supported protocols.

At least following protocols shall be analyzed and compared:
·     Physical layers: Ethernet 100 BT / Gigabit Ethernet.
·     Network: IP and IPsec.
·     Transport: TCP and SCTP.

UP and SMP configurations should be used.

Analysis should include: message transfer latency, generated CPU load.

Message size and local / remote location of addresses should be studied, IP
performances should take into account forwarding route table size.