[lxc-devel] Poor bridging performance on 10 GbE

Ryousei Takano ryousei at gmail.com
Wed Mar 18 08:56:53 PDT 2009

Hi Daniel,

On Wed, Mar 18, 2009 at 7:10 PM, Daniel Lezcano <dlezcano at fr.ibm.com> wrote:
> Ryousei Takano wrote:
>> Hi all,
>> I am evaluating the networking performance of lxc on 10 Gigabit Ethernet
>> by
>> using netperf benchmark.
> Thanks for doing benchmarking.
> I did two years ago similar tests and there is an analysis of the
> performances at:
> http://lxc.sourceforge.net/network/benchs.php
> It is not up to date, but that will give you some clues of what is happening
> with this overhead.
I am using VServer because other virtualization mechanisms, including OpenVZ,
Xen, and KVM cannot fully utilize the network bandwidth of 10 GbE.

Here are the results of netperf bencmark:
	vanilla (2.6.27-9)		9525.94
	Vserver (	9521.79
	OpenVZ (	2049.89
	Xen (		1011.47
	KVM (2.6.27-9)		1022.42

Now I am interesting to use LXC instead of VServer.

>> Using a macvlan device, the throughput was 9.6 Gbps. But, using a veth
>> device,
>> the throughput was only 2.7 Gbps.
> Yeah, definitively the macvlan interfaces is the best in terms of
> performances but with the restriction of not being able to communicate
> between containers on the same hosts.
This restriction is not so big issue for my purpose.

> There are some discussions around that:
> http://marc.info/?l=linux-netdev&m=123643508124711&w=2
> The veth is a virtual device hence it has not offloading. When the packet
> are sent out, the network stack looks at the nic offloading capability which
> is not present. So the kernel will compute the checksums instead of letting
> the nic to do that either if the packet is transmitted through the physical
> nic. This is a well known issue related to network virtualization and xen
> has developed a specific network driver:
> http://www.cse.psu.edu/~bhuvan/teaching/spring06/papers/xen-net-opt.pdf
>> I think this is because the overhead of bridging devices is high.
> Yes, bridging adds some overhead and AFAIR bridging + netfilter does some
> skb copy.
Thank you for pointers.

>> I also checked the host OS's performance when I used a veth device.
>> I observed a strange phenomenon.
>> Before issuing lxc-start command, the throughput was 9.6 Gbps.
>> Here is the output of brctl show:
>>        $ brctl show
>>        bridge name     bridge id               STP enabled     interfaces
>>        br0             8000.0060dd470d49       no              eth1
>> After issuing lxc-start command, the throughput decreased to 3.2 Gbps.
>> Here is the output of brctl show:
>>        $ sudo brctl show
>>        bridge name     bridge id               STP enabled     interfaces
>>        br0             8000.0060dd470d49       no              eth1
>>                                                                veth0_7573
>> I wonder why the performance is greatly influenced by adding a veth device
>> to a bridge device.
> Hmm, good question :)
>> Here is my experimental setting:
>>        OS: Ubuntu server 8.10 amd64
>>        Kernel: 2.6.27-rc8 (checkout from the lxc git repository)
> I would recommend to use the 2.6.29-rc8 vanilla because this kernel does no
> longer need patches, a lot of fixes were done in the network namespace and
> maybe the bridge has been improved in the meantime :)
I checked out the 2.6.29-rc8 vanilla kernel.
The performance after issuing lxc-start improved to 8.7 Gbps!
It's a big improvement, while some performance loss remains.
Can not we avoid this loss?

>>        Userland tool: 0.6.0
>>        NIC: Myricom Myri-10G
>> Any comments and suggestions will be appreciate.
>> If this list is not proper place to talk about this problem, can anyone
>> tell me
>> the proper one?
> The performances question is more related to the network virtualization
> implementation and should be sent to netdev@ and containers@ (added in the
> Cc' of this email), of course people at lxc-devel@ will be interested by
> these aspects, so lxc-devel@ is the right mailing list too.
> Thanks for your testings
>  -- Daniel

Best regards,
Ryousei Takano

More information about the Containers mailing list