[Bridge] Packet reflection breaks Linux bridge

Sat Nov 30 08:54:40 UTC 2013

Hello,
I want to use OpenVPN to extend a layer 2 segment over a routed
connection. I was not able to ping a system on the same subnet, so I
tried to reproduce it with a different network and was _not_ able to: It
worked out of the box. It turned out that I had some Exchange sytem that
sends every packet it gets (including broadcast packets) out again.
After 3 hours of debugging with Michael Gernoth (on CC), it turned out
that the bridge was thinking that the MAC address of the device was on
eth1 instead of 'vpn' and the bridge in Linux was not forwarding any
packet. We we're able to fight the symptoms by issuing the following
command:

ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22

With this command we can access all systems using Layer 2, but not the
two Exchange systems which replicate every packet they're seeing.

This is what the Linux box sends:

(miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
tcpdump: WARNING: vpn: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:35.402501 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 35, length 64
09:35:36.410454 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 36, length 64
09:35:37.418526 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 37, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

This is how it looks, after the Microsoft Exchange settings saw the
package: As you can see, the packets are duplicated at least once,
sometimes twice. This also happens with arp who-has and is-at packets
which confused the Linux bridge because it thinks that the mac is now at
another port:

(miniwheezy64) [~] tcpdump -e -n -i br0 ether host 00:50:56:98:12:ed
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:42.458571 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
09:35:42.458693 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
09:35:42.459019 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:42.459220 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:42.459340 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:43.466531 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
09:35:43.466758 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
09:35:43.467162 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:43.467169 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:43.467366 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:44.474551 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
09:35:44.474774 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
09:35:44.475108 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
09:35:44.475150 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
09:35:44.475160 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
^C
15 packets captured
19 packets received by filter
0 packets dropped by kernel
(miniwheezy64) [~] brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.005056980a22       no              eth1
                                                        vpn

(miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
  1     00:50:56:98:12:ed       no                 0.08

As you can see the mac address of the system is learned on port 1 but
should be learned on port 2.

This is the setup:

Exchange and other systems - VLAN 103 - eth1 - br0 - vpn - OpenVPN - vpn - br0 - eth1 - virtual switch - client (00:50:56:98:12:ed)

It seems that the Linux bridge behaves different than a physical switch.
Because with a physical switch it still forwards the packet. But the
Linux bridge gets offbeat by the reflected arp who-has and is-at packets
and thinks that the mac has moved to a different port even that it
hasn't. So I wonder why does the linux bridge gets confused by that, but
the physical switch not?

I found one other reference which confirms this problem.

http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply

Also I wonder what else than source natting broadcast request we could
do to fight the symptoms? For example I would like to drop all packets
with the source mac address from 00:50:56:98:12:ed which are comming in
via eth1. I tried this using ebtables, arptables and iptables but did
not get the syntax right. Has someone, something I could try?

I'm running Debian wheezy with Linux minisqueeze 3.2.0-4-686-pae #1 SMP
Debian 3.2.51-1 i686 GNU/Linux.

If I set the MAC adress of a system static on the system behind the
layer 2 bridge, the system gets connectivity. If I remove OpenVPN from
the equation and use a layer 2 gre tunnel or just another physical
network card, I have the same behaviour.

Cheers,
        Thomas