[Bridge] Bridging and NAT: Confusion when linux sees the packet for a second time

[Jarrod Lowe]

Hi,

I have a rather odd problem.

This is a bit of a complicated setup, if you want to know why, I have
included more detail at the bottom.



I have two bridges - lets call them "A" and "B". They have an IP address on
each, so we can route between them. There are also two virtual machines
attached to both "A" and "B" (call them "vs1" and "vs2"), and another
virtual machine attached only to "A" (call it "src")

src sends a packet to an address on the "B" network. It forwards this via
the Host's address on "A".

The Host applies an SNAT rule, to change the source address to another
address of the host (which is on a completely separate interface).

The Host forwards the packet via "B" to vs1.

vs1 forwards the packet via "A" to vs2.

At this point, the *host*, instead of copying the packet (as any good bridge
should do :)  does something... weird. It sends a RST packet, from a
*different* source port, instead of bridging!

/proc/sys/ip_conntrack, at this point, has *two* entries for the connection:
(A=10.17.192.*, B=192.168.148.*,  Host=172.16.64.76)
tcp      6 119 SYN_SENT src=10.17.192.254 dst=192.168.148.2 sport=41869
dport=80 packets=2 bytes=120 [UNREPLIED] src=192.168.148.2 dst=172.16.64.76
sport=80 dport=41869 packets=0 bytes=0 mark=0 secmark=0 use=2

tcp      6 9 CLOSE src=172.16.64.76 dst=192.168.148.2 sport=41869 dport=80
packets=2 bytes=100 src=192.168.148.2 dst=172.16.64.76 sport=80 dport=1088
packets=1 bytes=60 mark=0 secmark=0 use=2

Note that the CLOSE connections match says sport=41869, but its action
dport=1088.

I am utterly confused. Can anyone explain what is happening here, or how to
get around it?








In more detail:
I am using KVM to test what is, in live deployments, a bunch of independent
pieces of kit.

The real machine - "H" is acting as the upstream router (on "A" and
"Internet", internal switch (for "B"), and the firewall (passing through
"H").

    "Internet"
        |
A       |        B
|       He       |
+--[Ha  H   Hb]--+
|                |
+--[V1a v1 V1b]--+
|                |
+--[V2a v2 V2b]--+
|                |
|  [   src  Sb]--+
|                |

v1 and v2 are running Linux IPVS (load balancing).

"A"=10.17.192.0/24, "B"=192.168.148.0/24,  "He"=172.16.64.76

For the particular connections I am having problems with, v1 is the master -
it additionally has the floating IP address Fa. It then chooses that v2
should handle this connection.

In this context, "B" is a private network, on RFC1918 addressing. If it
wants to access the "Internet" side ("A") is must be NATd - in this case
behind the address He. (Yes, "A" and "He" are also on RFC1918 addressing
here - but that is because it is in my lab. A real deployment would have
real IPs there - and "He" would be inside "B". Also, "A" and "B" would be on
physical switches, rather than bridges on a linux box)

I have only one iptables rule; on the nat table, in POSTROUTING; From "B" (
10.17.192.0/24) to anything not "B" (!10.17.192.0/24), SNAT behind "He"
(172.16.64.76)

So "src" wants to talk to "Fa".

I can see a packet "Sb->Fa", send from "src" to "Hb".

I can see a packet "He->Fa" (SNAT'd behind He), send from "H" to "v1" on
"A".

I can see a packet "He->Fa", send from "v1" to "v2" on "B" going up to the
bridge "B".

I can see an *incorrect* RST packet "He->Fa", send from "H" to "v2" on "A",
with the wrong destination port.

I suspect this is to do with ip_conntrack. I think it is getting confused,
and building two conntrack entries, but getting the packets against the
wrong one, possibly?

Is there any way I can force conntrack to not look at bridging packets?

Any Ideas?

Thanks,
-- 
Jarrod Lowe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.linux-foundation.org/pipermail/bridge/attachments/20100727/1c128931/attachment-0001.htm