Source-MAC Filter iptables

Question

I have a HAProxy box with centos7 and doing load balance proxy for a smtp cluster providing mail relay to our customers.

default gateway => 10.0.0.1
master-relay.example.net => 10.0.0.254
relay1.example.net => 10.0.0.10 | gateway 10.0.0.1
relay2.example.net => 10.0.0.11 | gateway 10.0.0.1
relay3.example.net => 10.0.0.12 | gateway 10.0.0.1

Each relay have postfix configured to listen on port 25 and 587.

What I need to achieve is when anyone tries to send a mail connecting directly to one of the 3 relays (relay1, relay2 or relay3) forward the answer packets to the default gateway. This is usually when any external mail server tries to send us an email and connects randomly to one of the 3 MX servers with the same priority.

BUT, when an mobile or webmail client connects to the cluster to relay an email it does to master-relay and this one connects to one of the 3 MX servers to deliver the mail.

This is the HAProxy's configuration for the cluster:

# Puerto 25 - SMTP (Postfix Cluster)
frontend frontend-smtp-25
        bind 10.0.0.254:25 transparent
        option tcplog
        default_backend backend-smtp-25

backend backend-smtp-25
        option tcplog
        source 0.0.0.0 usesrc clientip
        server mx1 10.0.0.10:25 check
        server mx2 10.0.0.11:25 check
        server mx3 10.0.0.12:25 check

# Puerto 587 - STARTTLS (Postfix Cluster)
frontend frontend-smtp-587
        bind 10.0.0.254:587 transparent
        option tcplog
        default_backend backend-smtp-587

backend backend-smtp-587
        option tcplog
        source 0.0.0.0 usesrc clientip
        server mx1 10.0.0.10:587 check
        server mx2 10.0.0.11:587 check
        server mx3 10.0.0.12:587 check

Kernel Parameters:

net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_recycle
net.ipv4.ip_local_port_range = 1025 65535

net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1

Firewall rules:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Now with this configuration if I change the default gateway of my relay boxes to the master-relay ones all goes fine and in postfix log I can see the client's IP address and not the master-relay's one, but here I have a problem, if anyone connects directly to the relay1 for example this one answers throught master-relay and not throught the gateway and the client discards the packet because don't come from the relay1 box.

What I was trying o do is to mark the source mac-address of the master-relay on all the 3 relay boxes and all that match the marks change the default gateway to the master-relay's one.

All the IP's are public IP's and all are visible from internet.

What I can do is put 2 IP's in the same interface and if any packet comes to IP1 then reply to one gateway and if it comes to IP2 reply to the other gateway but I really preffer the mac rules if it is possible.

I can't do a rule to force that if packets come from master-relay ip address reply to it again because it will come to the proxy with the client ip address.

Thanks in advance

Answer 1

Here are the goals and means for a relay (ie relay1, relay2 or relay3) to correctly handle traffic coming both through default-gateway (10.0.0.1) or through the HAProxy master-relay (10.0.0.254) in transparent mode, acting as a gateway:

1. The relay should use the normal gateway default-gateway for normal traffic, that is:

   • locally initiated traffic,
   • outbound replies of remotely initiated traffic which already came through default-gateway.

2. The relay should use the alternate gateway master-relay for HAProxy's transparently relayed traffic, thus only for:

   • outbound replies of remotely initiated traffic which already came through master-relay acting as an alternate gateway.

3. The selector to distinguish routing case 1. from case 2. is the source MAC address of the alternate gateway master-relay. Let's say 02:03:04:05:06:07.

4. The choice, once done at the first packet of a connection, should stay the same for all other packets part of this connection.

This requires policy routing, with an alternate routing table (ip route add table ..., an additional routing decision (ip rule add fwmark...) relying on iptables using the mac match module for the MAC address selector, the MARK target to alter the routing decision, and the CONNMARK target to memorize the decision for the whole connection.

Case 1. being the default case with no special handling and case 2. the exception, you should revert your routing change, to use default-gateway as default gateway, as usual:

ip route replace default via 10.0.0.1

Case 2. will be stored in an alternate routing table. No need to name it, any number is fine, let's choose 1000254 (254 was reserved, it's the main table...):

ip route add table 1000254 default via 10.0.0.254

The routing decision to use it will be triggered by a mark value (coming from iptables's MARK target). Let's choose 254:

ip rule add fwmark 254 lookup 1000254

As can be seen in Packet flow in Netfilter and General Networking, an iptables rule in mangle/PREROUTING or mangle/OUTPUT can set a mark before the routing decision (or the reroute check) is done. That's how in the end iptables will change the route. So for a single incoming packet this would be:

iptables -t mangle -A PREROUTING -m mac --mac-source 02:03:04:05:06:07 -j MARK --set-mark 254

Now to memorize it for the whole connection, it should be wrapped with CONNMARK calls, which store and retrieve the mark in the conntrack entry for this flow and allows to set it only the first time, without forgetting the OUTPUT direction. Some explanations are in this blog: To Linux and beyond ! Netfilter Connmark. There's no need to mark packets for LAN traffic so filter it out (and this can help when using conntrack -L see at the end). In the end, including the previous rule it becomes:

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN
iptables -t mangle -A PREROUTING ! -s 10.0.0.0/24 -m mac --mac-source 02:03:04:05:06:07 -j MARK --set-mark 254
iptables -t mangle -A PREROUTING -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

That's it. This case doesn't even require setting rp_filter to loose mode because there's only one network interface involved.

Please note that you can easily insert more tables, rules and marks to have more than one master-relay, should you need redundancy (or simply just add an additional MAC to master-relay if its IP might change its MAC address for whatever reason). Example for an additional HAProxy with IP 10.0.0.250 and MAC 0A:09:08:07:06:05:

ip route add table 1000250 default via 10.0.0.250
ip rule add fwmark 250 lookup 1000250
iptables -t mangle -I PREROUTING 4 ! -s 10.0.0.0/24 -m mac --mac-source 0A:09:08:07:06:05 -j MARK --set-mark 250

conntrack -L will display the connmark and can thus be used to distinguish if a connection came through master-relay instead of the default, because its mark will be 254 instead of 0:

# conntrack -L -s 198.51.100.1
tcp      6 431635 ESTABLISHED src=198.51.100.1 dst=10.0.0.10 sport=50230 dport=25 src=10.0.0.10 dst=198.51.100.1 sport=25 dport=50230 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
tcp      6 431527 ESTABLISHED src=198.51.100.1 dst=10.0.0.10 sport=49554 dport=25 src=10.0.0.10 dst=198.51.100.1 sport=25 dport=49554 [ASSURED] mark=254 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 2 flow entries have been shown.