How does SSLH's transparent mode work?

Question

I have a server that needs to accept TCP connections on port 443, figure out if a connection is an OpenVPN client or an HTTPS client, and either forward it to my web server or my OpenVPN server.

SSLH was designed specifically for this reason, and seems to work great. The only problem is that while setting it up in non-transparent mode works fine for me, using transparent mode is giving me trouble.

Can someone explain the theory behind how transparent mode is supposed to work?

The SSLH guide says that in order to use SSLH transparently, you need to:

• Set sslh.cfg to transparent: true;
  • I've done this
• sslh needs extended rights (CAP_NET_ADMIN)
  • I installed sslh from my CentOS 7's repos, which came with sslh.service for systemd. That service file contains the line CapabilityBoundingSet=... CAP_NET_ADMIN ..., so I assume this is already done by SystemD
• Setup iptables rules that mark packets, and some sort of local route
  • This is what I'm unclear about. Do these get setup on the SSLH server? Or do they get setup on the OpenVPN and HTTPS servers?

I get that in the example, iptables is told to mark any packets with a source port of 22 or 4443 with a 0x1 marking, a rule is created to have any packets marked with 0x1 use routing table 100, and routing table 100 is created that sets up some sort of local route that does something.

Why are these iptables rules and the route necessary? What is the route actually doing? I would think that the route should be on the web server and OpenVPN server, and point to the SSLH ip, but that doesn't seem to work for me either.

===

Update: It just occurred to me that it's probably a route that points back to localhost because in the example, the servers are all on the same machine, and they want the reply packets coming from the actual servers to go through SSLH before they leave the machine. Does that sound right? If so, what would I do in my case if my servers are on different machines? Setup traffic marking and a route back to the SSLH server on those machines?

Update 2: I just quickly setup the HTTPS server on the same box as the SSLH, and transparent mode does seem to work exactly the way it does in the example of SSLH's documentation. I need it to work when they are on different servers.

Answer 1

In non-transparent mode the client connects with client_ip:client_port to the sslh proxy on port 443. Then the sslh proxy opens a connection with sslh_ip:sslh_port to the internal server on port 4443. The internal web server will answer from web_ip:4443 to the proxy_ip:proxy_port. And finally the sslh proxy rewrites the source of the answer packet to sshl_ip:443 and sends it to the client.

In transparent mode for the connection between sslh proxy and internal server the source of the packets is set to the original client_ip:client_port. So the internal web server answers direct to client_ip:client_port with web_ip:4443 as source. But the client waits on packets form proxy_ip:443.

To manage the rewrite of answer packages from the internal server these packages must be routed through the sslh daemon. I think the daemon looking for answer packages to the client on the loopback interface. In the documentation only the case with sslh proxy and web internal server on the same machine is prescribed.

With try and error, I found the following solution: (on the sslh proxy)

ip route add local default dev lo table 100
ip rule add fwmark 0x1 lookup 100
iptables -t mangle -N SSLH
iptables -t mangle -A SSLH -j MARK --set-mark 0x1
iptables -t mangle -A SSLH -j ACCEPT

(to catch the answer packages from the internal server)

iptables -t mangle -A PREROUTING -p tcp -s **web_ip** --sport 4443 -j SSLH

... (similar rules for the other services e.g. ssh, openvpn) ...

This works only when the default route of the internal web server goes over the sshl proxy server to the internet.

Answer 2

Thanks Norbert - you've explained the general idea nicely. So to put that all together, as far as I can tell, this is what's happening specifically with all the commands we are running:

SSLH Server:

$ sudo iptables -t mangle -N SSLH
$ sudo iptables -t mangle -A PREROUTING -p tcp -m socket --transparent -j SSLH
$ sudo iptables -t mangle -A SSLH -j MARK --set-mark 0x1 
$ sudo iptables -t mangle -A SSLH -j ACCEPT
$ sudo ip rule add fwmark 0x1 lookup 100 
$ sudo ip route add local 0.0.0.0/0 dev lo table 100

Internal Web Server:

$ sudo iptables -t mangle -N SSLH
$ sudo iptables -t mangle -A OUTPUT -o eth0 -p tcp -m tcp --sport 4443 -j SSLH
$ sudo iptables -t mangle -A SSLH -j MARK --set-mark 0x1 
$ sudo iptables -t mangle -A SSLH -j ACCEPT
$ sudo ip rule add fwmark 0x1 lookup 100 
$ sudo ip route add default via [SSLH_IP] table 100 

On the internal web server:

• a web server (like apache or nginx) is listening on port 4443
• anything going out eth0 with a source port of 4443 gets sent to the SSLH chain
• anything in the SSLH chain gets a mark of 0x1 set on it
• a routing rule is created that says that anything marked as 0x1 should use routing table 100, instead of the main routing table
• routing table 100 has an entry that says to send traffic to the SSLH server, instead of the default gateway

This sequence of events basically forces traffic from the web server's process (nginx/apache/whatever) to be routed to the SSLH server, instead of the default gateway like the rest of the system's traffic.

On the SSLH server:

• anything coming from the web server (source port 4443) gets sent to the SSLH chain
• anything in the SSLH chain gets a mark of 0x1 set on it
• a routing rule is created that says that anything marked as 0x1 should use routing table 100, instead of the main routing table
• routing table 100 has an entry that says to force all traffic using this routing table to get redirected to the loopback interface

This sequence of events basically forces traffic from the internal web server to be redirected to the loopback interface on the SSLH server, where the SSLH process will rewrite its source IP and port and send it out.

Questions I still have:

• does that sound right?
• what does iptables -t mangle -A SSLH -j ACCEPT do? Is it necessary? This isn't the FILTER table - why are we accepting traffic here?
• in my case, the iptables prerouting rule on the SSLH host that works looks different than Norbert's. How does that work? How does it know if something is --transparent or not?

Answer 3

I followed the steps of the documentation on GitHub (https://github.com/yrutschle/sslh/blob/master/doc/config.md), using --transparent parameter in the file /etc/default/sslh.

Example if Apache is listening on port 4443 instead of 443 and if we are using OpenVPN also:

DAEMON_OPTS="--user sslh --transparent --listen 0.0.0.0:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:4443 --openvpn 127.0.0.1:1194 --pidfile /var/run/sslh/sslh.pid"

Note that you may need to replace 127.0.0.1:1194 by your VPN IP address followed by :1194 (https://github.com/yrutschle/sslh/issues/83#issuecomment-515675186).

If you want IPv6 support, you can replace 0.0.0.0 by a "dual-stack" domain name pointing to your remote IPv4 / IPv6 and replace 127.0.0.1 by localhost which should point to your local IPv4 / IPv6 if it is configured correctly in /etc/hosts.

To make it permanent after reboot on Debian, I had to add these post-up rules for the local interface in /etc/network/interfaces below the existing configuration:

# The loopback network interface
auto lo
iface lo inet loopback
# Configure routing for those marked packets (required by sslh transparent mode for IPv4)
        post-up ip rule add fwmark 0x1 lookup 100
        post-up ip route add local 0.0.0.0/0 dev lo table 100
iface lo inet6 loopback
# Configure routing for those marked packets (required by sslh transparent mode for IPv6)
        post-up ip -6 rule add fwmark 0x1 lookup 100
        post-up ip -6 route add local ::/0 dev lo table 100

And these rules in /etc/sysctl.conf:

# Set route_localnet = 1 on all interfaces so that ssl can use "localhost" as destination (required by sslh transparent mode)
net.ipv4.conf.default.route_localnet=1
net.ipv4.conf.all.route_localnet=1

In the documentation, there are also some iptables rules that can be made permanent with iptables-persistent for example (I don't know why but the transparent mode seemed to work without these rules when I tested):

# DROP martian packets as they would have been if route_localnet was zero
# Note: packets not leaving the server aren't affected by this, thus sslh will still work
iptables -t raw -A PREROUTING ! -i lo -d 127.0.0.0/8 -j DROP
iptables -t mangle -A POSTROUTING ! -o lo -s 127.0.0.0/8 -j DROP

# Mark all connections made by ssl for special treatment (here sslh is run as user "sslh")
iptables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f

# Outgoing packets that should go to sslh instead have to be rerouted, so mark them accordingly (copying over the connection mark)
iptables -t mangle -A OUTPUT ! -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f

For IPv6:

# DROP martian packets as they would have been if route_localnet was zero
# Note: packets not leaving the server aren't affected by this, thus sslh will still work
ip6tables -t raw -A PREROUTING ! -i lo -d ::1/128 -j DROP
ip6tables -t mangle -A POSTROUTING ! -o lo -s ::1/128 -j DROP

# Mark all connections made by ssl for special treatment (here sslh is run as user "sslh")
ip6tables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f

# Outgoing packets that should go to sslh instead have to be rerouted, so mark them accordingly (copying over the connection mark)
ip6tables -t mangle -A OUTPUT ! -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f