Bypass server VPN to access its HTTP port externally

Question

I'm running a home server with FreeBSD 12.1 with openvpn. The openvpn runs strictly on the server - the rest of my home network is non-openvpn.

I'm a noob with webservers, and am fooling around with using the server in that capacity via lighttpd. I set up the webserver for port 8080 (my ISP keeps 80 closed), and set my DD-WRT router to forward incoming port 8080 to server 8080.

What I want is to have my server do double duty as a VPN server and non-VPN webserver.

But my desired goal - to have the webserver accessible via my router's external IP (with ":8080" added) - works only if I stop openvpn. I can access the webserver w/ ovenvpn if I use my VPN IP (with ":8080" added), but that IP always changes when the server or openvpn restarts, so that's not practical.

In my rc.conf I have:

firewall_enable=yes
firewall_nat_enable=yes
gateway_enable=yes

(I'm unsure if the last two are even needed)

My ipfw rules are very basic. I set them up essentially as "kill-switches" for my torrent and NZB traffic if openvpn stops:

00001 allow ip from any to any via lo0
00010 allow ip from any to any via tun0
00101 allow ip from me to 192.168.1.0/24 uid transmission
00102 allow ip from 192.168.1.0/24 to me uid transmission
00103 deny ip from any to any uid transmission
00104 allow ip from me to 192.168.1.0/24 uid sabnzbd
00105 allow ip from 192.168.1.0/24 to me uid sabnzbd
00106 deny ip from any to any uid sabnzbd

Here's my ifconfig:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81049b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,VLAN_HWFILTER>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.1.250 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::xxxx:xxxx:xxxx:xxxx%tun0 prefixlen 64 scopeid 0x3
        inet xx.xxx.xx.xx --> xx.xxx.xx.xx netmask 0xffffffe0
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 4074

Is there any way this can work? Or would I be forced to have a separate webserver?

---

I followed Claus Andersen's instructions step-by-step, but was ultimately unsuccessful:

Stop both firewall and vpn. OK
Make sure that you can connect to your webserver from your localhost on port 8080. successful
Then from another machine on the same subnet. successful
And finally from the Internet. successful  (verified via https://validator.w3.org/)

How is the webserver bound to IP?
I can access it internally via local address (192.168.1.250:8080) or externally & internally via my router's public address (64.67.136.70:8080).

Next step would be to enable the firewall.
I enabled the firewall with the settings from my original post. All connections were still OK.

Finally enable the vpn.
I enabled it with original firewall settings, and all connections were OK except for the external public address (which I could still connect internally). Now I could also connect externally via my VPN address (178.73.218.69:8080).

I then added your suggested firewall rule ("00100 allow tcp from any to any 8080 in via em0") and restarted firewall. There was no change in connections.

Here's my ifconfig with no redactions:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81049b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,VLAN_HWFILTER>
        ether 34:17:eb:d1:30:df
        inet 192.168.1.250 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::3617:ebff:fed1:30df%tun0 prefixlen 64 scopeid 0x3
        inet 178.73.218.69 --> 178.73.218.65 netmask 0xffffffe0
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 4056

And here's the output from "netstat -4rn" with vpn and latest firewall in place:

Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          178.73.218.65      UGS        tun0
default            192.168.1.1        UGS         em0
127.0.0.1          link#2             UH          lo0
128.0.0.0/1        178.73.218.65      UGS        tun0
178.73.195.104/32  192.168.1.1        UGS         em0
178.73.218.64/27   178.73.218.65      UGS        tun0
178.73.218.65      link#3             UH         tun0
178.73.218.69      link#3             UHS         lo0
192.168.1.0/24     link#1             U           em0
192.168.1.250      link#1             UHS         lo0

Any further suggestions would be appreciated.

Answer 1

Yes - it can work. You do however not give enough information to fully answer and you have redacted parts which would help as well.

tl;dr Simplify & verify

You have a "complex" network setup. That is: you have one common network segment for em0 but then you have your vpn segment on tun0. All this is normal but you need to understand how you route between those. Maybe traffic is routed in on em0 and returned on tun0. Traffic from a http client on the Internet might then pass through to your server. But if the server routes all return traffic through tun0 the http client will never get any replies.

To examine that you need to look at and understand your routing table. Check netstat -4rn.

But even from the partial information you give then I would guess you should look at the ipfw settings. The ruleset you show boils down to:

Rule 1:       Allow everything on lo0
Rule 10:      Allow everything on tun0
Rule 101-103: Transmission related
Rule 104-106: sabnzbd related

My question to you is then: If your DD-WRT route forwards packets from port 8080 to your server on port 8080 - what do you expect to happen to them?

You have no rule in your firewall which handles this case. I you then expect the firewall to drop the packet. I use pf myself so I am not 100% sure of ipfw usage but it will be something along the lines of:

00100 allow tcp from any to any 8080 in via em0

If this works you might reduce any to any to any to me and you might choose to do it based on uid instead.

But you say it works when openvpn it not running. That makes me expect that your firewall is misconfigured and allows all traffic as you have no seperate rule to allow traffic for port 8080.

But yes - such a setup can be made to work. There are however a lot of moving parts. I would debug it like this.

1. Stop both firewall and vpn. Make sure that you can connect to your webserver from your localhost on port 8080. Then from another machine on the same subnet. And finally from the Internet. This is the first step to ensure the webserver is configured and working. You have tested via the vpn - so the configuration is most likely working. But how is the webserver bound to IP? With "*", "localhost", local address or public address.

2. Next step would be to enable the firewall. With the firewall enabled you can check that your rules work as expected. I think the issue in your case lies here.

3. Finally enable the vpn. This can give interesting routing issues. If the webserver becomes unreachable we need to exmaine the route. This is where we start looking at netstart -4rn.

So test each step fully with as few moving parts as possible and then progress one step at a time.