The problem is a missing default route in table main. This can't be completely emulated when checking with ip route get ... mark.

Because there's no default route, a route lookup to 8.8.8.8 fails with "Network is unreachable" before the packet has a chance to reach the mangle OUTPUT chain at all giving:

• packet is generated,
• route is checked in the ip rule table (routing decision in Packet flow in Netfilter and General Networking),
• "Network is unreachable". END.

Adding any default route in table main (even using a non-existing router as long as it's a valid syntax) would allow the intended flow:

• packet is generated,
• route is checked a first time in the ip rule table, deemed existing,
• traverses the mangle OUTPUT chain, inherits the mark,
• gets rerouted (reroute check in Packet flow in Netfilter and General Networking) and thus triggers for a second time a lookup in the ip rule table,
• grabs the early lookup on table 100 and gets its final route "via 192.168.0.1 dev wlp2s0".

The problem is a missing default route in table main. This can't be detected when checking with ip route get ... mark ... because the whole path is short-circuited with the "solution" handed directly.

Because there's no default route, a route lookup to 8.8.8.8 fails with "Network is unreachable" before the packet has a chance to reach the mangle OUTPUT chain at all giving:

• packet is generated,
• route is checked in the ip rule table (routing decision in Packet flow in Netfilter and General Networking),
• "Network is unreachable". END.

Adding any default route in table main (even using a non-existing router as long as it's a valid syntax) would allow the intended flow:

• packet is generated,
• route is checked a first time in the ip rule table, deemed existing,
• traverses the mangle OUTPUT chain, inherits the mark,
• gets rerouted (reroute check in Packet flow in Netfilter and General Networking) and thus triggers for a second time a lookup in the ip rule table,
• grabs the early lookup on table 100 and gets its final route "via 192.168.0.1 dev wlp2s0".

So, picking in the LAN an IP that doesn't belong to any host, let's say 192.168.0.250 and adding it in the script, thus giving:

ip route add table 100 default via 192.168.0.1 dev wlp2s0
ip rule add fwmark 0x2 table 100
ip route flush table main
ip route add 192.168.0.0/24 dev wlp2s0
ip route add default via 192.168.0.250 dev wlp2s0

will solve the test made in the question:

• any connection not made with destination tcp port 501 will trigger ARP requests to 192.168.0.250, which should time out 3 seconds later with the message "No route to host" (instead of "Network is unreachable")
• any connection made with destination tcp port 501 will (resolve the ARP request and) go via 192.168.0.1

The problem is a missing default route in table main.

Because there's no default route, a route lookup to 8.8.8.8 fails with "Network is unreachable". The packet is thus not actually generated, so doesn't go through the mangle OUTPUT chain at all.

Adding any default route in table main (even using a non-existing router as long as it's a valid syntax) would allow the intended flow:

• route is checked a first time in the ip rule table, deemed existing,
• packet is generated,
• traverses the mangle OUTPUT chain, inherits the mark,
• gets rerouted (as seen in Packet flow in Netfilter and General Networking) and thus triggers for a second time a lookup in the ip rule table,
• grabs the early lookup on table 100 and gets its final route "via 192.168.0.1 dev wlp2s0".

The problem is a missing default route in table main. This can't be completely emulated when checking with ip route get ... mark.

Because there's no default route, a route lookup to 8.8.8.8 fails with "Network is unreachable" before the packet has a chance to reach the mangle OUTPUT chain at all giving:

• packet is generated,
• route is checked in the ip rule table (routing decision in Packet flow in Netfilter and General Networking),
• "Network is unreachable". END.

Adding any default route in table main (even using a non-existing router as long as it's a valid syntax) would allow the intended flow:

• packet is generated,
• route is checked a first time in the ip rule table, deemed existing,
• traverses the mangle OUTPUT chain, inherits the mark,
• gets rerouted (reroute check in Packet flow in Netfilter and General Networking) and thus triggers for a second time a lookup in the ip rule table,
• grabs the early lookup on table 100 and gets its final route "via 192.168.0.1 dev wlp2s0".