[Openvpn-devel] multihome broken in the presence of asymmetric routing

  Hello,

I have the following multihomed setup:


            BGP1                        BGP2
             ^                           ^
             |                           |
+-----------+-----------+    +----------+-----------+
| IP1.1   IP1.2   IP1.3 |    | IP2.1  IP2.2   IP3.1 |
|                       |    |                      |
|                       |    |                      |
|          RTR1         |    |         RTR2         |
|                       |    |                      |
|                       |    |                      |
|          vIP          |    |         vIP          |
+-----------+-----------+    +----------+-----------+
             |                           |
             |                           |
             |                           |
             |            VRRP           |
             +---------------------------+


I.e. two routers with BGP to multiple ISPs. VRRP running at inner side 
keeping one virtual IP (vIP) up at the master router.

Someday I hope I'll be able to use the vIP only for openvpn server bind 
IP ("local" config option). Untill then, for legacy reasons, I have to 
use the "multihome" config option, so that clients could connect to each 
of the 7 IPs (IP1.1, IP1.2, IP1.3, IP2.1, IP2.2, IP2.3, vIP).

Unfortunately, Linux would not respond to /some/ clients over UDP. It 
seems like this is due to the way "multihome" forces the output 
interface using IP_PKTINFO when we have asymetric path to/from vpn clients.

To provide single-socket UDP multihoming, openvpn uses the IP_PKTINFO 
data from recvmsg() and passes it as-is to sendmsg(). Apparently 
ipi_ifindex behaves in an interesting way under Linux. man 7 ip states:

     If  IP_PKTINFO  is  passed  to  sendmsg(2)  and ipi_spec_dst  is  not
     zero, then it is used as the local source address for the routing table
     lookup and for setting up IP source route options.  When ipi_ifindex is
     not zero, the primary local address of the interface  specified  by
     the  index overwrites ipi_spec_dst for the routing table lookup.


In my tests it's like /ipi_ifindex/ will override any routing table 
decision. But then if there is no routing table entry for the 
destination via
that interface, the destination will be assumed to be on-link and will 
not be routed via a gateway. No error is returned to
userspace, but if the destination does not respond to an ARP request on 
that link, the packet will be silently dropped. That's what I see with 
tcpdump on the openvpn server: arp requests for the dst address of the 
openvpn client. For example if a.b.c.100 is an openvpn client and 
x.y.z.1 is the virtual IP on the openvpn server to which the client 
tries to connect:

11:52:40.695830 ARP, Request who-has a.b.c.100 tell x.y.z.1, length 28

I got it working by patching socket.c in the following manner:



I guess this should be confirmed by someone else too, and I haven't 
looked into IPv6 side at all, but the above patch works for me ™.

Best regards,
Teodor Milkov
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
  

Am 12.03.18 um 11:37 schrieb Teodor Milkov:
> Hello,
> 
> I have the following multihomed setup:
> 
> 
>            BGP1                        BGP2
>             ^                           ^
>             |                           |
> +-----------+-----------+    +----------+-----------+
> | IP1.1   IP1.2   IP1.3 |    | IP2.1  IP2.2   IP3.1 |
> |                       |    |                      |
> |                       |    |                      |
> |          RTR1         |    |         RTR2         |
> |                       |    |                      |
> |                       |    |                      |
> |          vIP          |    |         vIP          |
> +-----------+-----------+    +----------+-----------+
>             |                           |
>             |                           |
>             |                           |
>             |            VRRP           |
>             +---------------------------+
> 
> 
> I.e. two routers with BGP to multiple ISPs. VRRP running at inner side
> keeping one virtual IP (vIP) up at the master router.
> 
> Someday I hope I'll be able to use the vIP only for openvpn server bind
> IP ("local" config option). Untill then, for legacy reasons, I have to
> use the "multihome" config option, so that clients could connect to each
> of the 7 IPs (IP1.1, IP1.2, IP1.3, IP2.1, IP2.2, IP2.3, vIP).
> 
> Unfortunately, Linux would not respond to /some/ clients over UDP. It
> seems like this is due to the way "multihome" forces the output
> interface using IP_PKTINFO when we have asymetric path to/from vpn clients.
> 
> To provide single-socket UDP multihoming, openvpn uses the IP_PKTINFO
> data from recvmsg() and passes it as-is to sendmsg(). Apparently
> ipi_ifindex behaves in an interesting way under Linux. man 7 ip states:
> 
>     If  IP_PKTINFO  is  passed  to  sendmsg(2)  and ipi_spec_dst  is  not
>     zero, then it is used as the local source address for the routing table
>     lookup and for setting up IP source route options.  When ipi_ifindex is
>     not zero, the primary local address of the interface  specified  by
>     the  index overwrites ipi_spec_dst for the routing table lookup.
> 
> 
> In my tests it's like /ipi_ifindex/ will override any routing table
> decision. But then if there is no routing table entry for the
> destination via
> that interface, the destination will be assumed to be on-link and will
> not be routed via a gateway. No error is returned to
> userspace, but if the destination does not respond to an ARP request on
> that link, the packet will be silently dropped. That's what I see with
> tcpdump on the openvpn server: arp requests for the dst address of the
> openvpn client. For example if a.b.c.100 is an openvpn client and
> x.y.z.1 is the virtual IP on the openvpn server to which the client
> tries to connect:
> 
> 11:52:40.695830 ARP, Request who-has a.b.c.100 tell x.y.z.1, length 28
> 
> I got it working by patching socket.c in the following manner:

Sorry for replying so late. I am just going through all open patches.
but with this i am not sure that it does not break the other multihomed
use cases where this behaviour is actually intended and you also only
tested it for your specific use case, so I am a bit cautious to just
include it.

Arne