info_i_25x25.png See important information about Ubiquiti Devices and KRACK Vulnerability in this article. We will update this document as more information becomes available.

EdgeRouter - WAN load-balance and Tunnels

In this Knowledge Base article we'll cover some of the special considerations that are necessary with combining WAN load-balancing and tunnels.  For simplicity this example will use a GRE tunnnel (easier debugging with packet capture on non-encyrpted data), but if theory should work for other tunnels.

Our topology:

 

Router R1                                                                         Router R2

 

eth0      WAN1   20.0.0.2/30                                             eth0  WAN  30.0.0.2/30

eth1      LAN      192.168.1.1/24                                        eth1  LAN   172.16.1.1/24

pppoe0 WAN2

tun0      GRE     40.0.01/30                                               tun0   GRE  40.0.0.2/30

We'll start with the GRE tunnel, then add load-balancing after.

We want our GRE tunnel to go from R1 eth0 to R2 eth0.  So for R1 we add:

 tunnel tun0 {
     address 40.0.0.1/30
     encapsulation gre
     local-ip 20.0.0.2
     remote-ip 30.0.0.2
 } 

For R2:

 tunnel tun0 {
     address 40.0.0.2/30
     encapsulation gre
     local-ip 30.0.0.2
     remote-ip 20.0.0.2
 }

 

Then we need to open the firewall for protocol GRE for WAN_LOCAL:

 

ubnt@R1# show firewall name WAN_LOCAL rule 40
 action accept
 description "Allow GRE"
 protocol gre
 source {
     address 30.0.0.2
 }

 

Then we'll likely need TCP mss-clamp:

 

ubnt@R1# show firewall options 
 mss-clamp {
     interface-type tun
     interface-type pppoe
     mss 1412
 }

 

Last we need to add a static route to tell the system that R2's LAN network should be sent out tun0 interface.

     interface-route 172.16.1.0/24 {
         next-hop-interface tun0 {
         }
     }

At this point our GRE tunnel should work, but

ubnt@R1:~$ ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.832 ms
64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.741 ms
64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.635 ms
64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.654 ms
64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.644 ms
64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.614 ms
^C
--- 172.16.1.1 ping statistics ---
8 packets transmitted, 6 received, 25% packet loss, time 7006ms
rtt min/avg/max/mdev = 0.614/0.686/0.832/0.082 ms 

So it kind of works, but 25% packet loss is not good.  One theory is that since we have 2 Internet connection, that some packets are going out the wrong interface.  To test that we'll take down the 2nd WAN interface.

ubnt@R1:~$ disconnect interface pppoe0 
Bringing interface pppoe0 down...
ubnt@R1:~$ 
ubnt@R1:~$ 
ubnt@R1:~$ ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.789 ms
64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.606 ms
64 bytes from 172.16.1.1: icmp_req=3 ttl=64 time=0.585 ms
64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.667 ms
64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.650 ms
64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.707 ms
64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.752 ms
64 bytes from 172.16.1.1: icmp_req=8 ttl=64 time=0.613 ms
^C
--- 172.16.1.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7007ms
rtt min/avg/max/mdev = 0.585/0.671/0.789/0.069 ms

Problem solved! But how do we use the 2nd WAN. One way is to add a static route for the far GRE end-point and force it to go out eth0.

     route 30.0.0.2/32 {
         next-hop 20.0.0.1 {
         }
     }

Now we can bring back up the pppoe0 interface and still the GRE tunnel works fine.

The full configurations for router R1 and R2 for this part of the KB are:

R1 /config/config.boot GRE

R2 /config/config.boot GRE

 

Load-Balancing

 

Now we'll add WAN load-balance to R1.  Start with the load-balance section:

 

load-balance {
    group WLB {
        interface eth0 {
        }
        interface pppoe0 {
        }
    }
}

This is all we really need, but we're going to change the ping target to an address so that the health check doesn't need DNS.

load-balance {
    group WLB {
        interface eth0 {
            route-test {
                type {
                    ping {
                        target 8.8.8.8
                    }
                }
            }
        }
        interface pppoe0 {
            route-test {
                type {
                    ping {
                        target 8.8.8.8
                    }
                }
            }
        }
    }
}

 Then we'll add a firewall modify rule

    
   modify BALANCE {
        rule 10 {
            action modify
            description "Do not load-balance LAN to LAN traffic"
            destination {
                address 192.168.1.0/24
            }
            modify {
                table main
            }
        }
        rule 20 {
            action modify
            description "load-balance the rest of LAN to WAN traffic"
            modify {
                lb-group WLB
            }
        }
    }

Then apply the modify rule to the LAN "in".

    ethernet eth1 {
        address 192.168.1.1/24
        description LAN
        firewall {
            in {
                modify BALANCE
            }
        }
    }

 At this point our LAN traffic is being balanced over both WAN's fairly evenly:

 

ubnt@R1:~$ show load-balance status 
Group WLB
  interface   : eth0
  carrier     : up
  status      : active
  gateway     : 20.0.0.1
  weight      : 50
  flows
      WAN Out : 682
      WAN In  : 0
    Local Out : 20

  interface   : pppoe0
  carrier     : up
  status      : active
  gateway     : pppoe0
  weight      : 50
  flows
      WAN Out : 672
      WAN In  : 0
    Local Out : 22

 

But our GRE tunnel now doesn't work.  The first problem is that we really don't want to load-balance the LAN traffic that is supposed to go over the GRE tunnel.  So we'll add a rule before the lb-group rule like:

 

ubnt@R1# show firewall modify 
 modify BALANCE {
     rule 10 {
         action modify
         description "Do not load-balance LAN to LAN traffic"
         destination {
             address 192.168.1.0/24
         }
         modify {
             table main
         }
     }
     rule 20 {
         action modify
         description "Do not load-balance traffic for gre tunnel"
         destination {
             address 172.16.1.0/24
         }
         modify {
             table main
         }
     }
     rule 30 {
         action modify
         description "load-balance the rest of LAN to WAN traffic"
         modify {
             lb-group WLB
         }
     }
 }

 This is mostly working now, but sometimes I need to take down the pppoe interface for the GRE tunnel to come up correctly.  The next issue is related to how the load-balance feature is currently implemented with a separate routing table per WAN interface that only has a default route for that WAN.  This works fine for LAN to WAN sessions, but breaks when we need a connected route from the "main" routing table.  So the current work around is to instead of using the system generated separate routing table, instead create your own that has the default route for that WAN and the connected routes that are needed.

 

ubnt@R1# show protocols static table 
 table 1 {
     interface-route 20.0.0.0/30 {
         next-hop-interface eth0 {
         }
     }
     interface-route 192.168.1.0/24 {
         next-hop-interface eth1 {
         }
     }
     route 0.0.0.0/0 {
         next-hop 20.0.0.1 {
         }
     }
 }
 table 2 {
     interface-route 0.0.0.0/0 {
         next-hop-interface pppoe0 {
         }
     }
     interface-route 20.0.0.0/30 {
         next-hop-interface eth0 {
         }
     }
     interface-route 192.168.1.0/24 {
         next-hop-interface eth1 {
         }
     }
 }

 

And then change the load-balance config to use the over-ride routing table.

ubnt@R1# show load-balance           
 group WLB {
     interface eth0 {
         route {
             table 1
         }
         route-test {
             type {
                 ping {
                     target 8.8.8.8
                 }
             }
         }
     }
     interface pppoe0 {
         route {
             table 2
         }
         route-test {
             type {
                 ping {
                     target 8.8.8.8
                 }
             }
         }
     }
 }

The full config file for R1 with GRE and WAN load-balance is:

 

R1 /config/config.boot GRE with WAN load-balance

 

Below is the example config files for an IPSec tunnel between R1 and R2 while R1 has WAN load-balancing configured.

 

R1 /config/config.boot IPSec to R2 with WAN load-balance

R2 /config/config.boot IPSec to R1