EdgeRouter - Policy-based routing with WAN load-balancing

Overview


Note: this method of WAN load-balance an example before we had the load-balance feature (LINK).  The method below works, but the modify mark  will disable offload while the load-balance feature does not.

 

Readers will learn the how to perform WAN load balancing for routers with multiple ports designated WAN.

The example uses packet/connection marks to do dual WAN Load Balancing. In the previous Policy Based Routing examples the routing table decision was based on fixed source or destination port. For this wiki page we want to try to balance the new connections 50/50 over ISP1 and ISP2.

In the diagram below, we have 2 Internet connections, ISP-1 and ISP-2:

  1. ISP-1 on eth0 192.0.2.0/24
  2. ISP-2 on eth1 203.0.113.0/24

On the LAN side:

  1. eth2 172.16.0.0/24

Network Diagram


Routing Tables


We're using the same routing tables as from EdgeOS Policy Based Routing where the "main" routing table has a default route to both ISP's, table 1 only has a route to ISP1 and table 2 only has a route to ISP2, but with a simple addition. Here we're also associating a table with a packet mark - mark 1 for table 1 and mark 2 for table 2. This tells us to use table 1 if the packet has a mark of 1 and 2 when there's a mark of 2.

set protocols static route 0.0.0.0/0 next-hop 192.0.2.1
set protocols static route 0.0.0.0/0 next-hop 203.0.113.1
set protocols static table 1 mark 1
set protocols static table 1 route 0.0.0.0/0 next-hop 192.0.2.1
set protocols static table 2 mark 2
set protocols static table 2 route 0.0.0.0/0 next-hop 203.0.113.1

Load Balancing


When stateful firewall is in use, the system keeps track of all the connections in the system. Both the connection entry and the packet can have a mark applied to them and marks can be save/restored between the connection and the packet. Basically what we're trying to do here is for a new connection 50% of the time we'll mark the packet with mark 1 and the rest of the time we'll mark it with 2. Then we'll save that packet mark to the new connection entry. For existing connections there should already be a mark, so we'll restore that mark to the packet.

 set firewall modify balance rule 10 action modify
 set firewall modify balance rule 10 description 'restore mark from connection'
 set firewall modify balance rule 10 modify connmark restore-mark
 set firewall modify balance rule 20 action accept
 set firewall modify balance rule 20 description 'accept the packet if the mark isnt zero'
 set firewall modify balance rule 20 mark '!0'
 set firewall modify balance rule 30 action modify
 set firewall modify balance rule 30 description 'for new connections mark 50% with mark 1'
 set firewall modify balance rule 30 modify mark 1
 set firewall modify balance rule 30 protocol tcp_udp
 set firewall modify balance rule 30 state new enable
 set firewall modify balance rule 30 statistic probability 50%
 set firewall modify balance rule 40 action modify
 set firewall modify balance rule 40 description 'for packets with mark zero, mark with 2'
 set firewall modify balance rule 40 mark 0
 set firewall modify balance rule 40 modify mark 2
 set firewall modify balance rule 40 protocol tcp_udp
 set firewall modify balance rule 40 state new enable
 set firewall modify balance rule 50 action modify
 set firewall modify balance rule 50 description 'save the packet mark to the connection mark'
 set firewall modify balance rule 50 modify connmark save-mark

We then need to apply it to our LAN interface.

set interfaces ethernet eth2 firewall in modify balance
 

Testing


Here we can see that rules 30 and 40 are nearly balanced 50/50:

ubnt@PBR:~$ show firewall modify statistics
 --------------------------------------------------------------------------------
 IPv4 Firewall "balance"
 Active on (eth2,IN)
 rule packets bytes action description
 ---- ------- ----- ------ -----------
 10 399516 61839166 MODIFY restore mark from connection
 20 366897 59641481 ACCEPT accept the packet if the mark isn't zero
 30 16196 1094021 MODIFY for new connections mark 50% with mark 1
 40 16377 1101667 MODIFY for packets with mark zero, mark with 2
 50 32619 2197685 MODIFY save the packet mark to the connection mark
 10000 32619 2197685 ACCEPT DEFAULT ACTION

New WAN Connections


If we have port forwarding rules configured, then we may have new connections initiated from the WAN side. In order to keep those new connections going out the same interface they came in, we'll mark a 1 on new connections from ISP1 and mark 2 on new connections from ISP2.

 set firewall modify ISP1_IN rule 1 description 'use mark 1 for new ISP1 connections'
 set firewall modify ISP1_IN rule 1 action modify
 set firewall modify ISP1_IN rule 1 log enable
 set firewall modify ISP1_IN rule 1 modify connmark set-mark 1
 set firewall modify ISP1_IN rule 1 protocol tcp_udp
 set firewall modify ISP1_IN rule 1 state new enable
 set firewall modify ISP2_IN rule 1 description 'use mark 2 for new ISP2 connections'
 set firewall modify ISP2_IN rule 1 action modify
 set firewall modify ISP2_IN rule 1 log enable
 set firewall modify ISP2_IN rule 1 modify connmark set-mark 2
 set firewall modify ISP2_IN rule 1 protocol tcp_udp
 set firewall modify ISP2_IN rule 1 state new enable

Apply to WAN interfaces:

set interfaces ethernet eth0 firewall in modify ISP1_IN
set interfaces ethernet eth1 firewall in modify ISP2_IN
 Example Configuration
 firewall {
 modify ISP1_IN {
 rule 1 {
 action modify
 description "use mark 1 for new ISP1 connections"
 log enable
 modify {
 connmark {
 set-mark 1
 }
 }
 protocol tcp_udp
 state {
 new enable
 }
 }
 }
 modify ISP2_IN {
 rule 1 {
 action modify
 description "use mark 2 for new ISP2 connections"
 log enable
 modify {
 connmark {
 set-mark 2
 }
 }
 protocol tcp_udp
 state {
 new enable
 }
 }
 }
 modify balance {
 enable-default-log
 rule 10 {
 action modify
 description "restore mark from connection"
 modify {
 connmark {
 restore-mark
 }
 }
 }
 rule 20 {
 action accept
 description "accept the packet if the mark isn't zero"
 mark !0
 }
 rule 30 {
 action modify
 description "for new connections mark 50% with mark 1"
 modify {
 mark 1
 }
 protocol tcp_udp
 state {
 new enable
 }
 statistic {
 probability 50%
 }
 }
 rule 40 {
 action modify
 description "for packets with mark zero, mark with 2"
 mark 0
 modify {
 mark 2
 }
 protocol tcp_udp
 state {
 new enable
 }
 }
 rule 50 {
 action modify
 description "save the packet mark to the connection mark"
 modify {
 connmark {
 save-mark
 }
 }
 }
 }
 }
 interfaces {
 ethernet eth0 {
 address 192.0.2.2/24
 duplex auto
 firewall {
 in {
 modify ISP1_IN
 }
 }
 speed auto
 }
 ethernet eth1 {
 address 203.0.113.2/24
 duplex auto
 firewall {
 in {
 modify ISP2_IN
 }
 }
 speed auto
 }
 ethernet eth2 {
 address 172.16.0.1/24
 duplex auto
 firewall {
 in {
 modify balance
 }
 }
 speed auto
 }
 loopback lo {
 }
 }
 protocols {
 static {
 route 0.0.0.0/0 {
 next-hop 192.0.2.1 {
 }
 next-hop 203.0.113.1 {
 }
 }
 table 1 {
 mark 1
 route 0.0.0.0/0 {
 next-hop 192.0.2.1 {
 }
 }
 }
 table 2 {
 mark 2
 route 0.0.0.0/0 {
 next-hop 203.0.113.1 {
 }
 }
 }
 }
 }
 service {
 dhcp-server {
 disabled false
 shared-network-name LAN {
 authoritative disable
 subnet 172.16.0.0/24 {
 default-router 172.16.0.1
 dns-server 8.8.8.8
 lease 86400
 start 172.16.0.10 {
 stop 172.16.0.100
 }
 }
 }
 }
 gui {
 https-port 443
 }
 nat {
 rule 5000 {
 outbound-interface eth0
 type masquerade
 }
 rule 6000 {
 outbound-interface eth1
 type masquerade
 }
 }
 ssh {
 port 22
 protocol-version v2
 }
 }
 system {
 host-name WLB
 login {
 user ubnt {
 authentication {
 encrypted-password $1$zKNoUbAo$gomzUbYvgyUMcD436Wo66.
 }
 level admin
 }
 }
 name-server 8.8.8.8
 ntp {
 server 0.ubnt.pool.ntp.org {
 }
 server 1.ubnt.pool.ntp.org {
 }
 server 2.ubnt.pool.ntp.org {
 }
 server 3.ubnt.pool.ntp.org {
 }
 }
 syslog {
 global {
 facility all {
 level notice
 }
 facility protocols {
 level debug
 }
 }
 }
 time-zone UTC
 }
 /* Warning: Do not remove the following line. */
 /* === vyatta-config-version: "config-management@1:dhcp-relay@1:dhcp-server@4:firewall@4:ipsec@3:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
 /* Release version: v1.1.0beta3dev.4539852.130227.0102 */

Related Articles