EdgeRouter - Interface Bonding and Link Aggregation Groups (LAGs)

 Overview


Readers will learn how to set up bonded interfaces on EdgeRouters to connect to EdgeSwitch Link Aggregation Groups (LAGs) using the 802.3ad Link Aggregation Control Protocol (LACP).

warning_25x25.png Warning: The bonding of interfaces on EdgeRouters is achieved through software, not hardware. This means that the traffic entering a bonded interface is NOT eligible for offloading. Bonded interfaces should be configured in scenarios where there is a need for redundancy, not as a way to increase performance. More information about offloading can be found in the Hardware Offloading article.

 

book_25x25.png  Notes & Requirements:

Applicable to EdgeOS 1.9.7 + firmware in all EdgeRouter models. Knowledge of the Command Line Interface (CLI), and basic networking knowledge is required. Please see the Related Articles below for more information and see the attachments for the configurations used in this article.

 

Equipment used in this article:

- EdgeRouter-X (ER-X)

- EdgeSwitch-8-150W (ES-8-150W)

Table of Contents


  1. Network Diagram
  2. Steps: Interface Bonding
  3. Steps: Link Aggregation Groups
  4. Testing & Verification
  5. Related Articles

Network Diagram


Back to Top

The network topology is shown below. The following interfaces are in use on the EdgeRouter and EdgeSwitch.

ER-X

  • eth0 - 192.168.1.1/24 (temporary management)
  • bond0 - 10.0.99.1/24
  • bond0.10 (VIF 10) - 10.0.10.1/24
  • bond0.20 (VIF 20) - 10.0.20.1/24

ES-8

  • 3/1 (tagged) - VLAN10 / VLAN20
  • svi10 - 10.0.10.2/24
  • svi20 - 10.0.20.2/24 


Steps: Interface Bonding


Back to Top

In this example, the EdgeRouter (ER) is running the default configuration without any additions. For the purpose of this article, we will assume that the device is managed through the default IP address (192.168.1.1) on interface eth0.

The interfaces that will be bonded/bundled together are eth1 and eth2. All configuration that would normally apply to these interfaces (IP addressing, firewall rules, QoS policies, etc.) will now have to be configured under the bonded interface (bond0).

 

CLI_circle.png  CLI METHOD: Access the device's CLI. You can do this by using a SSH program such as PuTTY.

1. Enter configuration mode.

configure

2. Delete the existing configuration from the eth1 and eth2 interfaces.

delete interfaces ethernet eth1
delete interfaces ethernet eth2

3. Create the bond0 interfaces and define the hashing policy and bonding mode.

set interfaces bonding bond0 mode 802.3ad
set interfaces bonding bond0 hash-policy layer2

The available hash-policies are:

  • layer2 - use MAC addresses to generate the hash (802.3ad)
  • layer2+3 - combine MAC address and IP address to make hash
  • layer3+4 - combine IP address and port to make hash

4. Define the IP addresses and VIFs on the bond0 interface.

set interfaces bonding bond0 address 10.0.99.1/24
set interfaces bonding bond0 vif 10 address 10.0.10.1/24
set interfaces bonding bond0 vif 20 address 10.0.20.2/24

5. Assign interfaces to the bond group.

set interfaces ethernet eth1 bond-group bond0
set interfaces ethernet eth2 bond-group bond0

6. Commit the changes and save the configuration.

commit ; save ; exit

7. Verify the created interfaces:

show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
bond0 10.0.99.1/24 u/u
bond0.10 10.0.10.1/24 u/u
bond0.20 10.0.20.1/24 u/u

Steps: Link Aggregation Groups


Back to Top

In this example, the ES is running in the default configuration with the addition of SSH management access. The first step is to create the VLANs and VLAN interfaces (SVIs) and associate them with the uplink port (3/1). In EdgeSwitch all of the LAG ports start with 3/x and are created by bundling multiple ethernet interfaces together. In this case, the interfaces that will be associated with the LAG are port 0/1 and 0/2.

CLI_circle.png  CLI METHOD: Access the command line interface (CLI). You can do this by using a program such as PuTTY to connect via SSH, Telnet or the console.

1. Enter privileged mode.

enable

2. Define the network protocol, parameters and VLAN-id for the management VLAN.

network protocol none
network parms 10.0.99.2 255.255.255.0 10.0.99.1
network mgmt_vlan 99

3. Create the VLANs and VLAN-Interfaces (SVIs).

vlan database 
vlan 10,20,99
vlan routing 10 1
vlan routing 20 2
exit
info_i_25x25.png Note: The format is VLAN routing <vlan-id> <interface-id (1-15)>. The interface-id does not need to match the VLAN number and is used to separate the VLAN-Interfaces in the internal switch architecture.

4. Enter configuration mode.

configure

5. Verify that there is no configuration under the 0/1 and 0/2 interfaces.

show running-config interface 0/1
!Current Configuration:
!
interface 0/1
exit

show running-config interface 0/2
!Current Configuration:
!
interface 0/2
exit

6. (Optional) If there is any VLAN configuration under the interfaces, reset it to default by using:

interface 0/1
no vlan pvid
no vlan tagging 1-4093
vlan participation auto 1-4093
vlan participation include 1

interface 0/2
no vlan pvid
no vlan tagging 1-4093
vlan participation auto 1-4093
vlan participation include 1

7. Add the interfaces to the Link Aggregation Group (LAG) using port 3/1.

interface 0/1
addport 3/1
exit

interface 0/2
addport 3/1
exit
info_i_25x25.png Note: You may also use another port number if 3/1 is already in use (up to 3/6 on the ES-8-150W model).

8. Define the port-channel (LAG) load-balancing method and enable the 802.3ad Link Aggregation Control Protocol (LACP).

interface lag 1
 no port-channel static
port-channel load-balance 3
info_i_25x25.png Note: The default port-channel load-balancing method is 1 (source MAC address) and the default protocol is static. The load-balancing method does NOT have to match on both sides, in this example it is configured to match the EdgeRouter bonding hash-policy.

The available load-balancing methods are:

  • 1 - Src MAC, VLAN, EType, incoming port
  • 2 - Dest MAC, VLAN, EType, incoming port
  • 3 - Src/Dest MAC, VLAN, EType, incoming port
  • 4 - Src IP and Src TCP/UDP Port fields
  • 5 - Dest IP and Dest TCP/UDP Port fields
  • 6 - Src/Dest IP and TCP/UDP Port fields
info_i_25x25.png Note: Choosing the optimal load-balancing method is important and unique to each environment. In short, this mechanism decides which of the two links is chosen for each traffic flow. The traffic will NEVER be sent across both links simultaneously, which is why setting up LAGs and interface bonds will not lead to an increase in throughput. Whatever the setup, you should verify that both links are being utilized with the show commands in the Testing & Verification section of this article.

9. Assign the LAG port to the VLANs created above.

The configuration below tags port 3/1 for VLAN10 and VLAN20 (tagging) with VLAN99 as the native VLAN (pvid). Afterwards, unneeded VLANs are excluded from participating on the ports.

interface lag 1
vlan pvid 99
vlan tagging 10,20
vlan participation include 10,20,99
vlan participation exclude 1

10. Associate the SVIs with IP addresses and enable routing.

interface vlan 10
ip address 10.0.10.2 255.255.255.0
routing
exit

interface vlan 20
ip address 10.0.20.2 255.255.255.0
routing
exit

11. Globally enable routing functionality.

ip routing

12. Write the changes to the startup configuration.

write memory 

Testing & Verification


Back to Top

After configuring the ports and VLANs, verify the connections/state using the following commands:

1. The state of the bond0 interface on the ER:

show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
bond0 10.0.99.1/24 u/u
bond0.10 10.0.10.1/24 u/u
bond0.20 10.0.20.1/24 u/u

show interfaces bonding bond0
bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 80:2a:a8:8d:8f:18 brd ff:ff:ff:ff:ff:ff
inet 10.0.99.1/24 brd 10.0.99.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::822a:a8ff:fe8d:8f18/64 scope link
valid_lft forever preferred_lft forever

RX: bytes packets errors dropped overrun mcast
9550 107 0 1 0 40
TX: bytes packets errors dropped carrier collisions
29522 335 0 0 0 0

show interfaces bonding bond0 vif 10
bond0.10@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 80:2a:a8:8d:8f:18 brd ff:ff:ff:ff:ff:ff
inet 10.0.10.1/24 brd 10.0.10.255 scope global bond0.10
valid_lft forever preferred_lft forever
inet6 fe80::822a:a8ff:fe8d:8f18/64 scope link
valid_lft forever preferred_lft forever

RX: bytes packets errors dropped overrun mcast
820 17 0 0 0 0
TX: bytes packets errors dropped carrier collisions
828 12 0 0 0 0

show interfaces bonding bond0 vif 20
bond0.20@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 80:2a:a8:8d:8f:18 brd ff:ff:ff:ff:ff:ff
inet 10.0.20.1/24 brd 10.0.20.255 scope global bond0.20
valid_lft forever preferred_lft forever
inet6 fe80::822a:a8ff:fe8d:8f18/64 scope link
valid_lft forever preferred_lft forever

RX: bytes packets errors dropped overrun mcast
1970 42 0 0 0 0
TX: bytes packets errors dropped carrier collisions
548 6 0 0 0 0

2. The port state and VLANs the LAG interface on the ES:

show interfaces switchport general 
Intf PVID Ingress Acceptable Untagged Tagged Forbidden Dynamic
Filtering Frame Type Vlans Vlans Vlans Vlans
--------- ----- ---------- ---------- --------- --------- --------- ---------
0/1 1 Disabled Admit all 1
0/2 1 Disabled Admit all 1
3/1 99 Disabled Admit all 99 10,20 1

show interfaces switchport 3/1

VLAN Membership Mode: General
General Mode PVID: 99
General Mode Untagged VLANs: 99
General Mode Tagged VLANs: 10,20
General Mode Forbidden VLANs: 1

show ip interface brief
Interface State IP Address IP Mask TYPE Method
---------- ----- --------------- --------------- --------------- ------
vlan 10 Up 10.0.10.2 255.255.255.0 Primary Manual
vlan 20 Up 10.0.20.2 255.255.255.0 Primary Manual

show lacp partner all
Sys System Admin Prt Prt Admin
Intf Pri ID Key Pri Id State
------- ----- ----------------- ----- ----- ----- -----------
0/1 65535 80:2A:A8:8D:8F:18 17 255 1 ACT|AGG|LTO
0/2 65535 80:2A:A8:8D:8F:18 17 255 2 ACT|AGG|LTO

show lacp actor all
Sys Admin Port Admin
Intf Priority Key Priority State
------- -------- ----- -------- -----------
0/1 32768 0 128 ACT|AGG|LTO
0/2 32768 0 128 ACT|AGG|LTO

info_i_25x25.png Note: The LACP output shows that the default system priority of the ES is 32768, whereas the ER is set to 65535 (maximum priority). Lower priority is better, so this means that the ES will become the master in the bundle. 

 

The admin state values are explained in greater detail below:

  • ACT (Active mode) - Actively tries to negotiate port-channels by continuously sending LACP frames (LACPDUs)
  • PSU (Partner mode) - Passively negotiates port-channels by only responding to LACPDUs
  • AGG (Aggregate mode) - The port is bundled under a Link Aggregation Group (LAG)
  • IND (Individual mode) - The port is not bundled and is in a standalone/individual state
  • LTO (Long Timeout) - Less frequent LACP exchanges to maintain the connection
  • STO (Short Timeout) - More frequent LACP exchanges to maintain the connection

More information on LACP can be found in the Command Line Interface (CLI) and Admin Guides.

show port-channel all 
Log. Channel Adm. Mbr Device/ Port Port
Intf Name Min Link Mode Type Ports Timeout Speed Active
------- --------------- ---- ------ ---- ---- ------ ------------- --------- -------
3/1 ch1 1 Up En. Dyn. 0/1 actor/long Auto True
partner/long
0/2 actor/long Auto True
partner/long

show port-channel 3/1
Local Interface................................ 3/1
Channel Name................................... ch1
Link State..................................... Up
Admin Mode..................................... Enabled
Type........................................... Dynamic
Port-channel Min-links......................... 1
Load Balance Option............................ 3
(Src/Dest MAC, VLAN, EType, incoming port)

Mbr Device/ Port Port
Ports Timeout Speed Active
------ ------------- --------- -------
0/1 actor/long Auto True
partner/long
0/2 actor/long Auto True
partner/long 

3. The load-balancing policy and traffic counters on the ER interfaces:

show interfaces bonding bond0 slaves 
Interface        RX: bytes  packets     TX: bytes  packets   
bond0            11056      119         30266      341       
    eth1         2688       21          5086       44        
    eth2         8368       98          25180      297     

show interfaces ethernet eth1
eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond0 state UP group default 
link/ether 80:2a:a8:8d:8f:18 brd ff:ff:ff:ff:ff:ff
    RX:  bytes    packets     errors    dropped    overrun      mcast
          2816         22          0          0          0         22
    TX:  bytes    packets     errors    dropped    carrier collisions
          5086         44          0          0          0          0

show interfaces ethernet eth2
eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond0 state UP group default 
link/ether 80:2a:a8:8d:8f:18 brd ff:ff:ff:ff:ff:ff
    RX:  bytes    packets     errors    dropped    overrun      mcast
          8496         99          0          0         0 32
    TX:  bytes    packets     errors    dropped    carrier collisions
          5086         44          0          0          0          0

Related Articles


Back to Top