EdgeRouter - Hardware Offloading


Overview


This article explains the function, benefits, and implementation of hardware offloading. As of EdgeOS firmware version v1.9.1, all EdgeRouter models support some type of hardware offloading.

NOTES & REQUIREMENTS: 
Applicable to the latest EdgeOS firmware on all EdgeRouter models. Please see the Related Articles below for more information.

Table of Contents


  1. What is Hardware Offloading?
  2. EdgeRouter Features Eligible for Offloading
  3. IPsec Offloading Support
  4. How to Enable/Disable Offloading
  5. Optional Offloading Optimizations and Testing
  6. Related Articles

What is Hardware Offloading?


Back to Top

Offloading is used to execute functions of the router using the hardware directly, instead of a process of software functions. The benefit of offloading in EdgeOS is increased performance and throughput by not depending on the CPU for forwarding decisions.

There are many processes/features that can take advantage of the offloading engine. One of the most basic examples is IPv4 traffic forwarding. Without offloading enabled, IPv4 traffic will be routed via the CPU and will be limited to around 300Mbps on the EdgeRouter Lite (ERLite-3). With offloading enabled, the throughput will be about 950Mbps.


EdgeRouter Features Eligible for Offloading


Back to Top

Some processes currently cannot take advantage of hardware offloading. This can lead to confusion where it is assumed that offloading is disabled. A more accurate statement is that some features are not eligible to be offloaded and will always depend on the CPU.

There are two platforms that are used by different EdgeRouter models. Each platform has its own offloading support and unique commands to enable the functionality. The platforms are:


MediaTek


Used by the following EdgeRouter models:

  • ER-X
  • ER-10X
  • ER-X-SFP
  • EP-R6

Cavium


Used by the following EdgeRouter models:

  • ERLite-3
  • ERPoE-5
  • ER-8
  • ERPro-8
  • EP-R8
  • ER-4
  • ER-6P
  • ER-12
  • ER-12P
  • ER-8-XG

The tables below summarizes the features that can be offloaded on each platform.

MediaTek-Based Devices
Feature Offload Eligible
Command
Bridging  hwnat
DPI  hwnat
NAT  hwnat
VLANs  hwnat
GRE  hwnat
PPPoE  hwnat
IPsec  ipsec
QoS  - 
NetFlow  -
Bonding  -
Cavium-Based Devices
Feature Offload Eligible Command
Bridging  -
DPI  ipv4 forwarding
NAT  ipv4 forwarding
VLANs  ipv4 / ipv6 vlan
GRE  ipv4 gre 
PPPoE  ipv4 / ipv6 pppoe 
IPsec  ipsec 
QoS  -
NetFlow  - 
Bonding*  ipv4 bonding
* Offloading support for Bonding on Cavium devices is available starting from the v2.0.0 EdgeOS firmware release.

IPsec Offloading Support


Back to Top

IPsec offload provides significant IPsec performance improvements, increasing throughput for site-to-site and client-to-site tunnels by offloading the ESP (Encapsulated Security Payload) traffic. Not all available ESP hashing/encryption algorithms are compatible with offloading. IKE traffic is not offloaded but this is only used to establish the tunnel and will not affect the performance.

The tables below summarizes the algorithms that can be offloaded on each platform.

MediaTek-Based Devices
ESP Algorithm Offload Eligible

3DES

AES-128

AES-256

MD5

SHA-1

SHA-256

SHA-384

SHA-512

AES-128-GCM

AES-256-GCM

Cavium-Based Devices
ESP Algorithm Offload Eligible

3DES

AES-128

AES-256

MD5

SHA-1

SHA-256

SHA-384

SHA-512

AES-128-GCM

AES-256-GCM


How to Enable/Disable Offloading


Back to Top

That commands that enable and disable offloading are shown below. You should only need to enable offloading for these features if you are using them in your environment. However, enabling offloading for all features will not cause a negative impact if those features are not being used.

ATTENTION: Offloading needs to be explicitly defined with the set system offload ... and delete system offload ... commands. 

MediaTek


Used by the following EdgeRouter models:

  • ER-X
  • ER-10X
  • ER-X-SFP
  • EP-R6
CLI: Access the EdgeRouter Command Line Interface.

1. Enable hwnat and ipsec offloading.

configure

set system offload hwnat enable
set system offload ipsec enable

commit ; save

2. Disable hwnat and ipsec offloading.

configure

set system offload hwnat disable
set system offload ipsec disable

commit ; save
NOTE: IPsec offloading requires a device reboot to become active/inactive.

Cavium


Used by the following EdgeRouter models:

  • ERLite-3
  • ERPoE-5
  • ER-8
  • ERPro-8
  • EP-R8
  • ER-4
  • ER-6P
  • ER-12
  • ER-12P
  • ER-8-XG
CLI: Access the EdgeRouter Command Line Interface.

1. Enable ipv4/ipv6 and ipsec offloading.

configure

set system offload ipv4 forwarding enable
set system offload ipv4 gre enable
set system offload ipv4 pppoe enable
set system offload ipv4 vlan enable
set system offload ipv4 bonding enable

set system offload ipv6 forwarding enable
set system offload ipv6 pppoe enable
set system offload ipv6 vlan enable

set system offload ipsec enable

commit ; save

2. Disable ipv4/ipv6 and ipsec offloading.

configure

set system offload ipv4 forwarding disable
set system offload ipv4 gre disable
set system offload ipv4 pppoe disable
set system offload ipv4 vlan disable
set system offload ipv4 bonding disable

set system offload ipv6 forwarding disable
set system offload ipv6 pppoe disable
set system offload ipv6 vlan disable

set system offload ipsec disable

commit ; save
NOTE: It is currently not possible to enable IPv6 offloading for PPPoE and VLANs simultaneously. IPsec offloading requires a device reboot to become active/inactive.

Optional Offloading Optimizations and Testing


Back to Top

Starting from the v1.10.3 and v1.10.4 EdgeOS firmware releases, additional offloading optimizations are available:

  • Flow Flushing Controls whether the offloading flow table is cleared upon routing table changes.
  • Flow Lifetime Controls how long it takes before offloaded flows expire.
  • Flow Table Size Controls how large the offloading table table is.

The Flow Flushing feature is enabled by default, which means that the flows in the offloading engine are flushed when a routing table change is detected. This means that any routing updates are processed instantly, but it also means that CPU resources are used to process these changes. In environments where the routing table is constantly updated, it can be advisable to disable the Flow Flushing feature which will decrease the load on the CPU and increase the performance. Starting from the v1.10.3 release, the following commands are available to disable the feature:

set system offload ipv4 disable-flow-flushing-upon-fib-changes
set system offload ipv6 disable-flow-flushing-upon-fib-changes

The Flow Lifetime is set to 12 seconds by default on all EdgeRouter models. Increasing the lifetime will lead to flows expiring less frequently and thus less packets will be forwarded using the CPU. But this will also mean that it will take more time before routing table changes are detected if the Flow Flushing feature above is also disabled. Starting from the v1.10.3 release, the following command is available to increase or decrease the lifetime:

set system offload flow-lifetime ?
<0-4294967295> Set custom lifetime for flows in offload engine (12 seconds by default)

The Flow Table Size is set to 8192 buckets by default on all EdgeRouter models. Increasing the lifetime will lead to the offloading table being able to store more flows. But this will also mean that more memory is used by the offloading process. Starting from the v1.10.4 release, the following commands are available to increase or decrease the table size:

set system offload ipv4 table-size ?
  8192          memory consumption    1 MB -     6 MB
  16384         memory consumption    2 MB -    12 MB
  32768         memory consumption    4 MB -    23 MB
  65536         memory consumption    8 MB -    46 MB
  131072        memory consumption   16 MB -    91 MB
  262144        memory consumption   32 MB -   182 MB
  524288        memory consumption   64 MB -   364 MB
  1048576       memory consumption  128 MB -   728 MB
  2097152       memory consumption  256 MB -  1456 MB
  4194304       memory consumption  512 MB -  2912 MB
  8388608       memory consumption 1024 MB -  5824 MB
  16777216      memory consumption 2048 MB - 11648 MB

set system offload ipv6 table-size ? 8192 memory consumption 1 MB - 4 MB 16384 memory consumption 2 MB - 8 MB 32768 memory consumption 4 MB - 15 MB 65536 memory consumption 8 MB - 30 MB 131072 memory consumption 16 MB - 60 MB 262144 memory consumption 32 MB - 120 MB 524288 memory consumption 64 MB - 240 MB 1048576 memory consumption 128 MB - 480 MB 2097152 memory consumption 256 MB - 960 MB 4194304 memory consumption 512 MB - 1920 MB 8388608 memory consumption 1024 MB - 3840 MB 16777216 memory consumption 2048 MB - 7680 MB

You can verify the timers and table size with the following command:

show ubnt offload statistics 

 Forwarding cache size (IPv4)
=============================

table_size (buckets)                  16384
table size (bytes)                    2097152
flows_max (bytes)                     9830400

 Flow cache table size (IPv6)
=============================

table_size (buckets)                  8192
table size (bytes)                    1048576
flows_max (bytes)                     2883584

 Flow timers
=============================

cycles                                101066477597
clock_rate                            1000000000
HZ                                    100
timer_ticks                           268415529
new_flow_interval (timer_ticks)       1200
old_flow_interval (timer_ticks)       400

You can also verify the offloading state by running the following command:

show ubnt offload 

IP offload module   : loaded
IPv4
  forwarding: enabled
  vlan      : disabled
  pppoe     : disabled
  gre       : disabled
IPv6
  forwarding: disabled
  vlan      : disabled
  pppoe     : disabled

IPSec offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.480

When high throughput is flowing on the router and the traffic is not offloaded, you will see an increase in CPU usage. When traffic is offloaded the throughput will be higher and CPU usage will remain low. Using a tool like iPerf / iPerf3 is a common way to generate and test throughput. It is important to not use the EdgeRouter as the server or client for iPerf when running the test, as the routers are designed to route/forward traffic and not to generate it.


Related Articles


Back to Top

EdgeRouter - Which EdgeRouter Should I Use?

EdgeRouter - Beginners Guide to EdgeRouter

Intro to Networking - How to Establish a Connection Using SSH


We're sorry to hear that!