This article will help users diagnose a UniFi Network Controller running under a high load, as well as describe modifications that can be done for optimal performance.
Table of Contents
- Symptom: High CPU Usage
- Symptom: Heartbeat Missed or Slow to Provision
- Database Connection Tuning
- Related Articles
There are a few things to take into consideration when managing large UniFi installs with hundreds of devices and several sites connected to a single UniFi Network Controller. A Controller running under such a high load might encounter some issues if configured the same way as a Controller managing a much lighter load. There are some symptoms that can be recognized, diagnosed and worked on to improve the controller's performance.
WARNING: Before proceeding with the configurations outlined below, make sure to create backups of your machine and controller configurations. A typo error could break the system.
Symptom: High CPU Usage
One of the most important metrics to monitor, is CPU usage on your UniFi Network Controller. High CPU usage is the first indication that there is an issue. Unfortunately, there is no silver bullet for this and just increasing the CPU is not necessarily the answer.
Allocating Additional Memory
Before increasing the size of your box, try increasing the `XMX` and `XMS` options. By default, a UniFi Network Controller has these set to 1 GB. They can be increased by making the following entries in the
system.properties file (see Related Articles below for more information on the system.properties file).
User Tip: Units here are in MB.
The changes above would increase the memory the UniFi Network Controller is allowed to consume from 1 GB to 2GB. The reason why not having enough memory could cause high CPU usage, is because the Java Virtual Machine would be spending too many CPU cycles collecting garbage to stay within the 1 GB of memory allocated to the Controller. So before moving to a machine with more CPU, it is recommended to max out the available memory on that machine with the above settings. Then observe if CPU usage decreases.
If 2GB is not sufficient, administrators may want to raise the limit to 4-8GB depending on the scale. In this case it would simply be increasing the value above in increments of 1024, i.e. 4 GB= 4096. To test whether your memory is sufficient on your UniFi Network Controller server, you can use the
jstat -gcutil command to check. For more information on doing this, see the following Oracle documentation: jstat - Java Virtual Machine Statistics Monitoring Tool
Enabling High Performance Java Garbage Collector
If increasing memory does not solve the problem, administrators may consider also adding this line to the system.properties file (note this only applies to Controllers other than Cloud Key):
This enables a new Java Garbage Collector that can help optimize performance. However, if after these changes high CPU usage does continue after memory increase, a larger machine with more CPU cores and more memory to handle the workload may be necessary.
Changing Mongo Version/Engine
In persistent cases, administrators may consider updating MongoDB version to 3.2+ with WiredTiger as storage engine, as a means of better scaling the UniFi Network Controller. See the following links for instructions:
Symptom: Heartbeat Missed or Slow to Provision
No matter the number, all devices will try to inform back to the Controller. By default, the controller can handle 200 simultaneous connections from devices, so devices with heartbeats missed shouldn't be an issue unless a single Controller is managing thousands of devices. If it is only managing a few hundred devices the adjustment described in this section can be tried out, but it may not have the desired results. The number of simultaneous inform messages that can be processed can be set in the system.properties file by adding the following lines and adjusting the values to best suit your needs. It does not matter where these lines are added in the file.
The default value is 200 and
max_keep_alive_requests should always be lower than
num_thread. Try adjusting up from there. An increase in device stability should be seen, and by pushing the configuration out to other devices, become even more stable.
Database Connection Tuning
When running a large UniFi installation, it may be desired to run an external mongo cluster to be able to scale the database independently from the UniFi Network Controller application. Discussion on that can be found here on our Community beta forum (need access?). If high CPU usage is seen on the mongo process, it can indicate the need of a bigger box or the need to separate the mongodb process as mentioned above. Once that is done, the following can be tuned to see if it results in better application performance:
The best way to increase performance and provide stability for large installations is to ensure you have monitoring on your system resources. Beyond that, practices such as offloading the database workload, and increasing memory can allow the UniFi Network Controller to serve more clients and devices. This should be reflected in the resource usage and the performance of the UniFi Network Controller UI.
UniFi offers a management solution that can take care of all this and more, find more information about UniFi Elite here.