After reading through this article readers should expect to have a cursory knowledge about how to identify database issues on the Cloud Key. This article can also be applicable to Debian-based Linux.
NOTES & REQUIREMENTS:
This article includes instructions that require the use of an SSH client such as PuTTY, or native Linux/macOS terminals, etc.
Devices used in this article:
Table of Contents
What to look for:
- Statistics not reporting within the data retention period. Typically the statistics will only record for a few days and then be cut off or intermittent.
- Not being able to make a backup that includes statistics (7-day, 30-day, etc).
- If the settings-only backup cannot complete, it may be corruption in the ace database or something else unrelated.
- Error 400 upon loading controller GUI
Total data doesn't have to exceed 2GB on the UC-CK, it only needs to be big enough for Mongo to expand. On 64-bit machines not affected by the MongoDB limitation, there can also be overall space issues on the host that can cause issues when trying to repair the database. Check this with
32-bit MongoDB Limitations
32-bit MongoDB collections are limited to a total size of 2GB. While running the MMAPv1 storage engine this can cause issues when trying to shrink the aggregate collection sizes. When running a compact command on the MongoDB, the database is rewritten and defragmented without error correction. This process does not return useable disk space to the controller host when the storage engine is MMAPv1.
Getting back that valuable disk space on a 32-bit MongoDB instance will require a repair of the database. This repair may be needed for those who are suspecting invalid entries, corruption, or collection scaling issues. To be able to run a database repair will require that the Cloud Key has enough free disk space equal to the size of the current data set plus 2 gigabytes. If the dataset has already grown too large to proceed with the db.repairDatabase() command there are instructions below to work around that.
NOTE: The Cloud Key (UC-CK) utilizes the following:
Steps: How to Identify and Repair a Database
1. Begin by identifying the problem. The commands below will help users diagnose and verify information on the Cloud Key. Begin by accessing the command line interface (CLI). You can do this using an SSH client.
To verify how much space is used run the following command:
du -shc /usr/lib/unifi/data/db/ace* /usr/lib/unifi/data/db/journal/* /usr/lib/unifi/data/db/local* | grep total
To verify what mongoDB version is being used, run the following:
mongod --version | grep "db version"
To verify what storage engine version is being used run the following two commands:
mongo localhost:27117 db.serverStatus().storageEngine.name
2. Once the problem is identified, download the mongo pruning script. As written, the script will only keep entries from the past 7 days. It will not remove entries that are required for UniFi administration. Do so by running the following command:
3. Stop the UniFi service with the following:
service unifi stop
4. The UniFi service will attempt to shut down the mongo server. Make sure that the mongo server has been completely shut down and then restart it with this command:
mongod --dbpath /usr/lib/unifi/data/db --smallfiles --logpath /usr/lib/unifi/logs/server.log --fork
5. Execute the script to prune the
stat collections that are causing the problem:
mongo < /tmp/CK_repair.js
6. Shut down the mongo server with this command:
mongod --dbpath /usr/lib/unifi/data/db --smallfiles --logpath /usr/lib/unifi/logs/server.log --shutdown
7. The last command in the script, "db.repairDatabase()", may result in an error if the stats collections have already grown too large to be repaired while the mongo server is running. If you see such an error then you will need to repair the database from the command line by running this command:
mongod --dbpath /usr/lib/unifi/data/db --smallfiles --logpath /usr/lib/unifi/logs/server.log --repair
8. Finally, restart the UniFi service:
service unifi start
9. The Cloud Key should now be able to run the Controller. If you still have database issues please contact Ubiquiti Support.