info_i_25x25.png See important information about Ubiquiti Devices and KRACK Vulnerability in this article. We will update this document as more information becomes available.

EdgeRouter - Last resort recovery of failed EdgeOS device

 Overview


This article describes a method to completely re-install the EdgeOS platform on the EdgeRouter. If the EdgeRouter becomes unresponsive (e.g., web server will not load), you can use this as a last action method to rescue the device.

warning_25x25.png Note: The information in this tutorial was provided by user (find original post here), and it is not supported by Ubiquiti. Use under your own risk. This tutorial is for ER-Lite and ER-PoE only.

Table of Contents


  1. Intro
  2. Steps
  3. Related Articles

Intro


Back to Top

Technically speaking, this is the EdgeOS kernel recompiled and stripped down using embedded initramfs with recovery tools. You connect to your router via serial, boot it via TFTP and get into a minimal rescue system.

The following scripts for automated recovery procedures are included:

  1. emrk-factory-reset — removes the config.boot, optionally making a backup copy inside the config dir so you can access it easily after reboot.
  2. emrk-remove-user-data — removes any data from writable dirs, getting the system back to clean EdgeOS (all config data, user installed packages on so on are removed). It also removes copies of user data left after resets, as pressing reset button doesn't really remove the data but renames the old writable dir to some random name and creates a new one. Useful if you are going to give away your ERL to an untrusted third party.
  3. emrk-reinstall — downloads release tarball from URL and reinstalls EdgeOS from scratch.

Links to download scripts: http://packages.vyos.net/tools/emrk/

warning_25x25.png Note: emrk-remove-user-data and emrk-reinstall are highly disruptive! The former will remove everything but the stock EdgeOS image. The latter will remove and re-create all the partitions so all your data, if you still have any, they will be lost and irrecoverable.

Note: After trying the basic reset methods, use this recovery method only if there is no hope to recover your installation. Use at your own risk!

 


Steps


Back to Top

1. Download the image: http://packages.vyos.net/tools/emrk/0.9c/ (MD5 sum: 3545d69a8912fd967c7301031c47960e)

2. Install a TFTP server. On UNIX systems some is always available from ports/repos, Windows users may use this one: http://tftpd32.jounin.net/ Copy the image to your TFTP directory.

It's better to set up a DHCP server. On ISC DHCPD, specify

option tftp-server-name "x.x.x.x"; # Your TFTP server address
next-server x.x.x.x; # Your TFTP server address
option bootfile-name "emrk-0.9c.bin";
filename "emrk-0.9c.bin

3. Connect to your router via serial. Power on or reboot and before EdgeOS kernel loads, press any key (I usually press any key repeatadly since it starts booting). This will get you to the bootloader console where you can boot your router manually.

4. Connect router eth0 to your network.

5. If you use a DHCP server, issue the following command in U-Boot CLI:

dhcp;tftpboot;bootoctlinux $loadaddr

 If you don't have a DHCP server, you may configure network manually:

set ipaddr x.x.x.x (where x.x.x.x is desired router IP address)

set netmask 255.255.255.0 (or whatever mask you want)

set serverip y.y.y.y (where y.y.y.y is your TFTP server address)

set bootfile emrk-0.9c.bin

##Optionally
set gatewayip z.z.z.z (where z.z.z.z is your default gateway)

tftpboot

bootoctlinux $loadaddr

6. If everything goes right, after bootoctlinux command you will see the kernel booting and will be asked if you want to continue after a scary disclaimer. Answer "yes" (case insensitive) or "no", than asked to configure network. You may choose DHCP or configure it statically.

7. To reinstall EdgeOS from scratch, you will need to download the tarball and put it on your HTTP/FTP/SSH server. 

8. Here is an example of session with reinstall (kernel boot messages omitted).

Loading EMRK 0.9a
Mounting filesystems
Bringing up eth0

Checking boot partition
Boot partition looks intact
Attempting to mount boot partition
Boot partition successfully mounted
Looking for kernel file
Found a kernel
Checking kernel MD5 sum file
Found kernel MD5 sum file
Checking kernel MD5 sum
Kernel MD5 sum is correct

Checking root partition
Root partition looks intact
Attempting to mount root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Root partition successfully mounted
Looking for system image file
Found a system image file
Checking system image MD5 sum file
Found system image MD5 sum file
Checking system image MD5 sum
System image MD5 sum is correct

**********************************************
Welcome to EdgeMAX Rescue Kit!

This tool is distributed under the terms of
GNU General Public License and other licenses

Brought to you by SO3 Group

WARNING: This tool is not developed, officially
supported or endorsed by Ubiquiti Networks!

Using it may lead to destroying your router
configuration or operating system

Ubiquiti Networks support will not help you
with using it or fixing consequences of
using it.

This tool itself is distributed without any
warranty and authors are not liable for
any damage it may cause

By using this tool you agree you are doing
it at your own risk and understand what
you are doing

*********************************************

Enter 'Yes' to proceed, 'No' to reboot
yes or no: yes

Do you want to configure network via DHCP?
yes or no: yes
udhcpc (v1.17.1) started
Sending discover...
Sending select for 10.91.19.175...
Lease of 10.91.19.175 obtained, lease time 86400
/usr/share/udhcpc/default.script: Resetting default routes
route: SIOCDELRT: No such process
/usr/share/udhcpc/default.script: Adding DNS 10.91.19.1

EMRK provides some scripts for automated
recovery procedures:

emrk-factory-reset -- reset config to factory default
emrk-remove-user-data -- remove all the user data including
    config and everything
emrk-reinstall -- reinstall EdgeOS from scratch
    (wipes any user data too)

Enter 'reboot' to reboot your router


BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/ash: can't access tty; job control turned off
EMRK>
EMRK>emrk-reinstall 
WARNING: This script will reinstall EdgeOS from scratch
If you have any usable data on your router storage,
it will be irrecoverably destroyed!
Do you want to continue?
yes or no: yes
Unmounting boot partition
Unmounting root partition
Re-creating partition table
Creating boot partition
Formatting boot partition
mkfs.vfat 3.0.9 (31 Jan 2010)
Creating root partition
Formatting root partition
Mounting boot parition
Mounting root partition
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
Enter EdgeOS image url: http://dl.ubnt.com/firmwares/edgemax/v1.6.0/ER-e100.v1.6.0.4716006.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 68.3M  100 68.3M    0     0  8319k      0  0:00:08  0:00:08 --:--:-- 6318k
Unpacking EdgeOS release image
Verifying EdgeOS kernel
Copying EdgeOS kernel to boot partition
Verifying EdgeOS system image
Copying EdgeOS system image to root partition
Creating EdgeOS writable data directory
Cleaning up
Installation finished
Please reboot your router

If you want to do things manually, there are busybox with symlinks to its applets (cat/grep/rm/cp/mv/tar/etc., ping, traceroute, nc etc.), curl, parted, and mkfs.ext3/mkfs.vfat.

I used it on my on ERL when I replaced the USB stick with a 8GB one yesterday. Hope you find it useful, any bug reports or suggestions are welcome.

Here are the scripts it uses: http://packages.vyos.net/tools/emrk 


Related Articles


Back to Top

How to Establish a Connection Using SSH