.. title: OpenBSD on Ubiquiti USG .. slug: openbsd-on-ubiquiti-usg .. date: 2018-04-05 05:40:19 UTC .. updated: 2018-11-17 21:29:41 UTC .. tags: openbsd .. category: .. link: .. description: .. type: text Ubiquiti Unifi Security Gateway (USG) is a great device to run at the edge of a home network. It becomes even better when it's running OpenBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well. .. TEASER_END: Read more .. contents:: Why? ==== This is a follow up to `OpenBSD Network Gateway on EdgeRouter Lite `_. Since the EdgeRouter Lite 3 (ERL) and USG are very similar devices, the install process is nearly identical as well. Required Tools ============== The following two cables connect to each other so you can connect from USB on your Mac to the serial port on USG. * USB to Serial interface cable * Serial to RJ45 Console Adapter Cable for Cisco Routers You also need: * ``screen`` installed on your Mac * Phillips head screwdriver size 0 * Wired network with DHCP server to use during install Download OpenBSD ================ At the time of writing the latest release of OpenBSD was 6.3 so that's what I downloaded. :: $ curl -O http://ftp.openbsd.org/pub/OpenBSD/6.3/octeon/miniroot63.fs Hardware Setup and First Boot ============================= Once you've downloaded the file it's time to write it to a USB drive. There are two options: overwrite the original drive in the USG or buy a new drive. I tried the second option and wrote to a new Samsung 32GB USB 3.0 Flash Drive Fit (MUF-32BB/AM). It is easy to take the drive out of the USG enclosure. Take off the rubber feet and take out four screws from the back to open the enclosure. Wiggle the USB drive up and down with a little more than gentle force and it'll slide out. Even though I didn't use the original drive that came in the USG, I still created a backup image using `dd`. On my macOS it showed up as /dev/disk2 upon insertion. Replace *rdrive596870* with the name of the drive on your machine. :: $ diskutil list $ diskutil unmountdrive /dev/rdrive596870 $ sudo dd if=/dev/rdrive596870 of=original-erl.img Overwrite the replacement drive (Samsung in my case) with the downloaded OpenBSD mini root file. :: $ sudo dd if=miniroot63.fs of=/dev/rdrive596870 bs=1m && sync $ diskutil eject /dev/rdrive596870 Reinsert the drive, put back the enclosure, and put back the screws. DO NOT power on USG just yet. Connect an ethernet cable to *WLAN 1* that's connected to your network. We need a DHCP server to give an IP address to USG. Connect to the serial port. Check which device your Mac has identified for the serial connection. In my case it was */dev/tty.usbserial*. :: $ ls -ltr /dev/*usb* | grep tty Start a ``screen`` session on port 115200. :: $ screen /dev/tty.usbserial 115200 Power on USG and watch it boot on the `screen` session. Install OpenBSD =============== Once the device is powered up and USG has booted successfully, you'll see this prompt. :: Octeon ubnt_e120# Load the miniroot. :: Octeon ubnt_e120# fatload usb 0 $loadaddr bsd.rd reading bsd.rd .......................... ................. 8789365 bytes read Boot the image. :: Octeon ubnt_e120# bootoctlinux After a lot of text scrolling across you'll be asked to make a choice from the following menu. :: Welcome to the OpenBSD/octeon 6.3 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? Follow each prompt and answer appropriately. A sample session of my install is below. :: Welcome to the OpenBSD/octeon 6.3 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i At any prompt except password prompts you can escape to a shell by typing '!'. Default answers are shown in []'s and are selected by pressing RETURN. You can exit this program at any time by pressing Control-C, but this can leave your system in an inconsistent state. Terminal type? [vt220] System hostname? (short form, e.g. 'foo') usg Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0. Which network interface do you wish to configure? (or 'done') [cnmac0] cnmac0 IPv4 address for cnmac0? (or 'dhcp' or 'none') [dhcp] dhcp cnmac0: no link ..... got link DHCPDISCOVER on cnmac0 - interval 1 DHCPOFFER from 10.10.10.1 (REDACTED_FOR_PRIVACY) DHCPREQUEST on cnmac0 to 255.255.255.255 DHCPACK from 10.10.10.1 (REDACTED_FOR_PRIVACY) bound to 10.10.10.74 -- renewal in 21600 seconds. IPv6 address for cnmac0? (or 'autoconf' or 'none') [none] autoconf Available network interfaces are: cnmac0 cnmac1 cnmac2. Which network interface do you wish to configure? (or 'done') [done] done DNS domain name? (e.g. 'bar.com') [my.domain] my.example.com Using DNS nameservers at 10.10.10.1 8.8.8.8 Password for root account? (will not echo) Password for root account? (again) Start sshd(8) by default? [yes] Setup a user? (enter a lower-case loginname, or 'no') [no] ubnt Full name for user ubnt? [ubnt] Password for user ubnt? (will not echo) Password for user ubnt? (again) WARNING: root is targeted by password guessing attacks, pubkeys are safer. Allow root ssh login? (yes, no, prohibit-password) [no] yes What timezone are you in? ('?' for list) [UTC] America/Los_Angeles Available disks are: sd0. Which disk is the root disk? ('?' for details) [sd0] Disk: sd0 geometry: 487/255/63 [7831552 Sectors] Offset: 0 Signature: 0xAA55 Starting Ending LBA Info: #: id C H S - C H S [ start: size ] ------------------------------------------------------------------------------- *0: 0C 0 1 2 - 1 103 38 [ 64: 22528 ] FAT32L 1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused 2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused 3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused Use (W)hole disk or (E)dit the MBR? [whole] W Creating a FAT partition and an OpenBSD partition for rest of sd0...done. /dev/rsd0i: 65372 sectors in 16343 FAT16 clusters (2048 bytes/cluster) bps=512 spc=4 res=1 nft=2 rde=512 mid=0xf8 spf=64 spt=63 hds=255 hid=64 bsec=65536 The auto-allocated layout for sd0 is: # size offset fstype [fsize bsize cpg] a: 1024.0M 65600 4.2BSD 2048 16384 1 # / b: 768.0M 2162752 swap c: 30592.0M 0 unused d: 1914.1M 3735616 4.2BSD 2048 16384 1 # /tmp e: 2995.4M 7655616 4.2BSD 2048 16384 1 # /var f: 2021.3M 13790112 4.2BSD 2048 16384 1 # /usr g: 1024.0M 17929696 4.2BSD 2048 16384 1 # /usr/X11R6 h: 4387.9M 20026848 4.2BSD 2048 16384 1 # /usr/local i: 32.0M 64 MSDOS j: 1748.5M 29013216 4.2BSD 2048 16384 1 # /usr/src k: 3969.0M 32594176 4.2BSD 2048 16384 1 # /usr/obj l: 10707.8M 40722752 4.2BSD 2048 16384 1 # /home Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] a Rounding size to bsize (32 sectors): 3920000 Rounding size to bsize (32 sectors): 6134496 Rounding size to bsize (32 sectors): 4139584 Rounding size to bsize (32 sectors): 8986368 Rounding size to bsize (32 sectors): 3580960 Rounding size to bsize (32 sectors): 8128576 /dev/rsd0a: 1024.0MB in 2097152 sectors of 512 bytes 6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0l: 10707.8MB in 21929664 sectors of 512 bytes 53 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0d: 1914.1MB in 3920000 sectors of 512 bytes 10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0f: 2021.3MB in 4139584 sectors of 512 bytes 10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0g: 1024.0MB in 2097152 sectors of 512 bytes 6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0h: 4387.9MB in 8986368 sectors of 512 bytes 22 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0k: 3969.0MB in 8128576 sectors of 512 bytes 20 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0j: 1748.5MB in 3580960 sectors of 512 bytes 9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0e: 2995.4MB in 6134496 sectors of 512 bytes 15 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/sd0a (33d72d1625866818.a) on /mnt type ffs (rw, asynchronous, local) /dev/sd0l (33d72d1625866818.l) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0d (33d72d1625866818.d) on /mnt/tmp type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0f (33d72d1625866818.f) on /mnt/usr type ffs (rw, asynchronous, local, nodev) /dev/sd0g (33d72d1625866818.g) on /mnt/usr/X11R6 type ffs (rw, asynchronous, local, nodev) /dev/sd0h (33d72d1625866818.h) on /mnt/usr/local type ffs (rw, asynchronous, local, nodev) /dev/sd0k (33d72d1625866818.k) on /mnt/usr/obj type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0j (33d72d1625866818.j) on /mnt/usr/src type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0e (33d72d1625866818.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid) Let's install the sets! Location of sets? (disk http nfs or 'done') [http] http HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] HTTP Server? (hostname, list#, 'done' or '?') [ftp.OpenBSD.org] Server directory? [pub/OpenBSD/6.3/octeon] Select sets by entering a set name, a file name pattern or 'all'. De-select sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'. [X] bsd [X] comp63.tgz [X] xbase63.tgz [X] xserv63.tgz [X] bsd.rd [X] man63.tgz [X] xshare63.tgz [X] base63.tgz [X] game63.tgz [X] xfont63.tgz Set name(s)? (or 'abort' or 'done') [done] -game63.tgz Set name(s)? (or 'abort' or 'done') [done] -game63.tgz [X] bsd [X] comp63.tgz [X] xbase63.tgz [X] xserv63.tgz [X] bsd.rd [X] man63.tgz [X] xshare63.tgz [X] base63.tgz [ ] game63.tgz [X] xfont63.tgz Set name(s)? (or 'abort' or 'done') [done] Get/Verify SHA256.sig 100% |**************************| 1376 00:00 Signature Verified Get/Verify bsd 100% |**************************| 5654 KB 00:04 Get/Verify bsd.rd 100% |**************************| 8583 KB 00:06 Get/Verify base63.tgz 100% |**************************| 86597 KB 00:29 Get/Verify comp63.tgz 100% |**************************| 49971 KB 00:17 Get/Verify man63.tgz 100% |**************************| 7031 KB 00:05 Get/Verify xbase63.tgz 100% |**************************| 16789 KB 00:06 Get/Verify xshare63.tgz 100% |**************************| 4421 KB 00:03 Get/Verify xfont63.tgz 100% |**************************| 39349 KB 00:13 Get/Verify xserv63.tgz 100% |**************************| 5036 KB 00:04 Installing bsd 100% |**************************| 5654 KB 00:00 Installing bsd.rd 100% |**************************| 8583 KB 00:02 Installing base63.tgz 100% |**************************| 86597 KB 01:13 Extracting etc.tgz 100% |**************************| 260 KB 00:01 Installing comp63.tgz 100% |**************************| 49971 KB 00:54 Installing man63.tgz 100% |**************************| 7031 KB 00:13 Installing xbase63.tgz 100% |**************************| 16789 KB 00:21 Extracting xetc.tgz 100% |**************************| 7049 00:00 Installing xshare63.tgz 100% |**************************| 4421 KB 00:09 Installing xfont63.tgz 100% |**************************| 39349 KB 00:50 Installing xserv63.tgz 100% |**************************| 5036 KB 00:05 Location of sets? (disk http nfs or 'done') [done] Time appears wrong. Set to 'Wed Apr 4 22:14:34 PDT 2018'? [yes] Saving configuration files...done. Making all device nodes...done. Relinking to create unique kernel...done. CONGRATULATIONS! Your OpenBSD install has been successfully completed! When you login to your new system the first time, please read your mail using the 'mail' command. INSTALL.octeon describes how to configure U-Boot to boot OpenBSD. Exit to (S)hell, (H)alt or (R)eboot? [reboot] reboot I configured *root* ssh access because I use Ansible later on and found it worked better when it logged in as *root*. Feel free to disable *root* ssh for better security. I created a regular user *ubnt* to use for ssh access. I did not install *game63.tgz* just because. I didn't feel it necessary to install it. Swap the Bootloader =================== Once USG has rebooted you'll be at the *Octeon ubnt_e120#* prompt again because the bootloader is not at the expected location. We'll swap the bootloader. But first, let's get a list of the existing environment variables. **Save this output and keep in a safe place**. You may need to undo risky steps ahead. :: Octeon ubnt_e120# printenv bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) bootdelay=0 baudrate=115200 download_baudrate=115200 nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size) autoload=n ethact=octeth0 loadaddr=0x9f00000 numcores=2 stdin=serial stdout=serial stderr=serial env_addr=0x1fbfe000 env_size=0x2000 flash_base_addr=0x1f400000 flash_size=0x800000 uboot_flash_addr=0x1f480000 uboot_flash_size=0x70000 flash_unused_addr=0x1f4f0000 flash_unused_size=0x710000 bootloader_flash_update=bootloaderupdate Notice *bootcmd* above. That's what we'll swap. :: Octeon ubnt_e120# setenv old_bootcmd ${bootcmd} Octeon ubnt_e120# setenv bootcmd 'fatload usb 0 $loadaddr bsd;bootoctlinux rootdev=/dev/sd0' Octeon ubnt_e120# setenv bootdelay 5 Octeon ubnt_e120# saveenv Octeon ubnt_e120# reset After USG reboots it should happily boot OpenBSD. A sample session is below. :: U-Boot 1.1.1 (UBNT Build ID: REDACTED) (Build time: Jun 9 2014 - 14:38:01) BIST check passed. UBNT_E120 r1:0, r2:16, f:8/135, serial #: REDACTED_FOR_PRIVACY MPR 13-02044-16 Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate) DRAM: 512 MB Clearing DRAM....... done Flash: 8 MB Net: octeth0, octeth1, octeth2 USB: (port 0) scanning bus for devices... 1 USB Devices found scanning bus for storage devices... Device 0: Vendor: Samsung Prod.: Flash Drive FIT Rev: 1100 Type: Removable Hard Disk Capacity: 30592.0 MB = 29.8 GB (62652416 x 512) 0 reading bsd ............................. 5790035 bytes read ELF file is 64 bit Allocating memory for ELF segment: addr: 0xffffffff81000000 (adjusted to: 0x1000000), size 0x5796f0 Allocated memory for ELF segment: addr: 0xffffffff81000000, size 0x5796f0 Processing PHDR 0 Loading 4ebb88 bytes at ffffffff81000000 Clearing 8db68 bytes at ffffffff814ebb88 ## Loading Linux kernel with entry point: 0xffffffff81000000 ... Bootloader: Done loading app on coremask: 0x1 bootmem desc 0x24108 version 3.0 avail phys mem 0x0000000000100290 - 0x0000000000fffce0 avail phys mem 0x00000000015796f0 - 0x0000000008100000 avail phys mem 0x0000000008100010 - 0x000000000fffdc00 avail phys mem 0x0000000410000000 - 0x000000041ff00000 Total DRAM Size 0x0000000020000000 mem_layout[0] page 0x0000000000000041 -> 0x00000000000003FF mem_layout[1] page 0x000000000000055F -> 0x0000000000002040 mem_layout[2] page 0x0000000000002041 -> 0x0000000000003FFF mem_layout[3] page 0x0000000000104000 -> 0x0000000000107FC0 boot_desc->Initial setup done, switching console. boot_desc->desc_ver:7 boot_desc->desc_size:400 boot_desc->stack_top:0 boot_desc->heap_start:0 boot_desc->heap_end:0 boot_desc->argc:2 boot_desc->flags:0x5 boot_desc->core_mask:0x1 boot_desc->dram_size:512 boot_desc->phy_mem_desc_addr:0 boot_desc->debugger_flag_addr:0xa44 boot_desc->eclock:500000000 boot_desc->boot_info_addr:0x1001f0 boot_info->ver_major:1 boot_info->ver_minor:2 boot_info->stack_top:0 boot_info->heap_start:0 boot_info->heap_end:0 boot_info->boot_desc_addr:0 boot_info->exception_base_addr:0x1000 boot_info->stack_size:0 boot_info->flags:0x5 boot_info->core_mask:0x1 boot_info->dram_size:512 boot_info->phys_mem_desc_addr:0x24108 boot_info->debugger_flags_addr:0 boot_info->eclock:500000000 boot_info->dclock:266000000 boot_info->board_type:20004 boot_info->board_rev_major:0 boot_info->board_rev_minor:16 boot_info->mac_addr_count:3 boot_info->cf_common_addr:0 boot_info->cf_attr_addr:0 boot_info->led_display_addr:0 boot_info->dfaclock:0 boot_info->config_flags:0x8 Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 6.3 (GENERIC) #0: Sun Mar 25 19:48:22 UTC 2018 visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC real mem = 536870912 (512MB) avail mem = 523943936 (499MB) mainbus0 at root: board 20004 rev 0.16 cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way clock0 at mainbus0: int 5 iobus0 at mainbus0 simplebus0 at iobus0: "soc" octciu0 at simplebus0 cn30xxsmi0 at simplebus0 com0 at simplebus0: ns16550a, 64 byte fifo com0: console dwctwo0 at iobus0 base 0x1180068000000 irq 56 usb0 at dwctwo0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev 2.00/1.00 addr 1 octrng0 at iobus0 base 0x1400000000000 irq 0 cn30xxgmx0 at iobus0 base 0x1180008000000 cnmac0 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2 cnmac1 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2 cnmac2 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2 /dev/ksyms: Symbol table not valid. umass0 at uhub0 port 1 configuration 1 interface 0 "Samsung Flash Drive FIT" rev 2.10/11.00 addr 2 umass0: using SCSI over Bulk-Only scsibus0 at umass0: 2 targets, initiator 0 sd0 at scsibus0 targ 1 lun 0: SCSI4 0/direct removable serial.REDACTED_FOR_PRIVACY sd0: 30592MB, 512 bytes/sector, 62652416 sectors vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets boot device: sd0 root on sd0a (33d72d1625866818.a) swap on sd0b dump on sd0b WARNING: No TOD clock, believing file system. WARNING: CHECK AND RESET THE DATE! Automatic boot in progress: starting file system checks. /dev/sd0a (33d72d1625866818.a): file system is clean; not checking /dev/sd0l (33d72d1625866818.l): file system is clean; not checking /dev/sd0d (33d72d1625866818.d): file system is clean; not checking /dev/sd0f (33d72d1625866818.f): file system is clean; not checking /dev/sd0g (33d72d1625866818.g): file system is clean; not checking /dev/sd0h (33d72d1625866818.h): file system is clean; not checking /dev/sd0k (33d72d1625866818.k): file system is clean; not checking /dev/sd0j (33d72d1625866818.j): file system is clean; not checking /dev/sd0e (33d72d1625866818.e): file system is clean; not checking setting tty flags pf enabled starting network cnmac0: no link .... got link cnmac0: bound to 10.10.10.190 from 10.10.10.1 (REDACTED_FOR_PRIVACY) reordering libraries: done. openssl: generating isakmpd/iked RSA keys... done. ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 starting early daemons: syslogd pflogd ntpd. starting RPC daemons:. savecore: /bsd: kvm_read: version misread checking quotas: done. kvm_mkdb: can't open /dev/ksyms clearing /tmp kern.securelevel: 0 -> 1 creating runtime link editor directory cache. preserving editor files. starting network daemons: sshd smtpd sndiod. running rc.firsttime Path to firmware: http://firmware.openbsd.org/firmware/6.3/ No devices found which need firmware files to be downloaded. starting local daemons: cron. Wed Apr 4 22:18:55 PDT 2018 OpenBSD/octeon (octeon1) (console) login: Configure as Gateway and Firewall ================================= The device is configured exactly as `OpenBSD Network Gateway on EdgeRouter Lite `_. You can use Ansible roles I created for that post if you're so inclined. They're available on `GitHub - openbsd-on-erl `_. Conclusion ========== I'm pretty happy with USG and OpenBSD. Its setup was nearly identical to ERL 3. Using Ansible means identical configuration can be applied to both kinds of devices to get the same functionality.