EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running OpenBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well. # Why? This is a follow up to [FreeBSD Network Gateway on EdgeRouter Lite](link://slug/freebsd-network-gateway-on-edgerouter-lite). I observed some issues, possibly related to network drivers, where the network would become unresponsive after some time. I first tried OpenWRT on the same hardware and didn't observe any unresponsiveness. I wanted to use a BSD instead of Linux so tried OpenBSD. OpenBSD has all the required ingredients to make a secure and easy-to-administer firewall. I find the syntax of pf much better than iptables and would always prefer it if I had a choice. Before you read this guide make sure you have read these resources which I used as well for this guide. - [INSTALLATION NOTES for OpenBSD/octeon 6.4](http://ftp.openbsd.org/pub/OpenBSD/6.4/octeon/INSTALL.octeon) - [Ubiquiti Edge Router Lite 3 OpenBSD Upgrade on OSX](https://github.com/ryanmaclean/ubiquiti-edge-router-lite-openbsd-osx) - [Building a Router](https://www.openbsd.org/faq/pf/example1.html) - [The modern OpenBSD home router](https://www.azabani.com/2015/08/06/modern-openbsd-home-router.html) - [The ultimate OpenBSD router](http://www.bsdnow.tv/tutorials/openbsd-router) I initially wrote this post with OpenBSD 6.1. I have updated it to work with OpenBSD 6.4. # Required Tools The following two cables connect to each other so you can connect from USB on your Mac to the serial port on ERL3. - USB to Serial interface cable - Serial to RJ45 Console Adapter Cable for Cisco Routers You also need: - ``screen`` installed on your Mac - Phillips head screwdriver size 0 - Wired network with DHCP server to use during install # Download OpenBSD At the time of writing the latest release of OpenBSD was 6.4 so that's what I downloaded. $ curl -O http://ftp.openbsd.org/pub/OpenBSD/6.4/octeon/miniroot64.fs # Hardware Setup and First Boot Once you've downloaded the file it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive. I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It *did not* work and I later found on some blog that those drives do not work. I bought another drive -- Samsung 32GB USB 3.0 Flash Drive Fit (MUF-32BB/AM) -- and that *did* work. I later bought another drive -- Samsung BAR Plus 32GB - 200MB/s USB 3.1 Flash Drive Titan Gray (MUF-32BE4/AM) -- which worked great as well. It is easy to take the drive out of the ERL enclosure. Take out three screws from the back to open the enclosure. Wiggle the drive up and down with a little more than gentle force and it'll slide out. Before overwriting the original drive I created a backup image using ``dd``. On my macOS it showed up as /dev/disk2 upon insertion. Replace *rdrive596870* with the name of the drive on your machine. $ diskutil list $ diskutil unmountDisk /dev/rdrive596870 $ sudo dd if=/dev/rdrive596870 of=original-erl.img With the backup created it was easy to overwrite the drive with the downloaded file. $ sudo dd if=miniroot64.fs of=/dev/rdrive596870 bs=1m && sync $ diskutil eject /dev/rdrive596870 Reinsert the drive, put back the enclosure, and put back the screws. DO NOT power on ERL3 just yet. Connect an ethernet cable to eth0 that's connected to your network. We need a DHCP server to give an IP address to ERL3. Connect to the serial port. Check which device your Mac has identified for the serial connection. In my case it was */dev/tty.usbserial*. $ ls -ltr /dev/*usb* | grep tty Start a ``screen`` session on port 115200. $ screen /dev/tty.usbserial 115200 Power on ERL3 and watch it boot on the ``screen`` session. # Install OpenBSD Once the device is powered up and ERL3 has booted successfully, you'll see this prompt. Octeon ubnt_e100# Load the miniroot. Octeon ubnt_e100# fatload usb 0 $loadaddr bsd.rd Boot the image. Octeon ubnt_e100# bootoctlinux After a lot of text scrolling across you'll be asked to make a choice from the following menu. Welcome to the OpenBSD/octeon 6.4 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? Follow each prompt and answer appropriately. A sample session of my install is below. Welcome to the OpenBSD/octeon 6.4 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i At any prompt except password prompts you can escape to a shell by typing '!'. Default answers are shown in []'s and are selected by pressing RETURN. You can exit this program at any time by pressing Control-C, but this can leave your system in an inconsistent state. Terminal type? [vt220] System hostname? (short form, e.g. 'foo') erl Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0. Which network interface do you wish to configure? (or 'done') [cnmac0] IPv4 address for cnmac0? (or 'dhcp' or 'none') [dhcp] cnmac0: no link ..... got link cnmac0: bound to from (REDACTED) IPv6 address for cnmac0? (or 'autoconf' or 'none') [none] autoconf Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0. Which network interface do you wish to configure? (or 'done') [done] Using DNS domainname my.example.com Using DNS nameservers at Password for root account? (will not echo) Password for root account? (again) Start sshd(8) by default? [yes] Setup a user? (enter a lower-case loginname, or 'no') [no] ubnt Full name for user ubnt? [ubnt] Password for user ubnt? (will not echo) Password for user ubnt? (again) WARNING: root is targeted by password guessing attacks, pubkeys are safer. Allow root ssh login? (yes, no, prohibit-password) [no] yes What timezone are you in? ('?' for list) [UTC] UTC Available disks are: sd0. Which disk is the root disk? ('?' for details) [sd0] Disk: sd0 geometry: 3900/255/63 [62656641 Sectors] Offset: 0 Signature: 0xAA55 Starting Ending LBA Info: #: id C H S - C H S [ start: size ] ------------------------------------------------------------------------------- *0: 0C 0 1 2 - 1 103 38 [ 64: 22528 ] FAT32L 1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused 2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused 3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused Use (W)hole disk or (E)dit the MBR? [whole] Creating a FAT partition and an OpenBSD partition for rest of sd0...done. /dev/rsd0i: 65372 sectors in 16343 FAT16 clusters (2048 bytes/cluster) bps=512 spc=4 res=1 nft=2 rde=512 mid=0xf8 spf=64 spt=63 hds=255 hid=64 bsec=65536 The auto-allocated layout for sd0 is: # size offset fstype [fsize bsize cpg] a: 1024.0M 65600 4.2BSD 2048 16384 1 # / b: 768.0M 2162752 swap c: 30594.1M 0 unused d: 1750.4M 3735616 4.2BSD 2048 16384 1 # /tmp e: 2729.4M 7320416 4.2BSD 2048 16384 1 # /var f: 1919.0M 12910208 4.2BSD 2048 16384 1 # /usr g: 995.4M 16840320 4.2BSD 2048 16384 1 # /usr/X11R6 h: 4081.0M 18878880 4.2BSD 2048 16384 1 # /usr/local i: 32.0M 64 MSDOS j: 1707.6M 27236768 4.2BSD 2048 16384 1 # /usr/src k: 5935.2M 30733920 4.2BSD 2048 16384 1 # /usr/obj l: 9652.1M 42889184 4.2BSD 2048 16384 1 # /home Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] /dev/rsd0a: 1024.0MB in 2097152 sectors of 512 bytes 6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0l: 9652.1MB in 19767456 sectors of 512 bytes 48 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0d: 1750.4MB in 3584800 sectors of 512 bytes 9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0f: 1919.0MB in 3930112 sectors of 512 bytes 10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0g: 995.4MB in 2038560 sectors of 512 bytes 5 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0h: 4081.0MB in 8357888 sectors of 512 bytes 21 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0k: 5935.2MB in 12155264 sectors of 512 bytes 30 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0j: 1707.6MB in 3497152 sectors of 512 bytes 9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/rsd0e: 2729.4MB in 5589792 sectors of 512 bytes 14 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each /dev/sd0a (a6dc8ad8dc03899c.a) on /mnt type ffs (rw, asynchronous, local) /dev/sd0l (a6dc8ad8dc03899c.l) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0d (a6dc8ad8dc03899c.d) on /mnt/tmp type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0f (a6dc8ad8dc03899c.f) on /mnt/usr type ffs (rw, asynchronous, local, nodev) /dev/sd0g (a6dc8ad8dc03899c.g) on /mnt/usr/X11R6 type ffs (rw, asynchronous, local, nodev) /dev/sd0h (a6dc8ad8dc03899c.h) on /mnt/usr/local type ffs (rw, asynchronous, local, nodev) /dev/sd0k (a6dc8ad8dc03899c.k) on /mnt/usr/obj type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0j (a6dc8ad8dc03899c.j) on /mnt/usr/src type ffs (rw, asynchronous, local, nodev, nosuid) /dev/sd0e (a6dc8ad8dc03899c.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid) Let's install the sets! Location of sets? (disk http nfs or 'done') [http] HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] HTTP Server? (hostname, list#, 'done' or '?') [ftp.OpenBSD.org] Server directory? [pub/OpenBSD/6.4/octeon] Select sets by entering a set name, a file name pattern or 'all'. De-select sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'. [X] bsd [X] comp64.tgz [X] xbase64.tgz [X] xserv64.tgz [X] bsd.rd [X] man64.tgz [X] xshare64.tgz [X] base64.tgz [X] game64.tgz [X] xfont64.tgz Set name(s)? (or 'abort' or 'done') [done] -game64.tgz [X] bsd [X] comp64.tgz [X] xbase64.tgz [X] xserv64.tgz [X] bsd.rd [X] man64.tgz [X] xshare64.tgz [X] base64.tgz [ ] game64.tgz [X] xfont64.tgz Set name(s)? (or 'abort' or 'done') [done] Get/Verify SHA256.sig 100% |**************************| 1365 00:00 Signature Verified Get/Verify bsd 100% |**************************| 5714 KB 00:04 Get/Verify bsd.rd 100% |**************************| 8626 KB 00:06 Get/Verify base64.tgz 100% |**************************| 87690 KB 00:29 Get/Verify comp64.tgz 100% |**************************| 50446 KB 00:19 Get/Verify man64.tgz 100% |**************************| 7093 KB 00:04 Get/Verify xbase64.tgz 100% |**************************| 17055 KB 00:09 Get/Verify xshare64.tgz 100% |**************************| 4432 KB 00:03 Get/Verify xfont64.tgz 100% |**************************| 39348 KB 00:24 Get/Verify xserv64.tgz 100% |**************************| 5046 KB 00:04 Installing bsd 100% |**************************| 5714 KB 00:00 Installing bsd.rd 100% |**************************| 8626 KB 00:00 Installing base64.tgz 100% |**************************| 87690 KB 00:50 Extracting etc.tgz 100% |**************************| 259 KB 00:00 Installing comp64.tgz 100% |**************************| 50446 KB 00:38 Installing man64.tgz 100% |**************************| 7093 KB 00:07 Installing xbase64.tgz 100% |**************************| 17055 KB 00:12 Extracting xetc.tgz 100% |**************************| 6961 00:00 Installing xshare64.tgz 100% |**************************| 4432 KB 00:07 Installing xfont64.tgz 100% |**************************| 39348 KB 00:19 Installing xserv64.tgz 100% |**************************| 5046 KB 00:03 Location of sets? (disk http nfs or 'done') [done] Time appears wrong. Set to 'Sat Nov 17 20:47:46 UTC 2018'? [yes] Saving configuration files...done. Making all device nodes...done. Relinking to create unique kernel... done. CONGRATULATIONS! Your OpenBSD install has been successfully completed! When you login to your new system the first time, please read your mail using the 'mail' command. INSTALL.octeon describes how to configure U-Boot to boot OpenBSD. Exit to (S)hell, (H)alt or (R)eboot? [reboot] I configured *root* SSH access because I use Ansible later on and found it worked better when it logged in as *root*. Feel free to disable *root* SSH for better security. I created a regular user *ubnt* to use for SSH access. I did not install *game64.tgz* just because. I didn't feel it necessary to install it. Time to reboot. # Swap the Bootloader Once ERL3 has rebooted you'll be at the *Octeon ubnt_e100#* prompt again because the bootloader is not at the expected location. We'll swap the bootloader. But first, let's get a list of the existing environment variables. **Save this output and keep in a safe place**. You may need to undo risky steps ahead. Octeon ubnt_e100# printenv baudrate=115200 download_baudrate=115200 nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size) autoload=n ethact=octeth0 bootdelay=5 bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) loadaddr=0x9f00000 numcores=2 stdin=serial stdout=serial stderr=serial env_addr=0x1fbfe000 env_size=0x2000 flash_base_addr=0x1f800000 flash_size=0x400000 uboot_flash_addr=0x1f880000 uboot_flash_size=0x70000 flash_unused_addr=0x1f8f0000 flash_unused_size=0x310000 bootloader_flash_update=bootloaderupdate Notice *bootcmd* above. That's what we'll swap. Octeon ubnt_e100# setenv old_bootcmd "${bootcmd}" Octeon ubnt_e100# setenv bootcmd 'fatload usb 0 $loadaddr bsd;bootoctlinux rootdev=/dev/sd0' Octeon ubnt_e100# setenv bootdelay 5 Octeon ubnt_e100# saveenv Octeon ubnt_e100# reset After ERL3 reboots it should happily boot OpenBSD. A sample session is below. U-Boot 1.1.1 (UBNT Build ID: REDACTED) (Build time: May 27 2014 - 11:16:22) BIST check passed. UBNT_E100 r1:2, r2:18, f:4/71, serial #: REDACTED MPR 13-00318-18 Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate) DRAM: 512 MB Clearing DRAM....... done Flash: 4 MB Net: octeth0, octeth1, octeth2 USB: (port 0) scanning bus for devices... 1 USB Devices found scanning bus for storage devices... Device 0: Vendor: Samsung Prod.: Flash Drive Rev: 1100 Type: Removable Hard Disk Capacity: 30594.0 MB = 29.8 GB (62656641 x 512)  4  3  2  1  0 reading bsd ............................. 5851467 bytes read ELF file is 64 bit Allocating memory for ELF segment: addr: 0xffffffff81000000 (adjusted to: 0x1000000), size 0x588c30 Allocated memory for ELF segment: addr: 0xffffffff81000000, size 0x588c30 Processing PHDR 0 Loading 4f7ef8 bytes at ffffffff81000000 Clearing 90d38 bytes at ffffffff814f7ef8 ## Loading Linux kernel with entry point: 0xffffffff81000000 ... Bootloader: Done loading app on coremask: 0x1 bootmem desc 0x24108 version 3.0 avail phys mem 0x0000000000100290 - 0x0000000000fffbe0 avail phys mem 0x0000000001588c30 - 0x0000000008100000 avail phys mem 0x0000000008100010 - 0x000000000fffdc00 avail phys mem 0x0000000410000000 - 0x000000041ff00000 Total DRAM Size 0x0000000020000000 mem_layout[0] page 0x0000000000000041 -> 0x00000000000003FF mem_layout[1] page 0x0000000000000563 -> 0x0000000000002040 mem_layout[2] page 0x0000000000002041 -> 0x0000000000003FFFInitial setup done, switching console. boot_desc->desc_ver:7 boot_desc->desc_size:400 boot_desc->stack_top:0 boot_desc->heap_start:0 boot_desc->heap_end:0 boot_desc->argc:2 boot_desc->flags:0x5 boot_desc->core_mask:0x1 boot_desc->dram_size:512 boot_desc->phy_mem_desc_addr:0 boot_desc->debugger_flag_addr:0xa44 boot_desc->eclock:500000000 boot_desc->boot_info_addr:0x1001f0 boot_info->ver_major:1 boot_info->ver_minor:2 boot_info->stack_top:0 boot_info->heap_start:0 boot_info->heap_end:0 boot_info->boot_desc_addr:0 boot_info->exception_base_addr:0x1000 boot_info->stack_size:0 boot_info->flags:0x5 boot_info->core_mask:0x1 boot_info->dram_size:512 boot_info->phys_mem_desc_addr:0x24108 boot_info->debugger_flags_addr:0 boot_info->eclock:500000000 boot_info->dclock:266000000 boot_info->board_type:20002 boot_info->board_rev_major:2 boot_info->board_rev_minor:18 boot_info->mac_addr_count:3 boot_info->cf_common_addr:0 boot_info->cf_attr_addr:0 boot_info->led_display_addr:0 boot_info->dfaclock:0 boot_info->config_flags:0x8 Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 6.4 (GENERIC) #0: Sat Oct 13 03:41:10 UTC 2018 visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC real mem = 536870912 (512MB) avail mem = 523878400 (499MB) mainbus0 at root: board 20002 rev 2.18 cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way clock0 at mainbus0: int 5 octcrypto0 at mainbus0 iobus0 at mainbus0 simplebus0 at iobus0: "soc" octciu0 at simplebus0 cn30xxsmi0 at simplebus0 com0 at simplebus0: ns16550a, 64 byte fifo com0: console dwctwo0 at iobus0 base 0x1180068000000 irq 56 usb0 at dwctwo0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev 2.00/1.00 addr 1 octrng0 at iobus0 base 0x1400000000000 irq 0 cn30xxgmx0 at iobus0 base 0x1180008000000 cnmac0 at cn30xxgmx0: RGMII, address REDACTED atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2 cnmac1 at cn30xxgmx0: RGMII, address REDACTED atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2 cnmac2 at cn30xxgmx0: RGMII, address REDACTED atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2 /dev/ksyms: Symbol table not valid. umass0 at uhub0 port 1 configuration 1 interface 0 "Samsung Flash Drive" rev 2.10/11.00 addr 2 umass0: using SCSI over Bulk-Only scsibus0 at umass0: 2 targets, initiator 0 sd0 at scsibus0 targ 1 lun 0: SCSI4 0/direct removable serial.REDACTED sd0: 30594MB, 512 bytes/sector, 62656641 sectors vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets boot device: sd0 root on sd0a (a6dc8ad8dc03899c.a) swap on sd0b dump on sd0b WARNING: No TOD clock, believing file system. WARNING: CHECK AND RESET THE DATE! Automatic boot in progress: starting file system checks. /dev/sd0a (a6dc8ad8dc03899c.a): file system is clean; not checking /dev/sd0l (a6dc8ad8dc03899c.l): file system is clean; not checking /dev/sd0d (a6dc8ad8dc03899c.d): file system is clean; not checking /dev/sd0f (a6dc8ad8dc03899c.f): file system is clean; not checking /dev/sd0g (a6dc8ad8dc03899c.g): file system is clean; not checking /dev/sd0h (a6dc8ad8dc03899c.h): file system is clean; not checking /dev/sd0k (a6dc8ad8dc03899c.k): file system is clean; not checking /dev/sd0j (a6dc8ad8dc03899c.j): file system is clean; not checking /dev/sd0e (a6dc8ad8dc03899c.e): file system is clean; not checking setting tty flags pf enabled starting network cnmac0: no link.... got link cnmac0: bound to from (REDACTED) reordering libraries: done. openssl: generating isakmpd/iked RSA keys... done. ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 starting early daemons: syslogd pflogd ntpd. starting RPC daemons:. savecore: /bsd: kvm_read: version misread checking quotas: done. kvm_mkdb: can't open /dev/ksyms clearing /tmp kern.securelevel: 0 -> 1 creating runtime link editor directory cache. preserving editor files. starting network daemons: sshd smtpd sndiod. running rc.firsttime Path to firmware: http://firmware.openbsd.org/firmware/6.4/ No devices found which need firmware files to be downloaded. starting local daemons: cron. Sat Nov 17 20:51:23 UTC 2018 OpenBSD/octeon (octeon1) (console) login: # Apply Patches Read [Apply Patch to OpenBSD](link://slug/apply-patch-to-openbsd) on how to apply security and other patches before continuing. Since patches gets released as needed, be sure to keep your system updated. # Network Design Comcast is my internet service provider (ISP) and thus the Ansible stuff is configured to work with it. Be especially careful with the IPv6 stuff since your provider may be different. I wanted both IPv4 and IPv6 running in my network and to reach out to the internet. For the wireless LAN (WLAN) or WiFi part of the network I'm using a simple access point (AP) that provides no routing, DHCP, or other services. It acts as a dumb AP providing WiFi services only. # Networking ERL3 has three network interfaces: eth0, eth1, and eth2. They are recognized by OpenBSD as cnmac0, cnmac1, and cnmac2 respectively. I'll use cnmac0 (eth0) as the WAN port. I'll bridge cnmac1 (eth1) and cnmac2 (eth2) for LAN. ## cnmac0 To troubleshoot my internet service I use a USB-to-ethernet dongle and its MAC address is what's configured with Comcast on my cable modem. I use the same MAC address to override any gateway device that connects to the cable modem since I can swap devices without having to contact Comcast to update settings at their end. I override the MAC address of cnmac0 in my config but you don't have to. I was confused for a while and got it wrong on getting an IPv6 non-temporary address (NA) via dhcp6c. Just like I would expect a client OS to ask a DHCPv6 server to hand it an IPv6 address, I was trying to do it on this router. I've explained what I was doing wrong in a section below. What I needed to understand was that the WAN interface (cnmac0) only needed IPv6 autoconf to get a link-local address going. Then it was a matter of adding a default route. With this setup ERL could reach any globally routable address through the ISP-provided gateway. # vi /etc/hostname.cnmac0 dhcp lladdr OV:ER:RI:DE:00:00 up inet6 autoconf !/usr/sbin/rcctl start dhcpcd It took me a while to get this right but it's very important that you understand why I did it this way. ## cnmac1 # vi /etc/hostname.cnmac1 up ## cnmac2 # vi /etc/hostname.cnmac2 up ## vether0 Create a virtual ethernet interface to be used in bridging cnmac1 and cnmac2. # vi /etc/hostname.vether0 inet ## bridge0 Create a bridge for LAN ports. # vi /etc/hostname.bridge0 add vether0 add cnmac1 add cnmac2 blocknonip vether0 blocknonip cnmac1 blocknonip cnmac2 up # dhcpcd Install dhcpcd. # pkg_add -U dhcpcd Add the following lines at the end of */etc/dhcpcd.conf*. Leave other lines untouched. # vi /etc/dhcpcd.conf nohook resolv.conf release denyinterfaces cnmac1 cnmac2 noipv4 interface cnmac0 ipv6rs ia_na 1 ia_pd 2 vether0/0/64 Enable and start dhcpcd. # rcctl enable dhcpcd # rcctl start dhcpcd # dhcp6c WARNING: DO NOT USE dhcp6c. USE dhcpcd INSTEAD. KEEPING THIS SECTION HERE FOR INFORMATIONAL PURPOSES ONLY. Install dhcp6. # pkg_add -U wide-dhcpv6 Add a line to */etc/rc.conf.local*. Leave other lines, if present, untouched. # vi /etc/rc.conf.local dhcp6c_flags=cnmac0 Append a line (``!/usr/sbin/rcctl restart dhcp6c``) at the end of */etc/hostname.cnmac0*. Leave other lines untouched. That makes the entire contents of */etc/hostname.cnmac0* to be. # vi /etc/hostname.cnmac0 dhcp lladdr OV:ER:RI:DE:00:00 up inet6 autoconf !/sbin/route add ::/0 -ifp cnmac0 fe80:: !/usr/sbin/rcctl restart dhcp6c Make sure */etc/dhcp6c.conf* looks like below for Comcast. You may need to alter some settings depending on your network and ISP. interface cnmac0 { send ia-pd 0; }; id-assoc pd 0 { prefix ::/64 infinity; prefix-interface vether0 { sla-id 1; sla-len 0; }; }; Create an init script. # vi /etc/rc.d/dhcp6c #!/bin/sh daemon="/usr/local/sbin/dhcp6c" . /etc/rc.d/rc.subr rc_reload=NO rc_cmd $1 Change permissions of the init script. # chmod ugo+rx /etc/rc.d/dhcp6c Enable and start dhcp6c. # rcctl enable dhcp6c # rcctl start dhcp6c The "Wrong" Config ------------------ This is the "wrong" way to configure dhcp6c.conf for a router. interface cnmac0 { send ia-pd 0; send ia-na 1; }; id-assoc na 1 { }; id-assoc pd 0 { prefix ::/64 infinity; prefix-interface vether0 { sla-id 1; sla-len 0; }; }; Correspondingly, */etc/hostname.cnmac0* was configured thusly. # vi /etc/hostname.cnmac0 dhcp lladdr OV:ER:RI:DE:00:00 up inet6 autoconf !/usr/sbin/rcctl restart dhcp6c Notice how I have ``send ia-na 1;`` and ``id-assoc na 1 { };`` in the "wrong" config above? You need these when the device is on the edge of the network, for example a server or a desktop. This tells the DHCPv6 server to assign your WAN a non-temporary IPv6 address. You would use this address to access the device (client) from the WAN side. You don't necessarily have to do the same on a router, primarily because you can access it from the LAN (vether0) side. I reached out to [misc@](https://lists.openbsd.org/cgi-bin/mj_wwwusr?func=lists-long-full&extra=misc) mailing list with a cry for help ([WAN interface loses IPv6 NA address after pltime/vltime expire](https://marc.info/?t=149824510400003&r=1&w=2)). I was running into a problem where -- with the above config -- WAN was getting an IPv6 address (IA_NA) when dhcp6c was started. The remaining lease time was provided by pltime and vltime values in ``ifconfig cnmac0``. When that time expired, dhcp6c would renew the address but it never got applied to the network interface (cnmac0). This resulted in my entire network losing IPv6 connectivity to the outside world. All devices would have an IPv6 address but they couldn't get to the Internet. When I restarted dhcp6c (``rcctl restart dhcp6c``), cnmac0 would get an IPv6 address again and things would start working. I ended up creating a cron entry to restart dhcp6c every hour. As Stuart Henderson notes in his reply to my email, There was a problem with vltime/pltime exported from the kernel being incorrect which was fixed after 6.4 I appeared to have been running into this bug since I was using OpenBSD 6.4. I'll try [wide-dhcpv6](http://ports.su/net/wide-dhcpv6) on OpenBSD 6.2 or later when they become available to see if my original problem is fixed. Otherwise it may be time to give [dhcpcd](http://ports.su/net/dhcpcd) a try instead, as suggested by Henderson. It seems to be a more actively maintained project and thus merits a deeper look. Ideally, WAN should also be assigned a globally routable IPv6 address by the ISP so it can be reached from outside. It works for me when I run [LEDE](https://lede-project.org/) ([OpenWRT](https://openwrt.org/) fork) or [RouterOS](https://mikrotik.com/software). It does work in this case, too, but obviously there are deficiencies and bugs. Once the kinks are ironed out I'd consider this to not be the "wrong" config. It's "wrong" as long as there are problems using it. # Unbound DNS Server I decided on [Unbound](https://unbound.net/) for DNS services in my LAN. Big thanks to [Unbound DNS Tutorial](https://calomel.org/unbound_dns.html) for all the help it provided in configuring Unbound. Create */var/unbound/etc/unbound.conf* with these contents. Modify as needed. # vi /var/unbound/etc/unbound.conf ## Authoritative, validating, recursive caching DNS ## modified form of unbound.conf from https://calomel.org retrieved on 2016-10-24 # server: # log verbosity verbosity: 1 # specify the interfaces to answer queries from by ip-address. The default # is to listen to localhost ( and ::1). specify and ::0 to # bind to all available interfaces. specify every interface[@port] on a new # 'interface:' labeled line. The listen interfaces are not changed on # reload, only on restart. interface: interface: ::1 interface: interface: ::0 # port to answer queries from port: 53 # Enable IPv4, "yes" or "no". do-ip4: yes # Enable IPv6, "yes" or "no". do-ip6: yes # Enable UDP, "yes" or "no". do-udp: yes # Enable TCP, "yes" or "no". If TCP is not needed, Unbound is actually # quicker to resolve as the functions related to TCP checks are not done.i # NOTE: you may need tcp enabled to get the DNSSEC results from *.edu domains # due to their size. do-tcp: yes # control which client ips are allowed to make (recursive) queries to this # server. Specify classless netblocks with /size and action. By default # everything is refused, except for localhost. Choose deny (drop message), # refuse (polite error reply), allow (recursive ok), allow_snoop (recursive # and nonrecursive ok) access-control: allow access-control: allow access-control: allow # Read the root hints from this file. Default is nothing, using built in # hints for the IN class. The file has the format of zone files, with root # nameserver names and addresses only. The default may become outdated, # when servers change, therefore it is good practice to use a root-hints # file. get one from ftp://FTP.INTERNIC.NET/domain/named.cache #root-hints: "/var/unbound/etc/root.hints" # enable to not answer id.server and hostname.bind queries. hide-identity: yes # enable to not answer version.server and version.bind queries. hide-version: yes # Will trust glue only if it is within the servers authority. # Harden against out of zone rrsets, to avoid spoofing attempts. # Hardening queries multiple name servers for the same data to make # spoofing significantly harder and does not mandate dnssec. harden-glue: yes # Require DNSSEC data for trust-anchored zones, if such data is absent, the # zone becomes bogus. Harden against receiving dnssec-stripped data. If you # turn it off, failing to validate dnskey data for a trustanchor will trigger # insecure mode for that zone (like without a trustanchor). Default on, # which insists on dnssec data for trust-anchored zones. harden-dnssec-stripped: yes # Use 0x20-encoded random bits in the query to foil spoof attempts. # http://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 # While upper and lower case letters are allowed in domain names, no significance # is attached to the case. That is, two names with the same spelling but # different case are to be treated as if identical. This means calomel.org is the # same as CaLoMeL.Org which is the same as CALOMEL.ORG. use-caps-for-id: yes # the time to live (TTL) value lower bound, in seconds. Default 0. # If more than an hour could easily give trouble due to stale data. cache-min-ttl: 3600 # the time to live (TTL) value cap for RRsets and messages in the # cache. Items are not cached for longer. In seconds. cache-max-ttl: 86400 # perform prefetching of close to expired message cache entries. If a client # requests the dns lookup and the TTL of the cached hostname is going to # expire in less than 10% of its TTL, unbound will (1st) return the ip of the # host to the client and (2nd) pre-fetch the dns request from the remote dns # server. This method has been shown to increase the amount of cached hits by # local clients by 10% on average. prefetch: yes # number of threads to create. 1 disables threading. This should equal the number # of CPU cores in the machine. Our example machine has 4 CPU cores. num-threads: 2 ## Unbound Optimization and Speed Tweaks ### # the number of slabs to use for cache and must be a power of 2 times the # number of num-threads set above. more slabs reduce lock contention, but # fragment memory usage. #msg-cache-slabs: 4 #rrset-cache-slabs: 4 #infra-cache-slabs: 4 #key-cache-slabs: 4 # Increase the memory size of the cache. Use roughly twice as much rrset cache # memory as you use msg cache memory. Due to malloc overhead, the total memory # usage is likely to rise to double (or 2.5x) the total cache memory. The test # box has 4gig of ram so 256meg for rrset allows a lot of room for cacheed objects. #rrset-cache-size: 256m #msg-cache-size: 128m # buffer size for UDP port 53 incoming (SO_RCVBUF socket option). This sets # the kernel buffer larger so that no messages are lost in spikes in the traffic. #so-rcvbuf: 1m ## Unbound Optimization and Speed Tweaks ### # Enforce privacy of these addresses. Strips them away from answers. It may # cause DNSSEC validation to additionally mark it as bogus. Protects against # 'DNS Rebinding' (uses browser as network proxy). Only 'private-domain' and # 'local-data' names are allowed to have these private addresses. No default. private-address: private-address: private-address: # Allow the domain (and its subdomains) to contain private addresses. # local-data statements are allowed to contain private addresses too. private-domain: home.lan # If nonzero, unwanted replies are not only reported in statistics, but also # a running total is kept per thread. If it reaches the threshold, a warning # is printed and a defensive action is taken, the cache is cleared to flush # potential poison out of it. A suggested value is 10000000, the default is # 0 (turned off). We think 10K is a good value. unwanted-reply-threshold: 10000 # IMPORTANT FOR TESTING: If you are testing and setup NSD or BIND on # localhost you will want to allow the resolver to send queries to localhost. # Make sure to set do-not-query-localhost: yes . If yes, the above default # do-not-query-address entries are present. if no, localhost can be queried # (for testing and debugging). do-not-query-localhost: no # File with trusted keys, kept up to date using RFC5011 probes, initial file # like trust-anchor-file, then it stores metadata. Use several entries, one # per domain name, to track multiple zones. If you use forward-zone below to # query the Google DNS servers you MUST comment out this option or all DNS # queries will fail. # auto-trust-anchor-file: "/var/unbound/etc/root.key" # Should additional section of secure message also be kept clean of unsecure # data. Useful to shield the users of this validator from potential bogus # data in the additional section. All unsigned data in the additional section # is removed from secure messages. val-clean-additional: yes # Blocking Ad Server domains. Google's AdSense, DoubleClick and Yahoo # account for a 70 percent share of all advertising traffic. Block them. # local-zone: "doubleclick.net" redirect # local-data: "doubleclick.net A" # local-zone: "googlesyndication.com" redirect # local-data: "googlesyndication.com A" # local-zone: "googleadservices.com" redirect # local-data: "googleadservices.com A" # local-zone: "google-analytics.com" redirect # local-data: "google-analytics.com A" # local-zone: "ads.youtube.com" redirect # local-data: "ads.youtube.com A" # local-zone: "adserver.yahoo.com" redirect # local-data: "adserver.yahoo.com A" # local-zone: "ask.com" redirect # local-data: "ask.com A" # Unbound will not load if you specify the same local-zone and local-data # servers in the main configuration as well as in this "include:" file. We # suggest commenting out any of the local-zone and local-data lines above if # you suspect they could be included in the unbound_ad_servers servers file. #include: "/etc/unbound/unbound_ad_servers" # locally served zones can be configured for the machines on the LAN. local-zone: "home.lan." static local-data: device1.home.lan. IN A local-data: device2.home.lan. IN A local-data-ptr: device1.home.lan local-data-ptr: device2.home.lan # Use the following forward-zone to forward all queries to Google DNS, # OpenDNS.com or your local ISP's dns servers for example. To test resolution # speeds use "drill calomel.org @" and look for the "Query time:" in # milliseconds. # forward-zone: name: "." forward-addr: # OpenDNS forward-addr: # Google forward-addr: 2604:180:1:22a::8c53 # OpenDNS Append these lines to */etc/rc.conf.local*. Leave other lines untouched. # vi /etc/rc.conf.local unbound_flags="" Edit */etc/dhclient.conf* so it doesn't overwrite the local nameserver. Leave other lines untouched. # vi /etc/dhclient.conf ignore domain-name-servers; Enable and start *unbound*. # rcctl enable unbound # rcctl start unbound # DHCP Append these lines to */etc/rc.conf.local*. Leave other lines untouched. # vi /etc/rc.conf.local dhcpd_flags="vether0" Create */etc/dhcpd.conf* with these settings. I like to have static assignments for IPv4 addresses based on MAC addresses. Modify as needed. authoritative; option domain-name "my.example.com"; option routers; option broadcast-address; option domain-name-servers; default-lease-time 43200; max-lease-time 90000; subnet netmask { range; } host device1 { hardware ethernet 00:00:00:00:be:ef; fixed-address; } host device2 { hardware ethernet 00:00:00:01:be:ef; fixed-address; } Notice that the range for dynamic assignment does not include the IPs provided in static assignment. This is on purpose since otherwise dhcpd complains that the same address has been assigned twice. Enable and start dhcpd. # rcctl enable dhcpd # rcctl start dhcpd # Routing Enable IPv4 and IPv6 forwarding. Append these lines to */etc/sysctl.conf*. Leave other lines untouched. # vi /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1 Enable these settings. # xargs sysctl < /etc/sysctl.conf Append these lines to */etc/resolv.conf.tail*. Leave other lines untouched. Modify as needed. # vi /etc/resolv.conf.tail search my.example.com nameserver nameserver lookup file bind # rad OpenBSD 6.4 introduced *[rad](https://man.openbsd.org/rad.8)* as a replacement for [KAME](http://www.kame.net/) *rtadvd* (*radvd*). For OpenBSD 6.3, for example, read the section in this post about *rtadvd*. Configure *rad* by editing *[/etc/rad.conf](https://man.openbsd.org/rad.conf.5)*. In this instance, we only send out router advertisements over the *vether0* interface. # vi /etc/rad.conf interface vether0 Append these lines to */etc/rc.conf.local*. Leave other lines untouched. # vi /etc/rc.conf.local rad_flags= Enable and start *rad*. # rcctl enable rad # rcctl start rad # rtadvd (radvd) WARNING: [OPENBSD 6.4](https://www.openbsd.org/64.html) REPLACED [KAME](http://www.kame.net/) *rtadvd* (*radvd*) WITH ANOTHER DAEMON, [rad](https://man.openbsd.org/rad.8), THAT THE OPENBSD PROJECT WROTE. Configure *rtadvd* with an empty file in */etc/rtadvd.conf*. The reason is that it is able to use the delegated prefixes assigned by dhcp6c to network interfaces to advertise downstream. When I tried using a non-empty config file, IPv6 didn't work as expected. # truncate -s 0 /etc/rtadvd.conf Append these lines to */etc/rc.conf.local*. Leave other lines untouched. # vi /etc/rc.conf.local rtadvd_flags="vether0" Enable and start *rtadvd*. # rcctl enable rtadvd # rcctl start rtadvd # Firewall with pf Replace */etc/pf.conf* with these settings, updated as needed. I have curated these rules from many sources, primarily [The modern OpenBSD home router](https://www.azabani.com/2015/08/06/modern-openbsd-home-router.html). I also make no claim that these rules have created a secure firewall. # vi /etc/pf.conf ### ~~~ Interface layout ~~~ ### # cnmac0: 802.3ab (ethernet) to cable modem # cnmac1: 802.3ab (ethernet) to internal switch # cnmac2: 802.3ab (ethernet) to internal switch # vether0: persists address # bridge0: Ethernet bridge over cnmac1 and cnmac2 # pflog0: target interface for blocked packets ### ~~~ Constants and variables ~~~ ### # All addresses associated with this host self = "{ (egress), (vether0) }" self_lan = "{ (vether0) }" # RFC 6890: Special-Purpose IP Address Registries: # https://www.iana.org/assignments/iana-ipv4-special-registry/ # https://www.iana.org/assignments/iana-ipv6-special-registry/ # Included below are all address blocks with either Forwardable = False, # Global = False, or both, but excluding 2001::/23 because it is often # superseded by more specific allocations, as of 2015-08-05. table const { \, \, \, \, \, \, \, \, \, \, \, \, \, \, \ ::1/128, \ ::/128, \ ::ffff:0:0/96, \ 100::/64, \ 2001::/32, \ 2001:2::/48, \ 2001:db8::/32, \ fc00::/7 \ } ### ~~~ Default rules ~~~ ### # Never touch loopback interfaces set skip on lo # Normalise packets, especially IPv4 DF and Identification match in all scrub (no-df random-id) # Limit the MSS on PPPoE to 1440 octets # match on pppoe0 scrub (max-mss 1440) # Block all packets by default, logging them to pflog0 block log ### ~~~ Link-scoped services ~~~ ### # DHCPv6 client: make IA_PD requests and receive responses to them pass out quick on egress inet6 proto udp from (egress) to ff02::1:2 port dhcpv6-server pass in quick on egress inet6 proto udp to (egress) port dhcpv6-client ### ~~~ Bulk pass rules ~~~ ### # Pass all traffic on internal interfaces # vether0 is necessary here, but bridge0 is not pass quick on { vether0 cnmac1 cnmac2 } # Pass all outbound IPv6 traffic pass out quick on egress inet6 from { egress, (vether0:network) } modulate state # Pass all outbound IPv4 traffic from this host pass out quick on egress inet from (egress) modulate state # NAT all outbound IPv4 traffic from the rest of our network pass out quick on egress inet from (vether0:network) nat-to (egress) modulate state ### ~~~ Block undesirable traffic ~~~ ### # These rules must not precede the DHCPv6 client or NAT rules above block log quick on egress from { no-route, urpf-failed, } block log quick on egress to { no-route, } ### ~~~ Pass some ICMP and ICMPv6 traffic ~~~ #### # Pass all inbound ICMP echo requests pass quick on { egress vether0 } inet proto icmp icmp-type { echoreq, echorep } pass quick on { egress vether0 } inet6 proto icmp6 icmp6-type { echoreq, echorep } # RFC 4890: Recommendations for Filtering ICMPv6 Messages in Firewalls pass log quick on { egress vether0 } inet6 proto icmp6 pass log quick on egress inet6 proto udp from (egress) to fe80::d62c:44ff:fe pass log quick on egress inet6 proto udp from fe80::d62c:44ff:fe to (egress) ### ~~~ Open services on this router ~~~ ### # OpenSSH server pass in on egress proto { tcp, udp } to $self_lan port ssh # Allow LAN to access DNS, DHCP, and NTP pass quick on egress inet proto udp from (vether0:network) to any port { 53, 67, 123 } pass quick on egress inet proto udp from $self to any port { 53, 67, 123 } pass quick on egress inet6 proto udp from (vether0:network) to any port { 53, 67, 123, 546 } pass quick on egress inet6 proto udp from $self to any port { 53, 67, 123, 546 } Reload rules. # pfctl -f /etc/pf.conf # Ansible You can use Ansible roles I created for this post if you're so inclined. They're available on [GitHub - openbsd-on-erl](https://github.com/codeghar/openbsd-on-erl). # Conclusion I'm pretty happy with ERL and OpenBSD. There is great community documentation on how to configure all the pieces of software that make a OpenBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available. My plan on upgrading the base OS is to get a third party USB drive that works, write a newer OpenBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives. Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well.