In this article we will migrate pfSense to be the edge device on the network.
A tutorial series on building a complete home lab system from the ground up that is beginner-friendly, versatile, and maintainable.
Introduction
Hello again! It’s been a while since my last article but I’m still at it, hopefully, you’ve had time to acquire some hardware for your pfSense instance, in this segment we’ll be moving pfSense to the edge of your network to realize a ton of benefits that are only possible in this configuration.
Why? Well, if you’re still here, then you’re clearly invested in this endeavor, so it’s time to take it to the next level by taking full ownership of your home network. This is, in my opinion, too big of a commitment to have put on you early in the series, it would’ve been easier to do it first, but much harder to clean up if you decided it wasn’t for you. A home lab is more than just a server, it involves the whole network.
You will need at least one machine to run pfSense with two or more network interfaces, an empty USB stick, and I highly recommend a layer 2 managed switch with VLAN support (an affordable choice to start out would be a Netgear GS308T), a standard router or unmanaged switch could be used in it’s place, but not a modem/router combo device because we want everything behind pfSense. You’ll also want a dedicated WiFi router to put behind pfSense.
IMPORTANT: if you’re using your modem as a WiFi router please skip this segment, it won’t work correctly with pfSense and it will be insecure if you bridge your modem.
Here are three of the many possible physical layouts of the network, from most desirable to least. In this sense “Devices” means Proxmox, workstations, your toaster, or anything else. The important bit is that only pfSense is plugged directly into the modem.
Prepare Installation Media
We will start by taking a backup of the current pfSense installation, head to Diagnostics > Backup & Restore > Download configuration as XML (don’t use encryption for this one!).
Next, we’ll need a fresh installer for pfSense and a USB stick, head to the pfSense download page and choose the appropriate architecture (AMD64, unless you have official Netgate hardware), and “USB Memstick Installer” for the installer option, and “VGA” for the console.
If you are on Mac or Linux, you can use the dd command to burn the installer to your USB stick, for Windows I recommend Balena Etcher.
Notes: The filename of the installer may have changed since the time of writing, and make sure you put the correct disk path for your USB drive instead of /dev/sdXX, you can list the available disks with the command lsblk.
sudo dd \
if=pfSense-netgate-memstick-vga-2.4.5-RELEASE-p1-amd64.img \
of=/dev/sdXX \
bs=4M
Once that’s done, you should have a FAT32 partition on the USB stick, copy your configuration backup to that partition and rename it to config.xml, we can restore the configuration during installation with this file.
Provision pfSense
We’ve done an install before, and this shouldn’t be much different, but you should see an option to restore the configuration from config.xml during the installation, choose yes here.
After the installation, you should plug a computer into the LAN port and connect to the web interface and make sure everything looks kosher, then go ahead and connect the WAN port to your modem.
VLANs
What is a VLAN? It stands for virtual local area network, you can just think of it as a separate network. Even though you will have all your devices plugged into the same physical network, devices on different VLANs will need to communicate through the pfSense firewall, allowing much better control over network rules.
Maybe you don’t want your housemates to have access to your VMs? Easy, just put them on their own VLAN, assign your machine a DHCP reservation on the LAN network, and create a firewall rule to allow only that IP address to communicate with the VLAN with your VMs on it. What about a separate VLAN for your kids (or whoever) that blocks certain websites or blocks the internet completely outside of designated hours? That can all be done with pfSense and VLANs.
If you don’t have a managed VLAN switch you can skip this part, but if you do it would be best to create a new VLAN interface in pfSense for your servers and leave the LAN network for your workstation, maybe add some VLANs for your WiFi or IoT devices to keep them separated. You’ll need to copy your LAN rules to the server VLAN, I also made a “DMZ” VLAN for VMs that are exposed to the internet, and specific rules for how they can communicate with the other VMs on my “Production” VLAN. So, how do we do that?
In pfSense, head to Interfaces > Assignments, then click on the VLANs tab, then the plus sign to add a new VLAN. Choose the parent device (the LAN port, which will be plugged into your switch later on), a VLAN Tag number, I’ll use 11 for this example (don’t use VLAN 1 though, it’s the default VLAN on most switches, and is therefore vulnerable to VLAN hopping). I usually leave the VLAN Priority blank because I don’t use any traffic shaping over VLANs, and give it a description.
Back over to the “Interface Assignments” tab, we can select the new VLAN next to “Available network ports”, and click the Add button. It’ll show up now as OPT1, if you click the interfaces tab again and then OPT1, you can change the name to something like “Prod”, or whatever you like. Go ahead and assign an IPv4 address space for this VLAN, and there’s no need for a gateway since pfSense will be the gateway for this VLAN.
Next, head over to Services > DHCP Server, and choose the tab for the new VLAN you renamed, we just need to enable DHCP and set the address pool range. Then head to Firewall > Rules, and just add an allow any rule for now, you can fine tune it as needed later on.
Now we can plug the switch into your workstation and get that configured. Since there are so many manufacturers and brands they’re all different, some are command line, some have a web GUI, but if you understand the concepts you just need to look up the specific commands or menus to configure it. The Netgear GS308T I recommended has a pretty decent GUI, the relevant settings here are under Switching > VLAN.
I’ll just cover the basics to keep this concise, the main thing you need to know is tagged vs untagged. Let’s say you have an internet data packet coming from pfSense destined for a host on VLAN 11, this packet is “tagged” for VLAN 11, the switch sees the tag and compares that with its own VLAN table to determine where to route the traffic, that’s easy enough. Untagged traffic is just that, it has no tag.
From the switch’s point of view, you can have tagged and untagged ports, by default all ports are usually untagged on VLAN 1. If a VLAN is untagged on a port, that means that all traffic going in or out of that port that is untagged will go through that VLAN. If a VLAN is tagged on a port, then traffic tagged for that VLAN will be allowed through the port. If a VLAN is not tagged or untagged on a port, no traffic for that VLAN will be allowed to pass through the port, it will be routed through pfSense instead.
So for our example, let’s say we have port 1 on the switch as the uplink to pfSense, and we want port 2 to be treated entirely as it’s own network on VLAN 11, we need to make the following changes:
- Remove untagged VLAN 1 from port 2
- Add untagged VLAN 11 to port 2
- Add tagged VLAN 11 to port 1
Now, traffic for a host on VLAN 11 can be passed from any LAN port through port 1 into pfSense, which will tag the packets for VLAN 11 (based on the destination IP address) and send them back through port 1 on the switch, which will route the traffic out port 2 to the host, and traffic originating from port 2 will be tagged for VLAN 11 by the switch when it passes through, going back out to pfSense on the VLAN 11 interface (since only port 1 is tagged for VLAN 11), which can then send it back to the switch untagged, and the switch will forward it to the target host on the LAN network.
To complete the package, you can add VLAN tags to your VMs in Proxmox by editing the VMs network interface in the hardware tab (after making the network VLAN aware in Host > System > Network > vmbrN). Ideally, you would have Proxmox plugged into a port on the switch that is untagged for a management VLAN (to access Proxmox), and allow tagged VLANs on that port for any VLANs you’ll put your VMs on (just make sure you don’t run any VMs without a VLAN tag, or they’ll have access to the Proxmox admin interface!). Personally, I have a Production VLAN for databases and web hosts, a DMZ VLAN for any host that is open to the internet, a LAB VLAN for playing around with things, and a handful of VLANs to separate workstations and devices on my LAN.
Migration
Take your time here, things can get messy really fast if you don’t take your time and think things through as you go, I would suggest planning it out in advance as well, walking through each step and considering anything that might become an issue. I’ll try to provide a broad overview here that should cover everything we’ve set up in the series so far, but keep in mind that there are so many considerations here that I can’t account for everything, don’t hesitate to reach out if you hit a snag!
I would start by plugging the pfSense device into the modem, then your switch into the LAN port on pfSense, and everything else on the network except Proxmox into the switch. Make sure everything is working as expected before moving Proxmox over.
Your WiFi router should be plugged into either the LAN port on pfSense, or the switch that is plugged into the LAN port on pfSense, and should be put in DHCP passthrough mode. If it doesn’t have a DCHP passthrough mode, just don’t use the WAN port on the WiFi router, plug the uplink cable into one of the LAN ports instead.
DHCP Passthrough mode: disables DHCP server on the router and assumes there is a DHCP server somewhere else in the network (pfSense), should also disable NAT or have that option.
Next, you may need to change the IP address and gateway for the Proxmox host to fit into the pfSense network or VLAN you created, this can be done at Host > System > Network > vmbr0, don’t forget to check the “VLAN aware” box if you’ll be using VLANs. Save the changes, and shutdown Proxmox before moving on.
If you don’t need to change the IP address of Proxmox or enable VLANs, you can just move the network cable to your switch now. Boot up Proxmox if needed and again, make sure it’s working, port forwards won’t be working yet, but you should have access to the Proxmox admin page.
Now we can go to each VM, under Hardware, and change the network device to use the main (vmbr0) interface (add your VLAN here if you’re using them as well), and shut down the pfSense VM in Proxmox. Check that the VMs are getting the expected IP addresses, and are accessible by IP address.
Finally, remove the domain overrides in your workstations hosts file, these should now be routable on the network because pfSense is managing DNS for the whole network. As for the hosts or domains that are behind a reverse proxy such as NGINX, head to Services > DNS Resolver on your physical pfSense host, and add a host override for each domain that points to the reverse proxy, and they should be accessible without any manipulation of the hosts file on your workstation.
Bridge Modem
IMPORTANT: Please do not bridge your modem unless the only device plugged into it is the pfSense machine or some other firewall.
This is an optional step, and again, there are many different models of modem, but you should be able to find instructions by searching “(your modem name) enable bridge mode” or something similar.
Bridging your modem just puts it into “dumb” mode, it connects to the internet and that’s all. when you plug pfSense into it, pfSense gets a public IP address directly from your ISP. This is handy in a lot of ways, most importantly it eliminates the modem as a configuration point.
Since they’re usually rented from an ISP, and by default are configured to set up a basic NAT network that’ll be safe-ish for consumers, we don’t need any of that, and we don’t need to care about the modem’s configuration.
I’ve also found that I can plug an extra port on my pfSense box into the modem again and set up dual WAN with different public IP addresses (this may be limited by some ISPs), which is very handy for playing around with mail servers and such.