Host an Intranet Site with pfSense and NAT - How to Home Lab Part 3

In this segment you will learn about setting up a pfSense firewall VM, port forwarding, VM templates, and DHCP reservation.

How to Home Lab

A tutorial series on building a complete home lab system from the ground up that is beginner-friendly, versatile, and maintainable.

Preface

In this segment, you will learn how to stand up a web app using some best practices and methodologies on your Proxmox VE host.

Proxmox provides some really great firewall features, unfortunately they don’t offer the configuration options we need for DHCP, DNS, and NAT, so we’ll be using pfSense to handle all of those.

Let’s start with NAT (Network Address Translation) and why we will be using it; you want to keep your home network and your lab network separated for a laundry list of reasons, namely security. The easiest way to do this without purchasing more hardware is to keep the lab network contained within your virtualization host (Proxmox), which has it’s benefits and drawbacks compared to a custom home network, that said, networking is a crucial component of home labbing and we will absolutely get to that in good time.

NAT is the process of shuffling internet protocol traffic from several hosts through a single gateway IP address, just like your home router does by default. Essentially any host behind the NAT gateway sends it’s outbound requests to the gateway, which forwards that traffic to its destination and keeps track of which private host made the request, when a response comes back the gateway forwards that back to the original host.

Note: NAT alone is NOT security! A crafty hacker can circumvent NAT, this is why we will be using pfSense to add a firewall to the private lab network.

We will utilize the DHCP and DNS features provided by pfSense in future articles to manage our VM guests on the private network instead of assigning them static IP addresses. The advantage to this approach is that when you have to make changes, say for example you set up a new version of some app and are ready to switch it over to production, it’s as easy as changing the configuration in pfSense instead of getting into all the relevant hosts and changing their static IP addresses.

Note: The firewall rules for protecting your home network from your lab network will be covered in a future article, this configuration alone is not secure for hosting to the open internet, we will get there soon, in the meantime have some fun with the journey and tinker around while you can still break things with minimal consequences!

Get pfSense

Head over to the pfSense download page and grab the latest release, you’ll want the AMD64 architecture, and the CD Image installer.

You’ll need to have some software installed to extract the archive, I use 7-zip for this because it adds a nice right-click menu option on Windows.

Once you have it in .iso format, you can upload it to your Proxmox host.

Configure VM Network

Before we set up the VM, we will need a private internal network on the Proxmox host. Head to Datacenter > (Name of your host) > Network, and click Create > Linux Bridge.

You must restart the Proxmox host before you continue so the new network will be available for pfSense to use.

Set up a pfSense VM

Create a new VM, if you check the advanced box at the bottom you will have the option to set up auto-start settings:

• Start at boot: Auto-start this VM when the Proxmox host boots up
• Start/Shutdown order: At boot time, the lower numbered VMs will start up first, and on shutdown/reboot the higher numbered VMs will shut down first. I like to start these in increments of 10 so you can slip new VMs in wherever they should go later, there is also nothing wrong with using the same number for multiple VMs, they will just be started/shut down at the same time.
• startup delay: The number of seconds to wait after booting this VM before booting the next VM.
• Shutdown timeout: The number of seconds to wait during a shutdown of the Proxmox host for this VM to finish shutting down, this prevents the system from hanging if a VM refuses to shut down, I usually leave it blank to use the default setting of 5 minutes.

You only need to add a comment for this network, all of the IP configuration will be handled by pfSense once we get it running.

In the OS tab, choose the ISO image for pfSense, and choose “other” for Guest OS.

Accept the default settings for the System tab, in the Hard Disk tab change the Bus/Device to VirtIO Block.

Always check the box for discard, this will free up space from files deleted on the VM guest, if unchecked the free space will be available to the VM guest but still marked as used on the Proxmox host and storage device

Under the CPU tab, set the CPU cores for this VM, 1 or 2 should be just fine, and set CPU units to 2048. CPU units give each VM a priority for CPU time, so VMs that are less critical can just use the default value of 1024, and critical VMs should get a higher value so they get priority if there is a high demand for CPU usage.

For memory, 512MiB should be just fine, you can always add more later if needed.

Under the Network tab, choose VirtIO for the Model, and leave the Bridge on vmbr0, we will add the new network after the wizard is finished.

Finish the setup wizard, then head over to Datacenter > (Name of your host) > pfSense > Hardware > Add > Network Device.

Choose vmbr1 for the bridge (our internal private network), and VirtIO for the model.

Install pfSense

Click the start button in the top menu bar, and then click the console tab on the left to run through the installer for pfSense, the default install options should be fine.

After rebooting the VM, you’ll be asked for the following input before bootup completes:

• Should VLANs be set up now? (n for no)
• Enter the WAN interface: (vtnet0)
• Enter the LAN interface: (vtnet1)

Configure pfSense

After the bootup is finished, choose option 8 and press enter to open a shell session.

We need to disable the packet filter (firewall) temporarily so we can access the web interface from the WAN side which is disabled by default, use the command pfctl -d to accomplish this, and note the IP Address of the WAN interface shown above the menu options, you’ll need it in the next step.

Note: You may get locked out a few times during the setup before we add a rule to allow the traffic, because changing certain settings will reload the packet filter and thus turn it back on, just go back to the console on Proxmox and run the pfctl -d command again.

From a web browser, visit https://(IP address of pfSense), in my case that is https://10.128.0.26, you’ll be greeted with the invalid SSL Certificate page, choose Advanced > Accept the risk and continue, or proceed to the website depending on which browser you are using. Then log in with the default credentials:

Username: admin Password: pfsense

Click through the first couple of pages, then choose a hostname, domain, and DNS servers. You can use the same domain as your Proxmox host or a new one, and any public DNS servers you wish.

The default time server is just fine, make sure you get your timezone correct though.

Configure the WAN interface with a static IP address within your home network subnet but outside of the DHCP range just like your Proxmox host, match the subnet mask and upstream gateway to your home network settings and click next at the bottom of the page, you can refer back to my Introduction to Networking article if you are unsure what to use for the network settings here and onward.

Next, configure the LAN IP address and subnet mask for pfSense, this is for the private internal network side, so it should be a new private network range that doesn’t already exist within your home network, I went with 172.16.44.1/24.

Let’s first set up an alias for the administration ports that pfSense uses, this way we can assign a rule to all of the ports at once instead of making a new rule for each port, we’re adding port 8080 to this alias because we’re going to move the admin interface to that port so we can free up ports 80 and 443 for regular web traffic behind NAT.

Head to Firewall > Aliases > Ports > Add

You can change the name and description values here to your taste, you can even leave the description for each port blank, just remember the name of the alias because we’ll be using it soon.

Now head to Firewall > Rules > WAN > Add

Note: Firewall rules in pfSense are applied to traffic going IN to the interface they are configured on, they do not apply to traffic going OUT of that interface, so just keep that in mind. In our use case, the WAN interface is your home network, and the LAN interface is your private server network that will only be accessible from VMs or through the pfSense firewall.

We want to add a rule to allow access from your home network to the admin interface of the pfSense firewall, here are the settings we need to accomplish that goal:

• Action: Pass
• Disabled: Unchecked
• Interface: WAN
• Address Family: IPv4
• Protocol: TCP
• Source
  • Network: Where the allowed traffic will come from, this should be your home network.
• Destination
  • Destination: This firewall (self)
  • Destination Port Range
    • From: pfSense_Admin_Ports (the alias we created earlier)
    • To: pfSense_Admin_Ports (the alias we created earlier)
• Extra Options
  • Description: This is just to help you identify rules, it’s helpful when you have a long list of rules on an interface

For the remainder of this article, when you see this message after changing something just go ahead and click Apply Changes, it is displayed because there are some cases where you may need to change several settings before applying so all the changes get applied at once, but we aren’t doing anything like that today.

Next, go to Interfaces > WAN, and uncheck the two boxes at the bottom for “Block private networks and loopback addresses”, and “Block bogon networks”, save and apply (We want to allow our local private network to access the WAN interface).

You should now be able to turn the packet filter back on and still access the admin page for pfSense, so back in your console session on Proxmox, run the command pfctl -e to enable the firewall again.

Let’s get rid of this warning about changing the password really quick, go ahead and click “Change the password…”

You know the drill, enter a new password twice and click save at the bottom of the page.

pfSense recommends disabling the large receive offload (LRO) setting when it’s running on a VM (if the admin interface feels slow, this should help that too), head to System > Advanced > Networking, check the box “Disable hardware large receive offload”, and click save.

As I mentioned, we need to move the admin interface to port 8080, head to System > Advanced > Admin Access.

Change the value “TCP Port” under “webConfigurator” to 8080, and click save at the bottom of the page. It should redirect you to the new port automatically after a short delay for the configuration to be applied. If it doesn’t just manually type in the URL bar of your browser https://(IP address):8080.

Set up a Template Server VM

If you don’t already have an ISO image for Ubuntu server, go ahead and grab the latest release from their download page , and upload it to your Proxmox host.

You are free to use any OS distribution you choose, I use Ubuntu here because it is a popular choice and there are many resources available for help if you need them.

I am going to show you how to use a VM template, essentially you will install an OS and any extra software you want to be on it like qemu-guest-agent (explained later), then convert the VM to a template. When you clone the template to a new VM you get to skip all of that configuration. There are some benefits and drawbacks to this approach so feel free to skip this step if you so choose (but do install qemu-guest-agent on all VMs that you can). Another option is to use a template VM without actually converting it to a template, you will lose the ability to create a linked clone and all the benefits that come with that feature, but you will be able to make changes and install updates on the template since the clones aren’t linked.

• Benefits of linked clones
  • Save disk space: the base OS files aren’t copied to the clone, they are read from the template instead, if they are changed in a clone a new copy will be made just for that clone
    • Save on RAM usage: Kernel Same-page Merging (KSM) works best when there are more common resources shared betweenVMs (more on KSM in my ZFS Memory Tuning article)
• Drawbacks of linked clones
  • Templates cannot be deleted or modified if any clones exist, so you can’t install updates or add configuration to the template for future clones, you’d have to make a new template, this can lead to many templates cluttering your VM list as OS releases come out and your pre-configuration grows
    • KSM can increase the attack surface for side-channel memory attacks

Add a new VM, here are the settings I used:

• General
  • VM ID: 1001 (I like to keep templates at the bottom of the VM list)
  • Name: Ubuntu-18.04.3-20190824 (I also like to put the creation date in the VM name)
• OS
  • CD ISO: ubuntu-18.04.3
  • Guest OS: Linux/5.x
• System
  • Defaults
• Hard Disk
  • Size: 8 GiB (you could even go down to 4 GiB, more space can be added to clones if needed)
  • Discard: checked (always check this box as explained above)
• CPU
  • Cores: 2 (this can be changed on the clones as needed)
• Memory
  • MiB: 512 (this can also be changed on the clones)
• Network
  • Bridge: vmbr1 (use the private VM network)

Run through the installer as usual, making sure that when you get to network connections that a DHCP address is assigned in the network range you set up for the LAN network in pfSense, if it is not correct here something has gone wrong (are you one the vmbr1 network?).

I also chose Use an entire disk and set up LVM during the install, this makes adding disk space later a lot easier.

When prompted, remove the installation media from the VM.

After the reboot open a console on the new VM, and run the following commands to install all available software updates.

sudo apt update
sudo apt -y dist-upgrade

Troubleshooting:

I got a bunch of errors when I ran the update command, I’m not entirely sure if it was due to running pfSense on a VM, or because of double NAT (my lab pfSense is running behind my production pfSense firewall).

If you get errors here like I did, try running the command ping google.com, if you get responses, try the command curl -I google.com, if that fails but the ping command works fine, go back to your pfSense admin page and head to System > Advanced > Networking, check the box Disable hardware checksum offload and click save, this got me up and running.

On your template VM console session in Proxmox, run the command sudo apt install -y qemu-guest-agent, this agent helps the Proxmox host communicate with the VM guest and lets the host run commands on the guest, for example running the shutdown command (if you use the shut down button in Proxmox on a VM without the guest agent, Proxmox just tells the VM that the power button has been pressed, but running the shutdown command with guest-agent is much cleaner and more reliable). When the installation is finished, go ahead and shut down the VM.

We need to tell Proxmox that the guest-agent is installed on this VM, head to (Name of VM) > Options and double click on Qemu Agent, check both boxes and click OK.

Running guest-trim after cloning a disk will mark free space on the virtual disk as free on the storage device, this is generally desired behavior.

Lastly, right-click on the name of your template VM and choose convert to template.

Create a Web Server VM

Right-click on your template VM and choose Clone.

Choose a name for the new VM, and use Linked Clone for the Mode.

Head to (Name of your VM) > Options, and double-click Start at boot, choose 20 for the order so it will boot up only after pfSense has booted, then double-click Start at boot and check the box for yes.

Under the Hardware tab, double-click on Network Device, and copy the MAC address. You can paste it into notepad for now, we will need it later to set up a DHCP reservation.

A MAC address (Media Access Control) is a unique identifier assigned to every network interface on a machine.

Start the VM and switch to Console until it gets to the login prompt, then switch to the Summary tab, you should see an IP address listed that is within the network range of the LAN interface on pfSense, if you see the address that means the qemu-guest-agent is installed and working properly, nice!

Switch back to the console for the next step.

Boilerplate

Anytime you clone a VM, you should run the following commands to ensure the new system is seen as a unique machine on the network and maintain some entropy for SSH if it is installed (it should also receive a unique MAC address for all network interfaces when you run the clone operation). All of the commands below should be run as the root user, this can be achieved on Ubuntu by preceding each command with sudo, or first running the command sudo -i to start a root session.

These commands are specific to Ubuntu, consult your distribution’s documentation if you are not using Ubuntu. What we are doing is deleting the machine-id from both locations /etc/ and /var/lib/dbus, creating a new machine-id in /etc/ and symlinking it to /var/lib/dbus.

rm -f /var/lib/dbus/machine-id
rm -f /etc/machine-id
dbus-uuidgen --ensure=/etc/machine-id
ln -s /etc/machine-id /var/lib/dbus/

These commands are only required if SSH is installed, if they fail it is probably not installed and you can safely skip these.

ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa -y
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa -y
ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -b 521 -y

Note: The newer Ubuntu releases have cloud-init enabled by default which can interfere with host-name changes. In my opinion cloud-init has great potential, but it’s not quite ready for prime-time yet. I recommend running the command sudo touch /etc/cloud/cloud-init.disabled to disable it for the time being.

These commands change the hostname and all references to it, replace ubuntu-18-04-3 with the hostname of your template VM, and nginx with the desired hostname of your new VM.

sed -i 's/ubuntu-18-04-3/nginx/g' /etc/hosts
sed -i 's/ubuntu-18-04-3/nginx/g' /etc/hostname
hostnamectl set-hostname nginx

The VM needs to be rebooted to complete the changes, but let’s hold off for now since we’ll need to reboot after configuring the DHCP reservation in pfSense.

Install NGINX

Run the command sudo apt install -y nginx, NGINX ships with a simple landing page by default to let you know it’s working, so that’s all there is for this step.

DHCP Reservation

Assigning a DHCP reservation means that when the configured host boots up and requests an IP address from the DHCP server (on pfSense), it will be given the specified address instead of getting an address from the DHCP pool which may not always be the same on the next reboot (the DHCP pool is a range of IP addresses that will be automatically handed out to new hosts).

From your pfSense admin interface, go to Services > DHCP Server.

Before we can assign a reservation, we need to change the address pool size, pfSense will not allow you to reserve an address that is within the DHCP pool. Change the values for Range under General Options to leave some addresses open for reservations, I changed my range from 172.16.44.1 - 172.16.44.254 to 172.16.44.1 - 172.16.44.100, leaving the range 172.16.44.101 - 172.16.44.254 for reservation assignments.

At the bottom, click the Add button under DHCP Static Mappings.

Paste in the MAC address for your NGINX VM that you copied earlier, I usually use the hostname as the client identifier (just make the client identifier is unique for every host on the network).

Choose an IP address outside of the DHCP pool, and add an optional description, then save and apply the changes.

Configure Port Forward

Let’s add a new alias for the NGINX server, I like to use aliases whenever possible because it makes it easier to change things later, updating an alias will update all the associated NAT and Firewall rules along with it.

Head to Firewall > Aliases > IP, and click that Add button.

Make sure to use your own IP address here instead of mine

Let’s also add an alias for web ports (80 and 443), click the Ports tab and then the Add button.

Now head to Firewall > NAT > Port Forward, and click the Add button.

For the port forward rule, choose these options:

• Interface: WAN (forward requests to this port coming from the WAN interface, which is your home network)
• Protocol: TCP
• Destination: WAN address (forward requests destined for the IP address of pfSense on your home network)
• Destination port range:
  • From/To: Web_Ports (the alias we just created)
• Redirect target IP: nginx (the alias for the NGINX server)
• Redirect target port: Web_Ports (alias)
• Description (optional): Forward HTTP(S) traffic to NGINX Server

Click save, and apply the changes.

Back in your console session in the Proxmox web interface, reboot the NGINX VM to apply all the changes, it should get the new DHCP reserved IP address when it boots up again (you can use the command reboot).

After a minute or two, put the IP address of your pfSense firewall in your web browser URL bar (http://(IP address)) without the :8080, because that port is for the pfSense admin interface, and without the s in http because we didn’t set up SSL on the NGINX server yet. You should see the default NGINX page, which means your NAT Port Forward is working as expected.

In case you didn’t already know it, when you visit a web page at http:// in your browser the request goes to port 80 using HTTP protocol (non-encrypted traffic), and https:// will go to port 443 and use HTTPS protocol (encrypted), adding a colon after the address will manually set the destination port but the protocol is still selected by the presence or absence of the letter s after http.

Serve an App on NGINX

If you intend to tinker with this while I work on the next segment in this series (and you absolutely should tinker!), you may want to take a backup or snapshot of the NGINX VM so we can restore it to a clean state later when we set it up as a reverse proxy.

Let’s put something useful on our NGINX server just for fun, I’ve chosen a mind-mapping app just because it’s super easy to get installed, but feel free to try other things and experiment.

Back to the NGINX server console on your Proxmox host, run the following commands as root to download the project into the default /var/www/html directory and set the correct permissions on the new files.

Here ’s a great explaination of linux file permissions from linux.com

apt isntall -y git
cd /var/www
rm -rf html
git clone https://github.com/ondras/my-mind.git html
chown -R www-data. /var/www/html
find /var/www/html/ -type d -exec chmod 755 {} \;
find /var/www/html/ -type f -exec chmod 644 {} \;

Now point your browser back to the WAN IP address of your pfSense firewall (http://(IP address)), and you should see the mind map app running.