In this segment you will learn how to set up and use a secure remote access server for your home lab network.
A tutorial series on building a complete home lab system from the ground up that is beginner-friendly, versatile, and maintainable.
Preface
Alright, this should be the last segment in the series before we broadcast our pirate signal and hack into the Matrix go live and host to the open internet. What we are doing here is creating a dedicated endpoint for remote access, and adding a reasonable amount of security to it. This gives us remote access to an actually usable console session into other guest VMs so we can do some great things like copy and paste for example.
We will be using SSH with a locked down configuration to allow access only with a key, no passwords allowed, and also configuring Fail2Ban as an added security measure. I will also mention some other tactics you can use to lock down your SSH server.
At the end of this segment, you will have access to your home lab network “remotely” from within your home network, you will not have access from outside of your home network until we set up the system to be open to the internet (likely in the next segment). I know, bad choice of words, but we are setting this up now so it will be ready once we do go live.
Set up an Access Point VM
Create a new linked clone of your template VM, I’ll call mine ap for Access Point, you can set up your VM ID’s however you want or just accept the default values, I like to have a little organization in my VM list so I’ll use 500 for this one.
Remember not to use ID’s lower than 100!
Networking
Just as in previous segments, we’ll need to set up a static IP address and NAT port forward for this VM, first grab the MAC address.
You’ll need to log in to your pfSense VM (https://ip-address:8080), if you forgot the IP address just open a console in Proxmox and pfSense will have the WAN IP address displayed on the welcome screen.
Head to Status > DHCP Leases, and take a look at what static addresses are in use to determine the next available address, in my case that would be 172.16.44.103, so that’s what I’ll be using.
Next, head to Services > DHCP Server, click the Add button, fill in the form, then save and apply the configuration.
We’ll create a new alias for this VM - Firewall > Aliases > IP > Add.
Finally, we’ll add the port forward - Firewall > NAT > Port Forward > Add.
Now all traffic going to the WAN address of pfSense on the SSH port (22) will be forwarded to our ap VM.
Why not just use SSH on pfSense? It’s best to have SSH available to the open internet only on a dedicated endpoint, in the unlikely event that your SSH endpoint is compromised you can simply disconnect the network from this VM in Proxmox, which allows you to stop the threat and keep the VM running while you investigate, without interfering with production services. You will still be able to SSH into pfSense, you just have to go through the access point first.
Boilerplate
Are you sick of this part? Me too, I built a little bash script to automate the process of generating a new machine ID and SSH server keys, and setting a new hostname. You can use it by starting a console session in your new VM (through Proxmox) and running the following command which will run the script straight from GitHub. (Won’t it be nice to have copy and paste when we finish this segment?).
sudo bash -c "bash <(wget -qO- https://raw.githubusercontent.com/dlford/ubuntu-vm-boilerplate/master/run.sh)"
You will want to restart when prompted because the VM needs to pick up the new IP addresses we reserved for it.
Note: Always use caution running remote scripts, or any scripts at all for that matter, especially as root, at the very least you should read through it to make sure it’s not doing anything untoward.
The source code for run.sh is available here so you can review it:
A simple bash script to handle boilerplate configurations for cloned Ubuntu VMs (Machine ID, SSH server keys, Hostname) - GitHub - dlford/ubuntu-vm-boilerplate: A simple bash script to handle boile...
SSH Configuration
Let’s first check that SSH Server is installed and running on the new VM, run the command systemctl status sshd, you should get something like the image below.
If you don’t, you’ll need to run the following commands.
Note: Skip this start command for now if you already have port 22 open to the internet, run it at the end, after the SSH configuration is secured
sudo apt install -y openssh-server
sudo systemctl enable sshd
sudo systemctl start sshd
We will come back to this VM to secure the SSH server after we set up an SSH key on your workstation.
SSH Key Pairs
I’d like to explain SSH to you in explicit detail, but Digital Ocean has already done a great job at it, so here’s a link if you are interested.
SSH, or secure shell, is an encrypted protocol used to communicate with remote servers safely. The practical uses of SSH are widely discussed in other guides. In this article, we'll examine the underlying encryption and connection procedures that make
For security, you should only create keys on the machine that will be using them, don’t create SSH keys on the server, or any other machine, and then transfer them to a new machine. If you have multiple workstations, create multiple keys, if you get rid of a workstation or re-install the OS, just create a new key (remove access for the old one from your server too!).
Create a Key Pair
From your workstation, run the following command from a terminal to create a new key pair.
If you are on Windows 10 run this from PowerShell, if you get a “no such command” error, you need to install the SSH client. Open the Settings app and search for “Manage Optional Features”, scroll down to “OpenSSH Client”, and click Install.
ssh-keygen -t rsa -b 4096 -C "yourname@devicename"
You will be prompted for a directory to store the key files in (press enter to accept the default location), and an optional passphrase which I recommended using at the very least on any device that ever leaves your home (you’ll need to supply the passphrase to use the key).
This should create a new 4096 bit rsa key pair with the comment yourname@devicename, the comment is nice for when you need to de-authorize a key from the server, you’ll know the user and device that has that particular private key.
You can see the files by running the command ls .ssh, the file id_rsa is the private key that should never leave this machine, and id_rsa.pub is the public key, which should be installed on any server that the corresponding private key will have access to.
Manage SSH Key Access
There is a handy tool for copying public keys to a server, run the command below to copy your new SSH public key to the ap VM.
username refers to the username on the remote server, not your local workstation, and the IP address should be that of your pfSense VM (which will forward the traffic to the ap VM).
ssh-copy-id username@ip-address
You will see a warning about The authenticity of host ... can't be established, this happens because the SSH server on the ap VM has a server key that is unknown to your workstation, choosing yes will continue connecting and save the servers key on your workstation. If this were a server you’d connected to before it should be concerning to get this warning, as it may be evidence of some foul play, for example a DNS hack to steal your SSH key by impersonating the server you were trying to connect to, fortunately the SSH client will flat out refuse to connect if there is a server key mismatch (a mismatch would be a situation where your workstation’s known hosts file has an entry for the target server but the saved key does not match the server’s key).
You’ll be asked for the password for the remote user, and then your public SSH key will be transferred to the server. Now run the command ssh username@ip-address to connect, if you entered a passphrase when creating your SSH key you’ll be asked for it here, otherwise you should be immediately connected.
If you check the .ssh directory on the ap server, you will see a file named authorized_keys, this is where all public keys are stored for this user, one per line. You can also add and remove keys directly to/from this file, if you run nano .ssh/authorized_keys and press the end key on your keyboard to jump to the end of the line, you will find the comment you entered when creating the key pair to identify the public key, pressing ctrl + k will “cut” that line in the event you wanted to remove a key (side note: ctrl + u will “paste” the previously cut line in the nano text editor).
Secure the SSH Server
- Fact: Hundreds if not thousands of bots run 24/7 connecting to any SSH server they can find and trying to log in with common usernames and passwords (this is actually called a word list attack, but is commonly lumped in with the term brute forcing, which is technically the process of trying every possible combination of characters one by one, from here on I’ll join the masses and use the term brute forcing to refer to either of the two)
- Fact: Just having the SSH port
22open to the internet will result in brute force attempts on your server - Fact: If a brute force bot or unauthorized human successfully logs in to your server, you are going to have a really bad day
We will implement the three most effective measures against SSH brute force attacks:
- Disallow the
rootuser from logging in remotely - Disallow any user from logging in via password (SSH keys only)
- Rate limit any failed login attempts with Fail2Ban.
There are additional tactics that may be employed, I won’t cover them here, but if you are interested please let me know, I may consider writing an article on a few of them if there is any interest. I feel the above three are a good balance of security and convenience for the average system, but I will list a few more tactics just to be thorough.
- Disable SSH (The most secure option, no remote access at all)
- 2FA (Two Factor Authentication)
- Whitelisting (Only allow access certain IP addresses, this could be easily implemented in your pfSense VM)
- *Port knocking (Only allow access temporarily to any IP address that has sent a series of packets to specific ports in sequence)
- *Change the SSH port to something other than
22
*These last two tactics do NOT provide any security at all, I cannot stress that fact enough, they are obfuscation techniques. While obfuscation (hiding the server) has its benefits in my opinion, namely in filtering out most bots and reducing server log spam, it is an important distinction to make that these tactics are easily bypassed if they are discovered, and it’s quite feasible that they could be uncovered by a bad actor. There are many who believe obfuscation to be bad practice due to a false sense of security, but I feel that they can be beneficial when used responsibly.
More SSH Configuration
Run the command sudo nano /etc/ssh/sshd_config to open up the configuration file, you’ll want to modify or add the following lines.
Don’t forget to replace your-username with your actual username on the server.
LoginGraceTime 2m
PermitRootLogin no
StrictModes yes
MaxAuthTries 1
AllowUsers your-username
HostbasedAuthentication no
IgnoreUserKnownHosts no
IgnoreRhosts yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM no
Restart the SSH service with the command sudo systemctl restart sshd to load the new changes from this file.
Note that from here on, you can’t use the ssh-copy-id command for new SSH key pairs since logging in via password is no longer allowed, you must manually add them to the authorized_keys file on the server or change the PasswordAuthentication setting to yes and restart the SSH service.
If you are curious, here is a link to the man (manual) page for sshd_config that explains all of these options in detail.
sshd(8) reads configuration data from /etc/ssh/sshd_config (or the file specified with -f on the command line). The file contains keyword-argument pairs, ...
Set up Fail2Ban
Fail2Ban is an incredibly handy tool, it scans log files for failed logins (or other undesired behavior) and dynamically blocks IP addresses for a configured amount of time, effectively rate limiting any brute force attempts.
Run the command sudo apt install -y fail2ban, then sudo nano /etc/fail2ban/jail.local to create a new configuration file.
if the file jail.local is not found, Fail2Ban will use the default configuration file jail.conf
Fail2Ban is very powerful and has a ton of configuration options, but lets keep it simple and just configure one jail.
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
findtime = 3600
bantime = 300
maxretry = 2
Now to enable Fail2Ban run the following commands:
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
Here’s a break down of the options in use:
[sshd]: The name of this jailenabled = true: Enforce the rules of this jailport = 22: Ban jailed IP addresses from accessing port22filter = sshd: The filter file contains RegEx patterns for the target log file, essentially telling Fail2Ban which log entries are failed login attempts and where to get the IP address from the log entry. Filter files are located in/etc/fail2ban/filter.d, there are many shipped with the package or you can write your own.logpath = /var/log/auth.log: Full path to the log file to watchfindtime = 3600: This is the window of time that hits from the log file will be counted, measured in secondsbantime = 300: The amount of time to ban an IP address, measured in secondsmaxretry = 2: The number of repeated attempts to allow before banning an IP address
Effectively, this jail will ban an IP address from accessing port 22 for five minutes if it has made more than two failed attempts to log in within one hour. The count is cumulative, so if an address was banned for five minutes and makes another failed attempt right after, the count would already be exceeded and it would be banned again for another five minutes.
I like using a long find time and short ban time because it sets a hard rate limit of 12 attempts per hour (after the first ban), which is slow enough severly impact a brute force attack, but the ban time is short enough that if you goof up you don’t have to wait too long before trying again.
If you were so inclined, you could create a second jail to cover the really persistent attacks that just don’t know when to quit, for example to ban an address for a week if it makes more than 19 failed attempts within a week (at a maximum of 12 attempts per hour from the previous jail, it would take around an hour and a half to get 19 failed attempts, so this jail would ban an address for a week if they persistently attack the server for over an hour and a half). That rule would look like this:
[sshd-persistent]
enabled = true
port = ssh
logpath = /var/log/auth.log
filter = sshd
bantime = 604800
findtime = 604800
maxretry = 19
Since this jail needs to remember events for longer than the amount of time that Fail2Ban keeps track by default, we’ll need to change that default. Run the command sudo nano /etc/fail2ban/fail2ban.conf, and edit the line dbpurgeage = 1d to read dbpurgeage = 8d. Note that this will require more resources for Fail2Ban to run.
You’ll need to run sudo systemctl restart fail2ban to load the new jail and the configuration change.
There is also a “recidive” jail that you can configure, which does almost the same thing as above by banning persistent addresses for a long period of time, the difference is that it doesn’t look at the log file for a specific event like a failed login, it looks at Fail2Ban’s own log file for ban events, essentially if an address has been banned x number of times in y window of time, ban it for even longer. I like the control of setting this up for each jail individually for each service though.
Test Your Setup
We want to make sure everything works, right? Open up a console session to your ap VM in Proxmox, and run the command sudo apt install -y lnav, which is a tool for navigating log files. Once that’s done go ahead and run lnav /var/log/fail2ban.log so we can monitor the file (you may need to tap the right arrow key on your keyboard to scroll the text into view, the date stamps are very long).
From your workstation run the command ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no username@ip-address, this tells the SSH client on your machine to try to log in with a password instead of your SSH key. You should see the Permission denied (publickey) error, that’s what we want to see, password login is disallowed. Now try ssh root@ip-address, you should see the same error, but also on your lnav window you should see [sshd] Found ip-address... with your workstations IP address. Try to log in as root again from your workstation and you should see [sshd] Ban ip-address in the lnav window, try it a third time and you will find that the connection times out because fail2ban has blocked the request.
Nice, everything works, but now you’re locked out! Let’s fix that, in your Proxmox console session to the ap VM, hit the q key to quit lnav, and run the command sudo fail2ban-client status sshd to see a list of banned IP addresses, which should obviously only be your workstation. You can unban the address by running the command sudo fail2ban-client set sshd unbanip your-workstation-ip-address.
Access Other Hosts
Now for the fun part, SSH into the ap VM from your workstation (ssh user@ip-address), now run ssh user@nginx, you’ll be asked to enter the password for that user on the NGINX VM. Since the ap VM is on the same domain as your other VMs and pfSense is configured to register DNS names for reserved DHCP addresses, you can SSH into the NGINX vm by it’s hostname nginx, the same should apply to the 4t web app we set up with the hostname 4t-app.
Note: If you can’t resolve by hostname, open up your pfSense admin page and make sure that the checkbox at Services > DNS Resolver > General Settings > Register DHCP static mappings in the DNS Resolver is checked.
Even better, if you are using the same username for all VMs, you don’t even need the username, you could simply run the command ssh nginx, or ssh 4t-app from the ap VM! Pretty neat, right?
If you are unable to connect from the ap VM to another, you might have to open up a console through Proxmox and install SSH on the target VM (see steps above for SSH Configuration).
Since these other VMs do not have the SSH port open to the internet, we can leave them on password authentication. If you decide to create an SSH key on the ap VM to access other VMs, make sure that key has a passphrase, and do not add it’s public key the the ap VMs authorized_keys file.
Tips and Tricks
That’s pretty much it for this segment, I’ll leave you with a small collection of useful command line and SSH tricks.
TMUX
tmux is a terminal multiplexer, it allows you to run multiple terminal sessions in the same host that persist across SSH sessions. Let me show you what I mean, from your ap VM run the command sudo apt install -y tmux, then run the command tmux to start a new session.
You’ll see a green bar at the bottom of the screen with some information on it. Run the command top to list the running processes on the ap VM, then press ctrl + b which changes focus to the tmux process, then press the d key to disconnect from the tmux session. You’re now back at the main console and should see something like [detached (from session 0)], now type exit to quit the SSH session, you’re back on your workstation’s terminal.
The tmux session and top command are still running, SSH back in to your ap VM and run the command tmux ls, you should see something like 0: 1 windows (created Sun Sep 29 17:44:57 2019) [180x59], run the command tmux attach and hey, there’s your top command still plugging away.
Next try pressing ctrl + b, and then c to create a new window, note that the bottom left corner now shows two windows 0:top-, and 1:bash. Run the command watch free here, this will repeat the command free which shows memory information, every two seconds at display the output. You now have two continuous commands running in different tmux windows.
Press ctrl + b, and then n or p to jump to the next or previous window, respectively. We’ve only scratched the surface of what tmux can do here, but you can surely see how useful it is. Press ctrl + b again, and hit d to disconnect. Running tmux ls now will show two windows open, run the command tmux kill-server to close all open tmux sessions and windows.
Quick Commands
If you just want to run a command on your remote server and then exit, you can append the command to the SSH connection command, for example ssh user@ip-address free will log in to the remote server, run the free command, and return the output to your workstation’s terminal.
Transfer Files
The scp command will transfer files over an SSH connection. SSH into your ap VM, and run the following commands to create a text file with a message inside.
touch hello.txt
echo "This is a test" > hello.txt
Now run exit to quit the SSH session, and then run scp user@ip-address:hello.txt ./, this will copy the hello.txt file from the remote host to your current directory, run ls to see it in your current directory. You could also copy a file from your workstation to the remote host, for example scp ./hello.txt user@ip-address:~/.
The syntax is scp FROM TO, either of these can the path to a local file, or a remote host connection followed by a colon : and then a path on the remote host. the tilde ~ symbol just means the current user’s home path (e.g. /home/username).
Tunnels
You can create SSH tunnels when establishing a connection with the -L flag, the syntax is ssh -L localport:remotehost:remoteport user@ip-address.
Try running from your workstation ssh -L 1234:4t-app:5000 user@ip-address, this will tunnel port 1234 on your workstation to port 5000 on the 4t-app VM. Now try visiting http://localhost:1234 in your workstation’s browser, the browser will connect to port 1234 on your workstation, which is tunneled through the ap VM to port 5000 on the 4t-app VM, and this works because the ap VM can locate and communicate with the 4t-app since they are both behind the firewall and on the same network.