In this segment you will learn how to set up and use a secure remote access server for your home lab network.
Alright, this should be the last segment in the series before we
broadcast our pirate signal and hack into the Matrix go live and host to the open internet. What we are doing here is creating a dedicated endpoint for remote access, and adding a reasonable amount of security to it. This gives us remote access to an actually usable console session into other guest VMs so we can do some great things like copy and paste for example.
We will be using SSH with a locked down configuration to allow access only with a key, no passwords allowed, and also configuring Fail2Ban as an added security measure. I will also mention some other tactics you can use to lock down your SSH server.
At the end of this segment, you will have access to your home lab network “remotely” from within your home network, you will not have access from outside of your home network until we set up the system to be open to the internet (likely in the next segment). I know, bad choice of words, but we are setting this up now so it will be ready once we do go live.
Set up an Access Point VM
Create a new linked clone of your template VM, I’ll call mine
ap for Access Point, you can set up your VM ID’s however you want or just accept the default values, I like to have a little organization in my VM list so I’ll use
500 for this one.
Remember not to use ID’s lower than
Just as in previous segments, we’ll need to set up a static IP address and NAT port forward for this VM, first grab the MAC address.
You’ll need to log in to your pfSense VM (
https://ip-address:8080), if you forgot the IP address just open a console in Proxmox and pfSense will have the WAN IP address displayed on the welcome screen.
Head to Status > DHCP Leases, and take a look at what static addresses are in use to determine the next available address, in my case that would be
172.16.44.103, so that’s what I’ll be using.
Next, head to Services > DHCP Server, click the Add button, fill in the form, then save and apply the configuration.
We’ll create a new alias for this VM - Firewall > Aliases > IP > Add.
Finally, we’ll add the port forward - Firewall > NAT > Port Forward > Add.
Now all traffic going to the WAN address of pfSense on the SSH port (22) will be forwarded to our
Why not just use SSH on pfSense? It’s best to have SSH available to the open internet only on a dedicated endpoint, in the unlikely event that your SSH endpoint is compromised you can simply disconnect the network from this VM in Proxmox, which allows you to stop the threat and keep the VM running while you investigate, without interfering with production services. You will still be able to SSH into pfSense, you just have to go through the access point first.
Are you sick of this part? Me too, I built a little bash script to automate the process of generating a new machine ID and SSH server keys, and setting a new hostname. You can use it by starting a console session in your new VM (through Proxmox) and running the following command which will run the script straight from GitHub. (Won’t it be nice to have copy and paste when we finish this segment?).
sudo bash -c "bash <(wget -qO- https://raw.githubusercontent.com/dlford/ubuntu-vm-boilerplate/master/run.sh)"
You will want to restart when prompted because the VM needs to pick up the new IP addresses we reserved for it.
Note: Always use caution running remote scripts, or any scripts at all for that matter, especially as root, at the very least you should read through it to make sure it’s not doing anything untoward.
The source code for
run.sh is available here so you can review it:
Let’s first check that SSH Server is installed and running on the new VM, run the command
systemctl status sshd, you should get something like the image below.
If you don’t, you’ll need to run the following commands.
Note: Skip this start command for now if you already have port
22 open to the internet, run it at the end, after the SSH configuration is secured
sudo apt install -y openssh-server sudo systemctl enable sshd sudo systemctl start sshd
We will come back to this VM to secure the SSH server after we set up an SSH key on your workstation.
SSH Key Pairs
I’d like to explain SSH to you in explicit detail, but Digital Ocean has already done a great job at it, so here’s a link if you are interested.
For security, you should only create keys on the machine that will be using them, don’t create SSH keys on the server, or any other machine, and then transfer them to a new machine. If you have multiple workstations, create multiple keys, if you get rid of a workstation or re-install the OS, just create a new key (remove access for the old one from your server too!).
Create a Key Pair
From your workstation, run the following command from a terminal to create a new key pair.
If you are on Windows 10 run this from PowerShell, if you get a “no such command” error, you need to install the SSH client. Open the Settings app and search for “Manage Optional Features”, scroll down to “OpenSSH Client”, and click Install.
ssh-keygen -t rsa -b 4096 -C "yourname@devicename"
You will be prompted for a directory to store the key files in (press enter to accept the default location), and an optional passphrase which I recommended using at the very least on any device that ever leaves your home (you’ll need to supply the passphrase to use the key).
This should create a new 4096 bit rsa key pair with the comment yourname@devicename, the comment is nice for when you need to de-authorize a key from the server, you’ll know the user and device that has that particular private key.
You can see the files by running the command
ls .ssh, the file
id_rsa is the private key that should never leave this machine, and
id_rsa.pub is the public key, which should be installed on any server that the corresponding private key will have access to.
Manage SSH Key Access
There is a handy tool for copying public keys to a server, run the command below to copy your new SSH public key to the
username refers to the username on the remote server, not your local workstation, and the IP address should be that of your pfSense VM (which will forward the traffic to the
You will see a warning about
The authenticity of host ... can't be established, this happens because the SSH server on the
ap VM has a server key that is unknown to your workstation, choosing yes will continue connecting and save the servers key on your workstation. If this were a server you’d connected to before it should be concerning to get this warning, as it may be evidence of some foul play, for example a DNS hack to steal your SSH key by impersonating the server you were trying to connect to, fortunately the SSH client will flat out refuse to connect if there is a server key mismatch (a mismatch would be a situation where your workstation’s known hosts file has an entry for the target server but the saved key does not match the server’s key).
You’ll be asked for the password for the remote user, and then your public SSH key will be transferred to the server. Now run the command
ssh username@ip-address to connect, if you entered a passphrase when creating your SSH key you’ll be asked for it here, otherwise you should be immediately connected.
If you check the
.ssh directory on the
ap server, you will see a file named
authorized_keys, this is where all public keys are stored for this user, one per line. You can also add and remove keys directly to/from this file, if you run
nano .ssh/authorized_keys and press the
end key on your keyboard to jump to the end of the line, you will find the comment you entered when creating the key pair to identify the public key, pressing
k will “cut” that line in the event you wanted to remove a key (side note:
u will “paste” the previously cut line in the
nano text editor).
Secure the SSH Server
- Fact: Hundreds if not thousands of bots run 24/7 connecting to any SSH server they can find and trying to log in with common usernames and passwords (this is actually called a word list attack, but is commonly lumped in with the term brute forcing, which is technically the process of trying every possible combination of characters one by one, from here on I’ll join the masses and use the term brute forcing to refer to either of the two)
- Fact: Just having the SSH port
22open to the internet will result in brute force attempts on your server
- Fact: If a brute force bot or unauthorized human successfully logs in to your server, you are going to have a really bad day
We will implement the three most effective measures against SSH brute force attacks:
- Disallow the
rootuser from logging in remotely
- Disallow any user from logging in via password (SSH keys only)
- Rate limit any failed login attempts with Fail2Ban.
There are additional tactics that may be employed, I won’t cover them here, but if you are interested please let me know, I may consider writing an article on a few of them if there is any interest. I feel the above three are a good balance of security and convenience for the average system, but I will list a few more tactics just to be thorough.
- Disable SSH (The most secure option, no remote access at all)
- 2FA (Two Factor Authentication)
- Whitelisting (Only allow access certain IP addresses, this could be easily implemented in your pfSense VM)
- *Port knocking (Only allow access temporarily to any IP address that has sent a series of packets to specific ports in sequence)
- *Change the SSH port to something other than
*These last two tactics do NOT provide any security at all, I cannot stress that fact enough, they are obfuscation techniques. While obfuscation (hiding the server) has its benefits in my opinion, namely in filtering out most bots and reducing server log spam, it is an important distinction to make that these tactics are easily bypassed if they are discovered, and it’s quite feasible that they could be uncovered by a bad actor. There are many who believe obfuscation to be bad practice due to a false sense of security, but I feel that they can be beneficial when used responsibly.
More SSH Configuration
Run the command
sudo nano /etc/ssh/sshd_config to open up the configuration file, you’ll want to modify or add the following lines.
Don’t forget to replace
your-username with your actual username on the server.
LoginGraceTime 2m PermitRootLogin no StrictModes yes MaxAuthTries 1 AllowUsers your-username HostbasedAuthentication no IgnoreUserKnownHosts no IgnoreRhosts yes PasswordAuthentication no PermitEmptyPasswords no ChallengeResponseAuthentication no UsePAM no
Restart the SSH service with the command
sudo systemctl restart sshd to load the new changes from this file.
Note that from here on, you can’t use the
ssh-copy-id command for new SSH key pairs since logging in via password is no longer allowed, you must manually add them to the
authorized_keys file on the server or change the
PasswordAuthentication setting to
yes and restart the SSH service.
If you are curious, here is a link to the
man (manual) page for
sshd_config that explains all of these options in detail.
Set up Fail2Ban
Fail2Ban is an incredibly handy tool, it scans log files for failed logins (or other undesired behavior) and dynamically blocks IP addresses for a configured amount of time, effectively rate limiting any brute force attempts.
Run the command
sudo apt install -y fail2ban, then
sudo nano /etc/fail2ban/jail.local to create a new configuration file.
if the file
jail.local is not found, Fail2Ban will use the default configuration file
Fail2Ban is very powerful and has a ton of configuration options, but lets keep it simple and just configure one jail.
[sshd] enabled = true port = 22 filter = sshd logpath = /var/log/auth.log findtime = 3600 bantime = 300 maxretry = 2
Now to enable Fail2Ban run the following commands:
sudo systemctl enable fail2ban sudo systemctl start fail2ban
Here’s a break down of the options in use:
[sshd]: The name of this jail
enabled = true: Enforce the rules of this jail
port = 22: Ban jailed IP addresses from accessing port
filter = sshd: The filter file contains RegEx patterns for the target log file, essentially telling Fail2Ban which log entries are failed login attempts and where to get the IP address from the log entry. Filter files are located in
/etc/fail2ban/filter.d, there are many shipped with the package or you can write your own.
logpath = /var/log/auth.log: Full path to the log file to watch
findtime = 3600: This is the window of time that hits from the log file will be counted, measured in seconds
bantime = 300: The amount of time to ban an IP address, measured in seconds
maxretry = 2: The number of repeated attempts to allow before banning an IP address
Effectively, this jail will ban an IP address from accessing port
22 for five minutes if it has made more than two failed attempts to log in within one hour. The count is cumulative, so if an address was banned for five minutes and makes another failed attempt right after, the count would already be exceeded and it would be banned again for another five minutes.
I like using a long find time and short ban time because it sets a hard rate limit of 12 attempts per hour (after the first ban), which is slow enough severly impact a brute force attack, but the ban time is short enough that if you goof up you don’t have to wait too long before trying again.
If you were so inclined, you could create a second jail to cover the really persistent attacks that just don’t know when to quit, for example to ban an address for a week if it makes more than 19 failed attempts within a week (at a maximum of 12 attempts per hour from the previous jail, it would take around an hour and a half to get 19 failed attempts, so this jail would ban an address for a week if they persistently attack the server for over an hour and a half). That rule would look like this:
[sshd-persistent] enabled = true port = ssh logpath = /var/log/auth.log filter = sshd bantime = 604800 findtime = 604800 maxretry = 19
Since this jail needs to remember events for longer than the amount of time that Fail2Ban keeps track by default, we’ll need to change that default. Run the command
sudo nano /etc/fail2ban/fail2ban.conf, and edit the line
dbpurgeage = 1d to read
dbpurgeage = 8d. Note that this will require more resources for Fail2Ban to run.
You’ll need to run
sudo systemctl restart fail2ban to load the new jail and the configuration change.
There is also a “recidive” jail that you can configure, which does almost the same thing as above by banning persistent addresses for a long period of time, the difference is that it doesn’t look at the log file for a specific event like a failed login, it looks at Fail2Ban’s own log file for ban events, essentially if an address has been banned x number of times in y window of time, ban it for even longer. I like the control of setting this up for each jail individually for each service though.
Test Your Setup
We want to make sure everything works, right? Open up a console session to your
ap VM in Proxmox, and run the command
sudo apt install -y lnav, which is a tool for navigating log files. Once that’s done go ahead and run
lnav /var/log/fail2ban.log so we can monitor the file (you may need to tap the right arrow key on your keyboard to scroll the text into view, the date stamps are very long).
From your workstation run the command
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no username@ip-address, this tells the SSH client on your machine to try to log in with a password instead of your SSH key. You should see the
Permission denied (publickey) error, that’s what we want to see, password login is disallowed. Now try
ssh root@ip-address, you should see the same error, but also on your
lnav window you should see
[sshd] Found ip-address... with your workstations IP address. Try to log in as root again from your workstation and you should see
[sshd] Ban ip-address in the
lnav window, try it a third time and you will find that the connection times out because fail2ban has blocked the request.
Nice, everything works, but now you’re locked out! Let’s fix that, in your Proxmox console session to the
ap VM, hit the
q key to quit
lnav, and run the command
sudo fail2ban-client status sshd to see a list of banned IP addresses, which should obviously only be your workstation. You can unban the address by running the command
sudo fail2ban-client set sshd unbanip your-workstation-ip-address.
Access Other Hosts
Now for the fun part, SSH into the
ap VM from your workstation (
ssh user@ip-address), now run
ssh user@nginx, you’ll be asked to enter the password for that user on the NGINX VM. Since the
ap VM is on the same domain as your other VMs and pfSense is configured to register DNS names for reserved DHCP addresses, you can SSH into the NGINX vm by it’s hostname
nginx, the same should apply to the 4t web app we set up with the hostname
Note: If you can’t resolve by hostname, open up your pfSense admin page and make sure that the checkbox at Services > DNS Resolver > General Settings > Register DHCP static mappings in the DNS Resolver is checked.
Even better, if you are using the same username for all VMs, you don’t even need the username, you could simply run the command
ssh nginx, or
ssh 4t-app from the
ap VM! Pretty neat, right?
If you are unable to connect from the
ap VM to another, you might have to open up a console through Proxmox and install SSH on the target VM (see steps above for SSH Configuration).
Since these other VMs do not have the SSH port open to the internet, we can leave them on password authentication. If you decide to create an SSH key on the
ap VM to access other VMs, make sure that key has a passphrase, and do not add it’s public key the the
Tips and Tricks
That’s pretty much it for this segment, I’ll leave you with a small collection of useful command line and SSH tricks.
tmux is a terminal multiplexer, it allows you to run multiple terminal sessions in the same host that persist across SSH sessions. Let me show you what I mean, from your
ap VM run the command
sudo apt install -y tmux, then run the command
tmux to start a new session.
You’ll see a green bar at the bottom of the screen with some information on it. Run the command
top to list the running processes on the
ap VM, then press
b which changes focus to the
tmux process, then press the
d key to disconnect from the
tmux session. You’re now back at the main console and should see something like
[detached (from session 0)], now type
exit to quit the SSH session, you’re back on your workstation’s terminal.
tmux session and
top command are still running, SSH back in to your
ap VM and run the command
tmux ls, you should see something like
0: 1 windows (created Sun Sep 29 17:44:57 2019) [180x59], run the command
tmux attach and hey, there’s your
top command still plugging away.
Next try pressing
b, and then
c to create a new window, note that the bottom left corner now shows two windows
1:bash. Run the command
watch free here, this will repeat the command
free which shows memory information, every two seconds at display the output. You now have two continuous commands running in different
b, and then
p to jump to the next or previous window, respectively. We’ve only scratched the surface of what
tmux can do here, but you can surely see how useful it is. Press
b again, and hit
d to disconnect. Running
tmux ls now will show two windows open, run the command
tmux kill-server to close all open
tmux sessions and windows.
If you just want to run a command on your remote server and then exit, you can append the command to the SSH connection command, for example
ssh user@ip-address free will log in to the remote server, run the
free command, and return the output to your workstation’s terminal.
scp command will transfer files over an SSH connection. SSH into your
ap VM, and run the following commands to create a text file with a message inside.
touch hello.txt echo "This is a test" > hello.txt
exit to quit the SSH session, and then run
scp user@ip-address:hello.txt ./, this will copy the
hello.txt file from the remote host to your current directory, run
ls to see it in your current directory. You could also copy a file from your workstation to the remote host, for example
scp ./hello.txt user@ip-address:~/.
The syntax is
scp FROM TO, either of these can the path to a local file, or a remote host connection followed by a colon
: and then a path on the remote host. the tilde
~ symbol just means the current user’s home path (e.g.
You can create SSH tunnels when establishing a connection with the
-L flag, the syntax is
ssh -L localport:remotehost:remoteport user@ip-address.
Try running from your workstation
ssh -L 1234:4t-app:5000 user@ip-address, this will tunnel port
1234 on your workstation to port
5000 on the
4t-app VM. Now try visiting
http://localhost:1234 in your workstation’s browser, the browser will connect to port
1234 on your workstation, which is tunneled through the
ap VM to port
5000 on the
4t-app VM, and this works because the
ap VM can locate and communicate with the
4t-app since they are both behind the firewall and on the same network.