chriswheeldon

Setting up a VPN with WireGuard

This is not so much intended as a guide to setting up WireGuard, but rather a log of the arduous, maybe embarrassing, process I took, with the intention of clarifying my understanding through writing.

I had previously set up WireGuard on my NAS box so that I could securely access my home network when traveling. That time I had used the wg-easy Docker image and so I didn’t have to think at all about the internal workings or configuration of the VPN. I’m loathe to admit it but, prior to this, I had barely considered what a VPN such as WireGuard actually was or how it functioned. This time I wanted to do it myself, with the intention of learning some of the basics of Linux networking, VPNs and WireGuard along the way.

It took a bit longer than I had expected.

Installing WireGuard on the Server

For detailed notes on how I installed, configured and started WireGuard, see this post. After completing the steps in that post, I had the following WireGuard configuration file on the server, /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = <server private key>
Address = 10.20.10.1
ListenPort = 51820

[Peer]
PublicKey = <client public key>
AllowedIPs = 10.20.10.2

I was using wg-quick up /etc/wireguard/wg0.conf to start the WireGuard interface.

Installing WireGuard on the Client

The process of setting up the client on my desktop was extremely similar, except I was using Ubuntu 24.04. Apart from installing WireGuard itself, the rest of the configuration was almost identical. After following roughly the same steps, my client’s /etc/wireguard/wg0.conf looked like this:

[Interface]
PrivateKey = <client private key>
Address = 10.20.10.2

[Peer]
PublicKey = <server public key>
AllowedIPs = 10.20.10.1
Endpoint = <server public IP>:51820

Note the symmetry between the interface addresses and the peer allowed IPs.

So, was it Working?

No.

Or, rather, I expected at this point I ought to have been able to ping the server’s WireGuard IP address (i.e. ping 10.20.10.1), but no response was received.

On a positive note, on my client there was an IP route for the Wireguard interface that looked to be intended to send out packets to the server’s WireGuard IP address over the WireGuard interface:

(client) $ ip route
default via ...
10.20.10.1 dev wg0 scope link
...

And Wireshark on the client did show that the client was sending packets to my server’s public IP address and it identified these packets as a WireGuard handshake. It seemed like packets were being successful pushed into the tunnel and sent out over the internet, so why wasn’t the server wasn’t responding with anything? I enabled the debug logging on the server, but it didn’t show anything besides log lines that indicated the WireGuard interface was up (hey, at least that was something).

With all the searching, reading forums posts, guides, documentation, occasionally quizzing an LLM, I had started to form a hazy model of the system. It was ill-founded in many places, probably even misleading me in others, but I was starting to sense that WireGuard was intended to be, and actually was very simple to set up. Whatever the problem was, it seemed likely to be something simple and obvious that I was missing. Here were some of the things I considered:

  • Perhaps ufw was blocking the traffic at ingress or somewhere else in the networking stack.
  • Perhaps my router was blocking the incoming UDP traffic from the server. I was vaguely aware of UDP hole punching from a previous job, but I didn’t know whether that was relevant here.
  • Perhaps ping’s ICMP packets are not able to be tunneled through WireGuard by default.
  • Many guides had used PostUp and PostDown commands in the server configuration file, but I had not added these as I did not understand exactly what they were necessary for (I would later learn these are necessary but not for basic point-to-point communication).
  • I knew that WireGuard created a network interface (wg0), so perhaps I needed to modify the iptables rules on the server to somehow direct traffic from the public interface into WireGuard (I would later learn that this is not necessary).

In this instance, my first thought was correct, disabling ufw resulted in something! Every time the client sent a ping packet, the server would now log a message like this:

wireguard: wg0: Invalid handshake initiation from <client public IP>:58065

Progress! This did at least mean the packets were reaching the server, but somehow the WireGuard handshake was failing. At this point I took a series of long, frustrating excursions; distracted by the idea that the issue might lie with the PostUp or the server’s routing tables I spent time, somewhat randomly and with more hope than rationality, exploring many different avenues.

I needed to take a break. I returned, and doubled checked my server’s configuration files. I spotted immediately that I had the wrong PublicKey for the client. I fixed this and the ping started pinging!

(client) $ ping 10.20.10.1 -I wg0
PING 10.20.10.1 (10.20.10.1) from 10.20.10.2 wg0: 56(84) bytes of data.
64 bytes from 10.20.10.1: icmp_seq=1 ttl=64 time=53.8 ms

What about the ufw firewall? When setting that up I had added rule to allow UDP traffic so why wasn’t it working? A simple answer; I had incorrectly specified the port number in the rule: I had used 51821/udp instead of 51820/udp.

So, after two careless configuration errors, I was finally able to ping the server over the VPN with the firewall enabled.

Next I tried to ping a public IP address over the VPN e.g. ping google.com -I wg0. This did not work, but I was able to ping the server’s public IP address (e.g. ping <server public IP> -I wg0). This suggested that the WireGuard tunnel was working, but that packets were not able to be forwarded by server onwards to other hosts. Rather than waste time, exploring random ideas again, it was time to learn a bit more about how WireGuard works and how packets are handled in Linux.

Understanding WireGuard (and some Linux Networking)

What does WireGuard do? Yes, it’s a VPN, but how does it work? How are packets actually tunneled between the client and server?

Given the client seemed to be doing roughly what I expected, it made sense to try to understand the server side of things. Here’s what I learnt.

At a high level, there are two parts to WireGuard.

  1. There is a network interface (e.g. wg0) that is used to send and receive packets over the tunnel.
  2. There is a kernel module that listens for incoming UDP packets on a specific port (51820 by default).

A network interface is a software abstraction that acts as a connection point between the kernel’s networking stack and the physical/virtual network interface (e.g. Ethernet, Wi-Fi, VPN). The details of how a packet is actually sent or received are an implementation detail, hidden behind the interface abstraction.

The kernel’s IP networking stack is responsible for coordinating between software processes, that wish to send or receive data, and the network interfaces that actually transmit that data, as packets, over the network. It maintains rules, in the form of a routing table, that determine how packets are sent. The routing table is consulted when a packet is sent to determine which interface it should be sent out on. For example, a rule such as 10.20.10.1 dev wg0 indicates that packets destined for the IP address 10.20.10.1 should be sent out over the wg0 interface.

What about receiving packets from a network interface?

When the interface receives a packet, it is passed to the kernel’s networking stack (via an interrupt?). Two things can happen:

  1. If there is a socket bound to the packet’s destination address and port, the kernel will deliver the packet to that socket. The process that owns the socket can then read the packet data and process it as necessary.
  2. If the packet is not destined for a process, it is passed to the kernel’s networking stack, which will then determine how to handle it based on the routing table i.e. the routing table may prescribe that the packet is forwarded (i.e sent out) over another interface.

What about WireGuard?

Remember, WireGuard has a network interface and a kernel module listening for UDP packets on a specific port. When a packet arrives at the server’s public IP address on port 51820 (possibly/likely via some real physical network interface), the kernel’s networking stack will deliver it to the WireGuard kernel module. Then, ignoring the cryptographic details:

  1. Decrypts the encapsulated/tunneled packet (sometimes called unencapsulation)
  2. Passes the unencapsulated packet to kernel’s networking stack via the WireGuard network interface (e.g. wg0).
  3. The kernel’s networking stack handles the packet as if it had been received on the wg0 interface.

When a packet is sent into the wg0 interface, the WireGuard interface implementation does the following:

  1. Encrypts and encapsulates the packet in a UDP datagram.
  2. Sets the destination of this UDP datagram to the public IP and port of the remote peer.
  3. Sends the UDP packet using the kernel’s networking stack.

Forwarding Packets onto other Networks

With this understanding, it felt clear to me that the issue was likely in the configuration of the server’s networking stack, rather than anything specific to WireGuard or the client. I believe that the server was receiving packets from the client, but was not forwarding them onwards to other networks (e.g. the internet). My attention was drawn to PostUp and PostDown commands in the WireGuard configuration file examples that I had seen during the initial configuration.

Most example configurations included the following PostUp command similar to the following:

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

This command is configuring the server’s iptables (a firewall) rules to allow packets to be forwarded from the WireGuard interface (%i is a placeholder for the interface name, e.g. wg0) to other interfaces (e.g. eth0), and vice versa. The MASQUERADE rule is used to modify the source IP address of outgoing packets so that they appear to come from the server’s public IP address, rather than the client’s private WireGuard IP address.

The latter is necessary so that an external host, receiving untunneled packets packets from the server, will send its response packets back to the server, from where they can be tunneled back to the client. Without this rule, the external host would attempt to send its response packets back to the client’s private IP address.

I added the PostUp and then attempted to curl a public host over the VPN and got the following error:

(client) $ curl -vv --interface wg0 google.com
...
* socket successfully bound to interface 'wg0'
* connect to 142.251.30.113 port 80 from 10.20.10.2 port 51284 failed: No route to host

no route to host.

Revisiting the Client’s Configuration

I wasn’t immediately sure if this was an issue on the client or the server; one thing that I did note was that there was no traffic captured by Wireshark on the client when I issued the curl command. I tried the ping google.com command again, this time with Wireshark open, and saw no packets were being sent. I had previously assumed that, given packets were visible when pinging the server itself, then they were also being sent when pinging google via the server. I should have checked Wireshark at the time, and it would have been obvious that the issue lay with the client, not the server. Although that mistake cost me some time, it did provide a great excuse to learn more about how WireGuard works and how packets are handled in Linux.

So this time it seemed reasonable to start debugging on the client. The client’s routing table showed the following:

(client) $ ip route
default via 192.168.1.254 dev enp3s0
10.20.10.1 dev wg0 scope link
192.168.1.0/24 dev enp3s0 scope link

My next thought was, instead of trying to set an --interface with curl, to set my Wireguard interface as my default route. I tried the following, but still no packets were captured by Wireshark on the client:

(server) $ sudo ip route del default
(server) $ sudo ip route add default via 10.20.10.1 dev wg0

I then thought that perhaps I needed to a have a rule to route the tunneled packets to the server’s public IP address. So I added the following rule:

(server) $ sudo ip route add <server public IP> via enp3s0

But still the same no route to host error. Although not helpful here, this wasn’t a bad idea and I would later learn that such a route is required- after all the tunneled packets do need to find themselves going out on a real interface.

By chance I stumbled across a post mentioning that the issue is likely with the AllowedIPs setting in the client’s configuration. Recall I had set it to the server’s WireGuard IP address, 10.20.10.1. I now paid attention to the Wireguard docs. This setting is used to configure the routing tables on the client, and so in this configuration only packets with a destination of 10.20.10.1 can be sent over the wg0 interface.

The client’s AllowedIPs needed changing to 0.0.0.0\0 to route all packets and, with that change made, I was able to curl a public host over the VPN!

The final mystery I now need to solve: why does wg0 not appear anywhere in my routing table?

(client) $ ip route
(client) $ ip rule
(client) $ ip route show table all

Closing Thoughts

  1. Double check configuration files.
  2. Starting with a simple, mostly correct mental model, use documentation and conversations with an LLM to build on it.
  3. Online posts tend to be terse and focused on very specific issues; without the solid mental model, they can be distracting or, even worse, misleading.
  4. An old saying rings true: 10 minutes spent thinking (in this case, reading) can save hours of debugging.