Why GitHub Hated My MTU (And How I Fixed It)

Let me tell you about the frustrating day I spent wrestling with my WireGuard VPN and why GitHub decided to be the only site that wouldn’t play nice. I’m hoping my experience, especially if you’re running WireGuard in a microk8s environment like I was, can save you some headaches.

The Setup: microk8s, Ubuntu, and WireGuard

My setup was a bit complex. I had a microk8s cluster running on an Ubuntu 24.04.2 LTS server node (GNU/Linux 6.8.0-54-generic x86_64). I was running my WireGuard VPN server as a pod within this microk8s cluster. This added a layer of network complexity that I didn’t initially account for.

The Problem: GitHub Was a No-Go (But Everything Else Was Fine)

Everything seemed fine. I could access most websites through my WireGuard VPN. Google, YouTube, Twitter – no problem. But https://github.com just wouldn’t load.

curl https://github.com hung indefinitely. I was scratching my head, wondering if it was a DNS issue, a firewall problem, or something else entirely.

The Debugging Journey: Diving into the Network

  1. Initial Tests:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

    curl github.com # Worked! Gave a 301 redirect to HTTPS
curl -v github.com
* Host github.com:80 was resolved.
* IPv4: 20.27.177.113
* Trying 20.27.177.113:80...
* Connected to github.com (20.27.177.113) port 80
> GET / HTTP/1.1
> Host: github.com
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Location: [https://github.com/](https://github.com/)
<
* Connection #0 to host github.com left intact

curl [https://github.com](https://github.com) # Hung...

    This was incredibly confusing. HTTP worked, but HTTPS didn’t. Other HTTPS sites like Google and Twitter were fine. This pointed to an issue specific to HTTPS and, potentially, GitHub.

  2. MTU Investigation: The Culprit Emerges

    I suspected an MTU issue, given the HTTPS connection problems. So, I checked my WireGuard client interface:

    1
2
3

    ip l show wg-client0
6: wg-client0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none

    Then, I attached to the WireGuard pod running in my microk8s cluster and checked the server-side interface:

    1
2
3

    ip l show wg0
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1370 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none

    Aha! Different MTUs. This was likely the source of my woes.

  3. MTU Adjustment and Testing: The Solution

    I started systematically lowering the MTU on my client configuration to match the server-side MTU within the pod.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    [Interface]
PrivateKey = ...
Address = ...
DNS = ...
MTU = 1370 # Matched the server-side MTU in the pod

[Peer]
PublicKey = ...
AllowedIPs = 0.0.0.0/0
Endpoint = your-server-ip:your-port
PersistentKeepalive = 25

    After restarting my WireGuard connection, I tried curl https://github.com again.

    1

    curl -v [https://github.com](https://github.com) # Success!

    And GitHub loaded perfectly in my browser.

Why This Happened: A Deep Dive into Networking

  • Path Fragmentation and microk8s: The microk8s network setup, combined with WireGuard’s encapsulation, added layers of potential fragmentation.
  • TLS Handshake and GitHub: GitHub’s TLS handshake, potentially due to its size or network path, was particularly sensitive to path fragmentation.
  • Local vs. Path Fragmentation: By lowering the MTU, I forced local fragmentation (controlled by my OS) instead of relying on unpredictable path fragmentation.

Understanding MTU and Fragmentation

MTU is the maximum size of an IP packet that can be transmitted over a network interface without being fragmented. Fragmentation occurs when a packet exceeds the MTU of a network link, causing it to be broken into smaller pieces. While local fragmentation (fragmentation by the sending device) is generally manageable, path fragmentation (fragmentation by routers along the network path) can lead to packet loss, reordering, or even blockage by firewalls.

The Impact of Fragmentation on TLS Handshakes

The TLS handshake is a critical process that establishes a secure, encrypted connection between a client and a server. It involves exchanging messages, including certificates and encryption keys. If any part of a handshake packet is lost due to fragmentation, the entire handshake can fail, leading to connection issues.

Why GitHub Was Affected and Others Weren’t

This is because websites have varying characteristics, including:

  • Packet Sizes: Websites with larger content (like GitHub) are more likely to generate larger packets, increasing the risk of fragmentation.
  • CDN and Server Differences: Different websites use different CDNs and servers, which can have varying MTU settings and network paths.
  • TLS Handshake Variations: The TLS handshake process can differ between websites, affecting packet sizes and sensitivity to fragmentation.

Resolving the Issue: Lowering the MTU**

The root cause was a mismatch in MTU between my WireGuard client and the WireGuard server running as a pod within my microk8s cluster. By setting the client’s MTU to the same value as the server’s (1370), I resolved the issue.

By lowering the MTU on the WireGuard client and server, the user could control where fragmentation occurred. Local fragmentation is generally more reliable than path fragmentation, as it’s handled by the sending device. This ensured that packets were fragmented into sizes that could traverse the network path without further fragmentation, preventing packet loss and ensuring a successful TLS handshake.

Key Takeaways

  • MTU and fragmentation can significantly impact network performance and reliability, especially for sensitive protocols like TLS.
  • Understanding the nuances of MTU and fragmentation is crucial for troubleshooting network connectivity issues.
  • By carefully adjusting MTU settings and controlling fragmentation, you can ensure smooth and reliable network communication.
  • If you’re running WireGuard in a containerized environment like microk8s, pay close attention to MTU settings.
  • Always check and match MTU settings between your WireGuard client and the server within its container environment.
  • Use curl -v, ip link show, and tcpdump (if necessary) to debug network issues.
  • Remember that the network path between your client, your microk8s server node, and the internet can have many different MTU values.

I hope this blog post has provided valuable insights into troubleshooting WireGuard VPN issues related to MTU and fragmentation.