Imagine that you wanted to build your own VPN protocol from the ground up. The first thing you should know is that all VPN protocols you know of today were all designed in a time for a place with a purpose and some reasons. With increasing scrutiny of commercial VPN providers and “digital ID” requirements cropping up around the globe seemingly all at the same time you can rest assured that someone, somewhere, is creating a new one.

It will have problems. All of them do. And so will yours if you decide to build one too.

As part of this blog, I’ll demonstrate how to design and build a simple “TOY” teaching VPN protocol using the unified API for veepin.

We’ll also cover some existing production VPNs which do things like send a JPEG as part of their authentication flow and use cryptographically broken SHA-0, in case you wanted to know how to do that.

The teachings can also be repurposed for many things (like I did), such as to leverage the new Post-Quantum cryptography functions available in Go 1.27.0 to backport and enforce post-quantum cryptography in 16 production VPN protocols. Now I can POST my meme JPEGs while I’m Post’n my Quantums.

*ba-dum tsss*

I had a few beers and ranted using speech-to-text for an AI model for a few weeks about VPN protocols. The result is veepin. veepin speaks sixteen production VPN protocols, client and server for every single one, from scratch, in pure Go. The only dependencies are golang.org/x/{crypto,net,sys}. IKEv2/ESP, WireGuard, OpenVPN, SSTP, SSH, L2TP/IPsec, L2TPv3, AnyConnect, Nebula, MASQUE, Fortinet, GlobalProtect, Cisco IPsec, Ivanti Connect Secure, SoftEther and AmneziaWG.

Every one of them is tested in Docker against the actual third-party implementation. strongSwan, wireguard-go, openvpn, openconnect, ocserv, sshd, the Linux kernel’s own L2TPv3, SoftEther’s own client. Because reality and interoperability don’t care about remembering an RFC number.

I learned pretty much nothing at all, but I did put on a headset and yap about the nuance of every VPN protocol I knew of to an AI model for several weeks. The AI model created the veepin repo and now if you were to point your AI agent at the same repo it could likely build you a custom VPN protocol in one shot. If that’s your goal, you can stop reading now.

Otherwise, stay tuned! This is very far from my first rodeo working with protocols and I’m actually genuinely very happy with the result. We’ll cover just enough of the shape of things to get you dangerous, and then we’re going to build a complete working VPN protocol from an empty file.

VPN protocols are far more similar than they look, and the places they differ are almost never the places you’d expect.

Everything is the same

Here’s WireGuard’s handshake. Two messages. Simple and inflexible.

sequenceDiagram
    participant I as Initiator
    participant R as Responder
    Note over I,R: static keys known (IK), optional PSK mixed in (psk2)
    I->>R: Handshake Initiation<br/>(ephemeral, encrypted static, encrypted timestamp, mac1/mac2)
    Note over R: verify mac1 (addressed to me?), TAI64N timestamp (anti-replay)
    R->>I: Handshake Response<br/>(ephemeral, empty AEAD, psk mixed)
    Note over I,R: both derive (send_key, recv_key) → transport.Session

Now here’s Ivanti Connect Secure, formerly Pulse Secure, formerly Junos Pulse. Everything else aside, it is wildy impressive how many times they’ve re-wrapped this protocol over the years.

sequenceDiagram
    participant C as client
    participant S as server
    C->>S: GET / … Upgrade: IF-T/TLS 1.0
    S->>C: 101 Switching Protocols
    C->>S: TCG/VersionRequest
    S->>C: TCG/VersionResponse
    C->>S: Juniper/ClientInfo ("clientHostName=…")
    S->>C: TCG/AuthChallenge (empty, meaning "begin")
    C->>S: EAP Response: Identity "anonymous"
    S->>C: EAP Request: Juniper/1 (server information AVPs)
    C->>S: EAP Response: Juniper/1 (OS, user agent)
    S->>C: EAP Request: Juniper/1 { EAP-Message: EAP Request Juniper/2, PASSREQ }
    C->>S: EAP Response: Juniper/1 { username AVP, EAP-Message: password }
    S->>C: EAP Request: Juniper/1 { session cookie AVP }
    C->>S: EAP Response: Juniper/1 (empty)
    S->>C: TCG/AuthSuccess
    S->>C: Juniper/1: address, netmask, DNS, MTU, routes, ESP parameters
    S->>C: Juniper/1: the server's ESP keying block
    C->>S: Juniper/1: the client's block + a copy of the server's
    C->>S: Juniper/5: "ncmo=1"
    S->>C: Juniper/0x8f: end of configuration
    Note over C,S: data, either ESP over UDP or Juniper/4 messages here

Eighteen messages. An HTTP upgrade, a standards-body version negotiation, EAP wrapped in a vendor EAP method wrapped in another vendor EAP method, and an identity of anonymous which is not anonymous in any way whatsoever.

And then there’s SoftEther, which looks as if it was somebody uploading JPEG images. You POST /vpnsvc/connect.cgi with a Content-Type: image/jpeg and a body of VPNCONNECT, and then you authenticate with SHA0(SHA0(password + UPPER(username)) ‖ random).

Yes, that’s SHA-0. In a protocol still shipping in 2026.

No, That’s not good.

But, these three protocols have nothing in common!

Incorrect! These three protocols have everything in common. Look at what each one actually accomplished:

1. Prove who you are. WireGuard uses a static Curve25519 key inside a Noise handshake. Ivanti uses a password inside EAP inside TLS. SoftEther uses a password digested through a hash that was broken in 2004.
2. Agree on a key. WireGuard does an ephemeral Diffie-Hellman. Ivanti has the server literally mail you the ESP keys inside the TLS session it already established. SoftEther just uses the TLS session it’s already sitting in.
3. Get told your address. Your tunnel IP, your netmask, your DNS servers, your MTU, maybe some routes.
4. Move packets. Frame it, seal it, number it, send it. Unframe it, check it, unseal it, deliver it.

That’s every VPN. The decoration varies enormously. The skeleton does not.

Most of that decoration is load-bearing history, too. A NAT that needed traversing in 2003, a firewall that only passed 443, an appliance vendor who needed to ship before a trade show. Which means that if VPN protocols intimidate you, the intimidating part is the surface, and the surface is exactly the part you get to skip when you write your own.

So write one. Not to deploy it, but to make the other sixteen readable. You’ll make every mistake the real protocols made, in miniature, in an afternoon, and afterward you’ll spot those same mistakes on sight in somebody else’s wire format.

Three questions you have to answer

Before any code, there’s three things every protocol decides differently, and yours will have to decide them too.

Which tunnel is this?

Every server has one socket and a whole lot of clients. A packet arrives. Which tunnel is this?

Every protocol answers differently, and the answer is never “the source address”, because NAT rebinds and clients roam. So the answer lives somewhere in the header.

• ESP puts a 32-bit SPI in the first four octets. Easy.
• WireGuard puts a receiver index at offset 4, but only on transport-data messages. The other three message types route somewhere else entirely.
• L2TPv3 puts a 32-bit Session ID at offset 4, hidden behind a T-bit that tells you whether this is data or control.
• TOY, the protocol we’re going to build, puts a 16-bit session at offset 6.

Same job. Four different offsets, three different widths, and one of them is conditional on the message type. There is no universal answer here, so the API has to take the answer as a parameter.

What does the handshake actually produce?

Every one of those sixteen handshakes, the two-message Noise exchange, the eighteen-message Juniper EAP soap opera, the JPEG cosplay, all of them, produce exactly this:

type Result struct {
    TUNName     string   // the interface the data path is bound to
    AssignedIP  net.IP   // the address the server gave you
    Netmask     net.IP
    AssignedIP6 net.IP   // dual-stack, or nil
    Prefix6     int
    Layer2      bool     // this tunnel carries Ethernet, not IP
    Gateway     net.IP   // the server's OUTER address. see below.
    DNS         []net.IP
    MTU         int
}

Nine fields, six of which are usually nil. That is the total information content of every VPN handshake ever designed.

And critically, Dial installs none of it. It doesn’t add the address. It doesn’t set routes. It doesn’t touch /etc/resolv.conf. It hands you the Result and you apply it. That’s the thing that lets the same dial path serve the CLI, which hands it to a router, and the NetworkManager plugin, which hands it to NM, and whatever you feel like embedding it in.

How do packets move?

The per-protocol data path is four methods:

type Tunnel interface {
	// InboundKey identifies this tunnel on the wire: inbound packets whose Demux
	// yields this key belong here. It must agree with the pump's Demux.
	InboundKey() uint32

	// Routes are the inner destinations this tunnel carries. An outbound TUN
	// packet goes to the tunnel whose route matches its destination most
	// specifically; a packet matching none is dropped.
	Routes() []netip.Prefix

	// PeerAddr is where encapsulated packets are sent.
	PeerAddr() *net.UDPAddr

	Encapsulate(ipPacket []byte) ([]byte, error)
	Decapsulate(pkt []byte) ([]byte, error)
}

Plus one function, which is the only protocol-specific part of inbound routing:

// It is the one part of inbound routing that is protocol-specific: ESP puts
// its SPI in the first four octets, whereas WireGuard's receiver index sits
// at offset 4 and only on transport-data messages.
type Demux func(pkt []byte) (key uint32, ok bool)

For ESP that’s binary.BigEndian.Uint32(pkt[:4]). For WireGuard it’s offset 4, on message type 4 only. For L2TPv3 it’s offset 4 behind the T-bit. Four lines of code, and the entire difference between every protocol’s inbound routing lives in there.

Everything else gets written exactly once, in a package that has never heard of IKEv2. Reading the TUN. Matching outbound packets to a tunnel by longest-prefix. Batching syscalls. Counting bytes. MTU. ICMP. Shaping.

Implement four methods and one function and you have a VPN:

flowchart TD
    H["Handshake: Hello → Welcome/Reject<br/>(nonces + proof)"] --> RES["client.Result<br/>(assigned addr, DNS, routes)"]
    RES --> PUMP["dataplane.Pump: framed data path"]
    PUMP <-->|Header + ciphertext + tag| PEER["peer (UDP)"]
    PUMP <--> TUN["TUN"]
    REG["client.Register / RegisterServer"] -.both roles.-> H

So let’s do that.

Now let’s build one

Right. Enough theory. Let’s build a VPN.

TOY is the seventeenth registered protocol in veepin and it is catastrophically insecure on purpose. It exists so that the shape of a VPN can be read in one sitting.

The full spec lives at internal/toy/SPEC.md and it’s written to be reimplementable. The interop harness proves that by talking to an independent Python implementation of the document.

The Python below is that implementation, and I checked every byte of it against the Go before pasting it here. Do not carry traffic over this.

Step 1: Twelve octets

Twelve octets on every single datagram, starting with the ASCII magic TOY. Big-endian, fixed width, so parsing is bounds checks and slicing and nothing else.

packet-beta
0-7: "T"
8-15: "O"
16-23: "Y"
24-31: "version"
32-39: "type"
40-47: "flags"
48-63: "session"
64-95: "counter"

import struct

def header(msg_type, session, counter, flags=0):
    return b'TOY' + bytes([1, msg_type, flags]) + struct.pack('>HI', session, counter)

The session at offset 6 is your demux key. Remember that whole discussion earlier? This is TOY’s answer to it.

The server finds the right client by reading offset 6, never by looking at the source address. That’s what lets a client survive a NAT rebinding, and it’s exactly why the field lives in the header instead of in the encrypted body.

The counter is per-direction and starts at 1. It does double duty as anti-replay and as the thing that keys the keystream.

Message types:

Step 2: One hash, three jobs

One 64-bit function does key derivation, the auth proof, and the packet tag.

It’s FNV-1a, picked for exactly one reason: it’s four lines in any language, which is what makes the spec reimplementable by a stranger.

FNV_OFFSET = 0xcbf29ce484222325
FNV_PRIME  = 0x100000001b3
MASK64     = (1 << 64) - 1

def digest(*parts):
    h = FNV_OFFSET
    for p in parts:
        for b in p:
            h = ((h ^ b) * FNV_PRIME) & MASK64
    return h.to_bytes(8, 'big')

Check yourself as you go. digest(b"abc") is e71fa2190541574b.

A real protocol would use three different, purpose-built constructions here. The fact that this is one function doing three jobs is itself a red flag, and it’s the kind you should train yourself to spot.

Step 3: Four messages

sequenceDiagram
    participant C as client
    participant S as server
    C->>S: HELLO session=0 counter=1<br/>nonce_c ‖ user
    Note over S: allocate session<br/>allocate address
    S->>C: CHALLENGE session=S counter=1<br/>nonce_s
    C->>S: AUTH session=S counter=2<br/>proof
    Note over S: verify proof
    alt proof verifies
        S->>C: WELCOME session=S counter=2<br/>address ‖ netmask ‖ gateway ‖ mtu ‖ dns
    else proof does not verify
        S->>C: REJECT session=S counter=2<br/>reason
    end
    Note over C,S: data path, both directions
    C-->>S: DATA session=S counter=3+<br/>tag ‖ ciphertext
    S-->>C: DATA session=S counter=3+<br/>tag ‖ ciphertext

Both sides derive the key once they’ve seen both nonces:

def derive_key(secret, nonce_c, nonce_s):
    if isinstance(secret, str): secret = secret.encode()
    return b''.join(digest(secret, nonce_c, nonce_s, bytes([i])) for i in range(4))

def proof(secret, nonce_c, nonce_s):
    if isinstance(secret, str): secret = secret.encode()
    return digest(secret, nonce_c, nonce_s, b'toy-auth')

The client sends the proof, the server recomputes it and compares.

Compare in constant time. It genuinely does not matter here, because the whole scheme is broken by inspection anyway, but a timing-variable compare in an auth path is precisely the habit you don’t want to teach anybody.

Now, two rules that are requirements and not optimizations. These are the two places where a naive implementation goes from merely inefficient to actually exploitable, and they generalize to every protocol you will ever write.

A repeated HELLO must not start a second handshake. The client retransmits on a timer because there’s no reliability layer. If your server allocates a fresh session and a fresh address per HELLO, then a lossy link, or one peer being obnoxious on purpose, drains your entire address pool without ever authenticating once. Key the pending handshake on the client nonce and replay the same CHALLENGE.

A failed AUTH must not discard the pending handshake. Send REJECT and otherwise leave the session completely alone. Session IDs travel in the clear, so anybody who saw the CHALLENGE knows one, and sending a wrong proof for it requires no secret whatsoever. Throw away state on that basis and a single forged datagram cancels a legitimate client’s handshake.

Both of those are the same rule:

Unauthenticated input must never destroy state.

Write it on a sticky note.

Step 4: Seal and open

The “encryption” is a repeating XOR pad. XOR is its own inverse, so encrypt and decrypt are the same function, which is convenient and is also a pretty strong hint that nothing of value is happening here.

def keystream(key, counter, buf):
    out = bytearray(buf)
    for i in range(len(out)):
        pad = key[(i + counter) % 32] ^ ((counter >> (8 * (i % 4))) & 0xFF)
        out[i] ^= pad
    return bytes(out)

def tag(key, hdr, ciphertext):
    return digest(key, hdr, ciphertext)

def seal(key, session, counter, msg_type, payload):
    hdr = header(msg_type, session, counter)
    ct  = keystream(key, counter, payload)
    return hdr + tag(key, hdr, ct) + ct

The tag covers the header. This is the one part of TOY worth copying verbatim into something real.

It means the type, session and counter cannot be edited in flight without invalidating the tag, so a receiver can actually trust the framing it just used to route the packet. A real protocol gets this by handing the header to an AEAD as additional authenticated data. Same idea, actual guarantee.

Test vectors so you can check your implementation against mine. secret="hunter2", nonce_c=01..08, nonce_s=09..10, session=0x1234, counter=7:

digest("abc")  = e71fa2190541574b
key            = 678a9337f51e0a09678a9237f51e0856678a9137f51e06a3678a9037f51e04f0
proof          = 7b1e45876f3fabb6
header         = 544f59010600123400000007
tag            = f7e2ce1dd33583ed

Now look really hard at that key. Split it into its four 8-octet blocks:

block 0  678a9337f51e0a09
block 1  678a9237f51e0856
block 2  678a9137f51e06a3
block 3  678a9037f51e04f0

Five of every eight octets are identical across all four blocks.

The blocks differ only in a trailing counter octet, and FNV-1a has effectively no avalanche, so they come out nearly the same. The “32-octet key” is carrying something closer to 12 octets of actual variation, and the keystream inherits that structure directly.

This is precisely what a real KDF like HKDF exists to prevent. Related inputs producing related outputs. You can literally see the failure sitting there in the hex, which I find a far more convincing argument for using a real KDF than any amount of prose about it.

Step 5: The order that gets you owned

def open_packet(key, window, pkt):
    hdr, body = pkt[:12], pkt[12:]
    got_tag, ct = body[:8], body[8:]
    counter = struct.unpack('>I', hdr[8:12])[0]

    # 1. VERIFY THE TAG FIRST.
    if not constant_time_eq(tag(key, hdr, ct), got_tag):
        return None

    # 2. THEN consult the replay window.
    if not window.accept(counter):
        return None

    # 3. THEN decrypt.
    return keystream(key, counter, ct)

That order matters. A lot. It’s the mistake I would bet actual money on you making if nobody told you.

Check the tag before you touch the replay window. If you admit an unauthenticated counter into the window, then anybody who can send you a datagram can advance your window to 0xFFFFFFFF and lock the real peer out of its own session. Permanently. With one packet. Without knowing any secret at all.

Same principle as the AUTH rule. Same sticky note.

The window itself is a highest-seen counter and a 64-entry bitmap behind it. Every protocol here needs one, and every single one of them implements the identical algorithm:

flowchart TD
    R["incoming counter c"] --> A{c > highest seen?}
    A -->|yes| SLIDE["slide window forward<br/>clear the slots scrolled past"]
    SLIDE --> ACCEPT["mark c seen · accept"]
    A -->|no| B{"c within window<br/>(highest − size, highest]?"}
    B -->|no, too old| REJECT["reject (below window)"]
    B -->|yes| C{slot for c already set?}
    C -->|yes| REJECT2["reject (duplicate)"]
    C -->|no| ACCEPT2["mark c seen · accept"]

Ahead of the highest, accept and slide. Inside the window and unseen, accept. Inside the window and already seen, or older than the window, discard.

I wrote that twice. Once in Nebula, once in TOY. Then I noticed they were byte-for-byte identical and pulled it out into its own package. That’s the tell for shared machinery, by the way. Not “these look kind of similar.” It’s “these are the same file, twice.”

The same rule governs peer address updates, incidentally. TOY sessions survive a NAT rebinding by updating the peer address, but only after a packet has authenticated, so the new address is attested by the key rather than merely claimed by whoever sent it. Take it from an unverified packet and anybody can redirect a live session by spoofing a single datagram.

Step 6: Wire it up

Now the veepin part, which after all of that is almost boring. Implement the four methods:

func (s *Session) InboundKey() uint32       { return uint32(s.ID) }
func (s *Session) Routes() []netip.Prefix   { return s.routes }
func (s *Session) PeerAddr() *net.UDPAddr   { return s.peer }

func (s *Session) Encapsulate(inner []byte) ([]byte, error) {
    return s.seal(MsgData, inner)
}

func (s *Session) Decapsulate(pkt []byte) ([]byte, error) {
    h, body, err := ParseHeader(pkt)
    if err != nil { return nil, err }
    if h.Type != MsgData { return nil, ErrMalformed }
    return s.open(pkt[:HeaderLen], h, body)
}

Note that Decapsulate receives the whole datagram, header included, because the header is what the tag covers. If your API hands the data path a body-only slice, you’ve already lost and you’ll find out later.

Then turn your WELCOME into a Result:

res := client.Result{
    TUNName:    tun.Name(),
    AssignedIP: w.AssignedIP,
    Netmask:    w.Netmask,
    Gateway:    server.IP,   // the OUTER address. not w.Gateway!
    MTU:        int(w.MTU),
}

That comment exists because I made the mistake. Twice, in two different protocols, and it cost me hours both times.

Gateway is the server’s outer address. The one you dialed, out on the physical network. It is not the gateway address inside the tunnel, even though every protocol helpfully hands you one of those, and it’s sitting right there in the config document you literally just finished parsing, practically begging you to use it. Its only job is to pin a host route through the physical interface so that your encapsulated packets don’t get routed into the tunnel that’s carrying them.

Fill in the inner gateway instead and here is exactly what happens:

• The handshake succeeds.
• The interface comes up.
• The routes install without error.
• Every single packet leaves by the wrong door, silently, forever.

There is no error message, because nothing errored. This is the worst kind of bug and I wrote it twice, so veepin now checks for it mechanically. If your “outer” address falls inside the tunnel’s own subnet, it says so, by name, at the exact moment you do it.

Then register it, in an init():

func init() { client.Register("toy", parseOptions) }

Done.

veepin (connect|serve|probe) toy works. The NetworkManager plugin can drive it, the management panel renders a form for it, and it shows up in the protocol list. All of that because everything downstream dispatches through the registry, and none of it has ever heard of TOY.

That’s the payoff for the nine fields, too. Change "toy" to "wireguard" or "ikev2" or "softether" in client.Dial(ctx, proto, opts) and nothing else in your program changes.

Step 7: Derive your MTU, don’t pick it

Small thing, big consequence:

const defaultMTU = dataplane.DefaultPathMTU - dataplane.OuterUDP4 - itoy.Overhead
// Overhead = HeaderLen + TagLen = 12 + 8

Comes out to 1452.

The comment next to this constant used to describe exactly that arithmetic while the constant itself said 1400, which is precisely the drift that deriving it prevents. Change the header and the MTU follows. Hardcode it and it silently doesn’t.

Why TOY is not secure, stated plainly

Because a reader who skipped straight to the code deserves to have been told.

1. The keystream repeats. 32 octets, keyed by a counter the attacker can read right off the wire. XOR two ciphertexts with related counters and the pad cancels, leaving you the XOR of two plaintexts. IP headers are extremely predictable. This falls apart immediately.
2. FNV-1a is not a MAC. It’s a hash-table hash. Forging a tag is arithmetic, not search.
3. The proof is replayable within a session and it reveals a digest of the secret.
4. There is no forward secrecy. There is no key exchange at all. Both nonces travel in the clear, so recovering the secret retroactively decrypts every session you ever recorded.
5. CHALLENGE is never authenticated, so an active attacker impersonates the server outright.
6. The KDF barely derives anything. See the hex above.

The left column exists so that the right column has somewhere obvious to be compared against. Each of those maps onto something a real protocol does properly. WireGuard’s Noise_IKpsk2 handshake covers 3 through 5. AES-GCM or ChaCha20-Poly1305 covers 1 and 2. An ephemeral key exchange covers 4. Swap those three things in and you’ve stopped writing a toy.

Both TOY roles print an unmissable warning on startup, every single time, with no flag to silence it. That’s deliberate. The failure mode this could otherwise cause is somebody finding it in a protocol list, noticing that it works fine, and shipping it.

Fin

You have everything you need now. Twelve octets of header, one demux offset, four messages, a key, a tag, a replay window, four methods and a function. Pick your own magic bytes, put your session ID wherever you feel like, and swap the crypto out for something that isn’t a punchline.

Dissect the protocol because it’s fun. Then go check it against something you didn’t write.