Skip to content
🤠 bgp.rodeo

Photo: User_Pascal / Unsplash

An IPv6-mostly home network, delivered by AS202585

ipv6nat64bgpopnsensehome-network

I’m moving house onto an Odido connection. Odido still doesn’t do IPv6. (I once built a website asking about that, and got a letter from their lawyers for it.)

That doesn’t bother the house much, because its IPv6 was never coming from the ISP. It comes from AS202585 over a tunnel to my router at Nikhef. The ISP underneath is a dumb pipe that moves IPv4 packets. This post covers how that’s built, and the parts that bit me along the way.

The longer, grumpier version with more Odido in it is on my personal blog.

IPv6-only, NAT64, IPv6-mostly

People mix these three up, so here they are one by one.

IPv6-only means the clients have no IPv4 address at all: no private one, no CGNAT one, nothing.

NAT64/DNS64 is what makes IPv6-only survivable. When a name has no AAAA record, the resolver synthesises one inside a translation prefix, usually the well-known 64:ff9b::/96. So the IPv4-only ipv4.tlund.se resolves to 64:ff9b::c1b2:3e2e. The client connects over IPv6, the NAT64 gateway strips the prefix off and talks IPv4 to the far end for it. The client never knows.

IPv6-mostly (RFC 8925) is the gentler version. You still run DHCPv4, but you hand out option 108, “IPv6-only preferred”. Clients that understand it switch off their IPv4 stack for a while. Clients that don’t get a normal v4 lease and carry on.

My VLAN 64 is the strict version, with no DHCPv4 at all. I sniffed the wire for six seconds and saw zero IPv4 packets, which is the point.

The clients also need to know where the NAT64 prefix is. RFC 8781 covers that: you put PREF64 in your router advertisements. In radvd:

interface vlan06 {
    AdvSendAdvert on;
    AdvLinkMTU 1468;
    prefix 2a12:bec0:5e1:64::/64 {
        AdvOnLink on;
        AdvAutonomous on;
    };
    nat64prefix 64:ff9b::/96 {
    };
    RDNSS 2a12:bec0:5e1:64::1 {
    };
};

Recent macOS, iOS and Android read nat64prefix and use it for 464XLAT, so IPv4 literals and old IPv4-only sockets still work.

The shape of it

 vlan64 clients (IPv6 only)
        │
        │  RA: /64 + PREF64 64:ff9b::/96 + MTU 1468
        ▼
 OPNsense ──── tayga (NAT64) ──── 192.168.255.0/24 ──┐
        │                                            │
        │  GRE tunnel over IPv4                      │ NAT to
        ▼                                            │ 185.244.24.145
 my router at Nikhef (AS202585) ◄────────────────────┘
        │
        ▼
    the internet
  • OPNsense at home, terminating the ISP connection.
  • A GRE tunnel from OPNsense to the router at Nikhef, carried over IPv4. That detail matters later.
  • BGP inside the tunnel. The home router announces its prefixes to the core, so the core doesn’t need static routes for the house.
  • tayga for NAT64, translating 64:ff9b::/96 into a private v4 pool.
  • unbound with the dns64 module for the DNS half.

The things that bit me

The architecture above is the easy part. These weren’t.

Your NAT64 exit address should be yours

The obvious way to do NAT64 is to translate into a private pool and NAT that out of your WAN address, like any other IPv4 traffic. It works, and it’s what I did at first.

It also wastes the ASN. Every IPv4-only service my v6-only clients reached saw my consumer ISP address, with its consumer geolocation, its consumer reputation, and no PTR record I control. It would also change with every ISP switch.

So the NAT64 pool now translates to an address from AS202585’s own space, routed to the house through the tunnel:

nat on gre0 inet from 192.168.255.0/24 to any -> 185.244.24.145

That’s one more IPv4 address spent than I’d like. In return, my v6-only clients show up on the IPv4 internet as my network, from an address that stays put when the ISP changes.

Two details:

  • Put the address on loopback. It has to be routed to you and be local on the box. Don’t put it on the tunnel interface: a connected /28 pointing down a point-to-point tunnel becomes a routing loop the first time a packet arrives for an unused address in it.
  • Send replies back down the tunnel. If you want the address to answer pings, replies to inbound sessions have to leave through the tunnel, not your default route. Otherwise they get source-NATed to the ISP address and vanish. On pf, that’s reply-to.

Policy routing, not a default route

This one cost me a working LAN.

The instinct is to make the tunnel the default IPv6 gateway, since the tunnel carries my IPv6. Don’t, as long as anything on the network still uses ISP address space.

My normal LAN still took its addressing from the ISP’s prefix delegation. As soon as the tunnel became the default route, that LAN’s traffic went out the tunnel with an ISP source address, and the first BCP38 filter dropped it. Pings from the LAN lost 100%. Pings from the v6-mostly VLAN were fine. That’s confusing until you remember what you just changed.

The fix is a firewall rule with a gateway on it. Only the VLAN’s traffic is policy-routed into the tunnel, and everything else keeps the ISP default. In OPNsense, that means setting Gateway on the interface’s pass rule and leaving the tunnel gateway at defaultgw=0.

A default route through the tunnel only makes sense once nothing on the network uses ISP space any more.

Mind the MTU, and tell the clients

GRE over PPPoE leaves 1492 − 24 = 1468 bytes. IPv6 routers don’t fragment, so getting this wrong gives the worst kind of symptom: small packets fine, DNS fine, ping fine, web pages hang forever.

You can rely on Path MTU Discovery and hope nobody filters ICMPv6 Packet Too Big. You can also advertise AdvLinkMTU 1468 in the RA and stop worrying. Do both.

A testing trap: FreeBSD’s ping6 sets IPV6_USE_MIN_MTU by default and fragments everything at 1280. I spent a while convinced I had a hard 1280-byte ceiling before remembering the -u flag. The ceiling was in my test tool.

DNS64 applies to everyone

unbound’s dns64 module has no notion of “only for this VLAN” unless you build views for it. Mine has none. So my normal dual-stack LAN also resolves IPv4-only names into 64:ff9b::, prefers the synthesised AAAA, and reaches those sites through NAT64.

I found out when a laptop on the normal LAN reported my NAT64 exit address as its IPv4 address. Nothing was broken, but a machine with a perfectly good IPv4 path was detouring through a tunnel and a translator.

If you want NAT64 scoped to one VLAN, you need a separate resolver instance or unbound views. Decide on purpose rather than finding out from a “what is my IP” page.

Config that only exists in RAM

This one has nothing to do with IPv6 and was entirely self-inflicted, but it did take a prefix off the internet.

Some of the FRR config on the home router had been typed into vtysh and never saved into OPNsense’s config. It worked fine until a service restart regenerated frr.conf from the saved config. Three settings that had only ever lived in the running process disappeared, and AS202585’s /48 dropped out of the global routing table for four minutes.

If a management layer owns a daemon’s config, make your changes through the management layer. And before restarting anything, diff what’s running against what’s stored.

Switching ISPs

What changes when the ISP swaps? Almost nothing, which is why it’s built this way.

The tunnel runs over IPv4 and is anchored to whatever the WAN address happens to be. The NAT64 exit address belongs to AS202585 and is routed through the tunnel. The BGP session lives inside the tunnel. None of that cares who provides the pipe, or whether the pipe has IPv6.

Two things do need work.

The LAN needs renumbering. The normal LAN uses the ISP’s delegated prefix today. Odido won’t delegate anything, so the LAN moves to a static /64 from my own /48. Then every network in the house is on AS202585 space, and the tunnel can finally be the default IPv6 gateway.

GRE doesn’t handle a dynamic address. Odido doesn’t do CGNAT, so you get a public IPv4 address. It’s dynamic on a consumer subscription, though, and GRE has no concept of roaming: the far end has my current address configured, and a renumber at 3am breaks the tunnel until I fix it by hand.

There are three reasonable answers: pay for a static business address, script the GRE endpoint update on the core router whenever the WAN address changes, or change transport.

I’m going with WireGuard. The core router lists the home peer by public key with no endpoint, my side initiates with a keepalive, and a renumber re-pins on the next handshake. It also authenticates and encrypts a tunnel that currently carries a BGP session in plaintext, which is worth doing whoever the ISP is.

It costs bytes. GRE adds 24 bytes of overhead, and WireGuard over IPv4 adds 60. On a 1500-byte path that’s 1476 against 1440. On a design that depends on advertising an honest MTU, 36 bytes is a real trade, but I’ll take roaming and authentication for it. If your ISP’s address turns out to be stable in practice, scripting the GRE endpoint and keeping the bytes is perfectly defensible.

Whichever you choose, alert on the tunnel going down. Once the LAN lives on tunnelled address space, a dead tunnel means no IPv6 in the house at all, not just a degraded VLAN.

Worth it?

The house now has an IPv6-only segment on address space I hold myself, behind an ISP that has spent a decade not deploying IPv6, and moving to that ISP will be a non-event for it.

This is what a small ASN is good for. You don’t need a big network to make one useful. It can also make your own home network independent of whichever ISP happens to serve your street.

← Back to all posts