What is dn42?
DN42 (wikipedia) is a big internet overlay network (also called darknet). It’s less about the services within this network and more about the routing protocols. The participants have to connect to each other usually over a vpn tunnel and through those vpn tunnels they establish a bgp session. Instead of building a small lab somewhere, this projects provides a big network of networks with more than 1500 subnets.
- Are you interested how the internet is build and how local internet registries (LIR) and autonomous systems (AS) are communicating with each other?
- What has a LIR to do? How do they allocate AS numbers and ip ranges?
Getting Started
You are interested and want to build your own peer? A raspberry pi or a small VPS which is always online is enough to connect. There is a great article how to get started on the dn42 wiki.
I really recommend to use the dn42 PeerFinder. Enter your public ip address and after a few minutes you know which peers are near to you. There are multiple peers which provide automatic peering. Usually through a website or a telegram bot it’s easily to peer with them.
How to get started
- First create your entries in the dn42 registry. Take your time, this needs a few hours or a day to get merged and is a required step. My suggestion is, to also request a (really small) ipv4 range.
- Create your node and install the whole operating system as you like.
- Request a first peering with one of the AS which provide automatic peering. For me it was really helpful to get started and didn’t need to contact people directly.
- Implement the wireguard peering and test if you can reach your peer.
- Implement a BGP peering with your peer.
More about those steps in detail follows.
Wireguard Setup
I differentiate between the outer and the inner vpn traffic. The routing tables are also separated. Usually the traffic is handled in the VRF default. I didn’t want to change that and implemented the vpn internal traffic in a separate VRF.
Outer Traffic
The vpn outer traffic is routed in the VRF default. The routing table is relatively small, only the interface eth0 is directly connected to this VRF. The routing table contains the default route to the default gateway.
+---------------+
Internet -----+ VRF default |
+-------+-------+
|
+----+---+
| eth0 |
+--------+
Routing table
dn42.ams(bash ~)# ip r
default via 192.0.2.1 dev eth0 proto dhcp src 192.0.2.2 metric 100
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.2 metric 100
192.0.2.1 dev eth0 proto dhcp scope link src 192.0.2.2 metric 100
Inner Traffic
Each AS is sending it’s known routes to my node and those routes are placed in the VRF inner. All routes in VRF inner are redistributed to each AS. In the following example, I have a peering with AS 4242421801 and 4242422301. My own AS is AS4242422005 (which isn’t shown in this graphic as I don’t have an interface representing it).
+------------------------------------+
| VRF inner |
| +--------------+ +--------------+ |
| | AS4242421801 | | AS4242422301 | |
| +--------------+ +--------------+ |
+------------------------------------+
Wireguard Config
Personally I implement each peering on a separate wireguard interface. Some peering partners require you to have a specific ip address and having multiple ip addresses on the same interface is more complex as needed. As the interface has to stick to a dedicated VRF and each of my peerings can potentially send ip packages from the same sources (it’s dynamic routing, a source can send traffic over each of your peerings), I ended up in creating the following config for wg-quick:
[Interface]
PrivateKey = UPAbkSwzQ9ekLnhK1NL/Dc/F6JhJmksVod/Y9vfObX4=
ListenPort = 21801
PostUp = ip link set dev %i master inner
PostUp = ip address add dev %i fe80::2005:1801/128 peer fe80::1801
Table = off
[Peer]
PublicKey = jMHiiz3Soq6lHZEbsqNRU+eVMDmsPvDLyFu4MboBiCk=
AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
Endpoint = 198.51.100.1:22005
- ListenPort: I personally use, as multiple others do, 20000+(last 4 digits of the remote AS). In this case, AS4242421801 results in 21801.
- PostUp 1) move the interface to VRF inner
- PostUp 2) set the ip address and peer (direct peering, no subnet in between)
- My ip address is fe80::(last 4 digits of my AS):(last 4 digits of remote AS)
- Others ip address is fe80::(last 4 digits of remote AS) or if provided, I use the provided ip address
- I always use only link-local ipv6 addresses
- Table: don’t write the allowed ips to the routing table. Other interfaces won’t come up if the routes already exist and we use BGP, no need to write them to the routing table.
How can I test if this link is working? Nothing easier than that:
ip vrf exec inner ping -c 1 fe80::1801%as4242421801
BGP Setup
I used FRR already in the past. I’m used to it, even if it’s not always the easiest solution. If you’re more familiar with Linux, perhaps you like BIRD more. But I’ll explain my FRR (version 10) config here.
Following I will describe the most important configuration snippets and the more complex ones to understand.
vrf inner
ip route 192.0.2.0/28 blackhole
ipv6 route 2001:db8::/48 blackhole
The kernel must know the full subnets. Either they are configured on an interface or the whole range is blackholed (which doesn’t mean the kernel will blackhole everything, as more specific routes have more presedence).
interface inner
ip address 192.0.2.1/32
ipv6 address 2001:db8::1/128
I added an ip address to my VRF interface. This ip address is used by the node itself if it wants to communicate with other nodes in the whole dn42 network. I configured this on my VRF interface, as this worked, and attaching the ip address to the link-local interface didn’t work (kernel used as source address the IPv6 link-local).
router bgp 4242422001 vrf inner
bgp router-id 192.0.2.1
bgp log-neighbor-changes
bgp default show-hostname
bgp deterministic-med
neighbor pg-dn42 peer-group
neighbor pg-dn42 remote-as external
neighbor pg-dn42 ebgp-multihop 3
neighbor pg-dn42 capability dynamic
neighbor pg-dn42 capability extended-nexthop
neighbor fe80::1801 peer-group pg-dn42
neighbor fe80::1801 interface as4242421801
neighbor fe80::1801 description 'https://as4242421801.example.net'
neighbor fe80::2301 peer-group pg-dn42
neighbor fe80::2301 interface as4242422301
neighbor fe80::2301 description 'https://as4242422301.example.net'
neighbor fd42:4242:2601:ac12::1 remote-as 4242422602
neighbor fd42:4242:2601:ac12::1 description 'https://lg.collector.dn42'
neighbor fd42:4242:2601:ac12::1 ebgp-multihop 10
There the BGP magic is. The router knows it’s attached to the VRF inner. I have a peer-group pg-dn42 as this helps below to do some configurations only once. Each neighbor has to be added to this peer-group and as I communicate over ipv6 link-local addresses I have to specify which interface this neighbor is located at. I also enabled the capabilities extended-nexthop as this allows me to forward ipv4 traffic over ipv6 link-local addresses.
!
address-family ipv4 unicast
network 192.0.2.0/28
neighbor pg-dn42 next-hop-self
neighbor pg-dn42 soft-reconfiguration inbound
neighbor pg-dn42 route-map rm-dn42-in-4 in
neighbor pg-dn42 route-map rm-dn42-out-4 out
exit-address-family
!
address-family ipv6 unicast
network 2001:db8::/48
neighbor pg-dn42 activate
neighbor pg-dn42 soft-reconfiguration inbound
neighbor pg-dn42 route-map rm-dn42-in-6 in
neighbor pg-dn42 route-map rm-dn42-out-6 out
exit-address-family
Inside the router both address families must be configured too. There we define which networks we want to redistribute (our networks, but not connected onces, as most probably those are not registered in dn42 to the AS). For ipv6 we have to activate the neighbors, that’s not needed for ipv4. And I added some route-maps for the dn42 neighbors which will filter outgoing and ingoing routes. Those are separated between ipv4 and ipv6 and separated between in and out.
ip prefix-list pl-our-4 seq 5 permit 192.0.2.0/24 le 32
ip prefix-list pl-dn42-4 seq 1 deny 172.22.166.0/24 le 32
ip prefix-list pl-dn42-4 seq 1001 permit 172.20.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1002 permit 172.21.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1003 permit 172.22.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1004 permit 172.23.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1100 permit 172.20.0.0/14 ge 21 le 29
ip prefix-list pl-dn42-4 seq 2001 permit 10.100.0.0/14 le 32
ip prefix-list pl-dn42-4 seq 2002 permit 10.127.0.0/16 le 32
ip prefix-list pl-dn42-4 seq 2003 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list pl-dn42-4 seq 3001 permit 172.31.0.0/16 le 32
!
ipv6 prefix-list pl-our-6 seq 5 permit 2001:db8::/32 le 128
ipv6 prefix-list pl-dn42-6 seq 10 deny fdd6:6302:7c04::/48 le 128
ipv6 prefix-list pl-dn42-6 seq 1001 permit fd00::/8 ge 44 le 64
ipv6 prefix-list pl-too-spec-6 seq 5 permit ::/0 ge 49 le 128
This bunch of ip prefix-lists are only lists. Not used directly in the router but only in route-maps. Each prefix-list is usually written as “permit”. This means those ip addresses are positively added to the list.
pl-our-4 and pl-our-6 specify which prefixes are disallowed to receive.
pl-dn42-4 and pl-dn42-6 specify which prefixes are allowed to receive.
route-map rm-dn42-in-4 deny 10
match ip address prefix-list pl-our-4
exit
!
route-map rm-dn42-in-4 permit 40
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-in-4 deny 9999
exit
The ingoing route-map denies all ips which are added positively to pl-our-4. If a route doesn’t match it goes further to the second statement where all prefixes are allowed which are part of dn42. If the route doesn’t match the last statement denies the prefix.
route-map rm-dn42-out-4 permit 10
match ip address prefix-list pl-our-4
set community 64511:41 64511:1756
exit
!
route-map rm-dn42-out-4 permit 40
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-out-4 deny 9999
exit
The outgoing route-map allows our own prefixes and adds a bunch of communities.
route-map rm-dn42-in-6 deny 10
match ipv6 address prefix-list pl-our-6
exit
!
route-map rm-dn42-in-6 permit 40
match ipv6 address prefix-list pl-dn42-6
set src 2001:db8::1
exit
!
route-map rm-dn42-in-6 deny 9999
exit
!
route-map rm-dn42-out-6 permit 10
match ipv6 address prefix-list pl-our-6
set community 64511:41 64511:1756
exit
!
route-map rm-dn42-out-6 permit 40
match ipv6 address prefix-list pl-dn42-6
exit
!
route-map rm-dn42-out-6 deny 9999
exit
In ipv6 it’s the same as for ipv4 just with the other prefix-lists made for ipv6. One special line was added to those route-maps. In route-map rm-dn42-in-6 permit 40 is a line set src. This is used to tells FRR to install routes learned from BGP with this address as source address. Normally, if an interface has only a ipv6 link-local address, the source address is this link-local address. This won’t work if the destination isn’t one of the direct peers. The same address must be present on an interface inside the VRF. That’s the reason why I added this address to the VRF interface.
Route Collector
dn42 has a route collector. More peers to this route collector means a better view on the network. Providing your routing information means benefit for the network. I added the following configs to get it working.
router bgp 4242422001 vrf inner
neighbor fd42:4242:2601:ac12::1 remote-as 4242422602
neighbor fd42:4242:2601:ac12::1 description 'https://lg.collector.dn42'
neighbor fd42:4242:2601:ac12::1 ebgp-multihop 10
!
address-family ipv4 unicast
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-in-4 in
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-out-4 out
exit-address-family
!
address-family ipv6 unicast
neighbor fd42:4242:2601:ac12::1 activate
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-in-6 in
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-out-6 out
exit-address-family
!
route-map rm-dn42-rs-in-4 deny 10
exit
!
route-map rm-dn42-rs-out-4 permit 10
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-rs-out-4 deny 9999
exit
!
route-map rm-dn42-rs-in-6 deny 10
exit
!
route-map rm-dn42-rs-out-6 permit 10
match ipv6 address prefix-list pl-dn42-6
exit
!
route-map rm-dn42-rs-out-6 deny 9999
exit
Full FRR Configuration
frr version 10.0
frr defaults traditional
hostname dn42
log syslog informational
service integrated-vtysh-config
!
ip prefix-list pl-dn42-4 seq 1 deny 172.22.166.0/24 le 32
ip prefix-list pl-dn42-4 seq 1001 permit 172.20.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1002 permit 172.21.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1003 permit 172.22.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1004 permit 172.23.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1100 permit 172.20.0.0/14 ge 21 le 29
ip prefix-list pl-dn42-4 seq 2001 permit 10.100.0.0/14 le 32
ip prefix-list pl-dn42-4 seq 2002 permit 10.127.0.0/16 le 32
ip prefix-list pl-dn42-4 seq 2003 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list pl-dn42-4 seq 3001 permit 172.31.0.0/16 le 32
ip prefix-list pl-our-4 seq 5 permit 192.0.2.0/24 le 32
ip prefix-list pl-our-rtbh-4 seq 5 permit 192.0.2.0/24 ge 32
!
ipv6 prefix-list pl-dn42-6 seq 10 deny fdd6:6302:7c04::/48 le 128
ipv6 prefix-list pl-dn42-6 seq 1001 permit fd00::/8 ge 44 le 64
ipv6 prefix-list pl-too-spec-6 seq 5 permit ::/0 ge 49 le 128
ipv6 prefix-list pl-our-6 seq 5 permit 2001:db8::/48 le 128
ipv6 prefix-list pl-our-rtbh-6 seq 5 permit 2001:db8::/48 ge 128
!
vrf inner
ip route 192.0.2.0/28 blackhole
ipv6 route 2001:db8::/48 blackhole
exit-vrf
!
interface inner
ip address 192.0.2.1/32
ipv6 address 2001:db8::1/128
exit
!
router bgp 4242422001 vrf inner
bgp router-id 192.0.2.1
bgp log-neighbor-changes
bgp default show-hostname
bgp deterministic-med
bgp graceful-restart
neighbor pg-dn42 peer-group
neighbor pg-dn42 remote-as external
neighbor pg-dn42 ebgp-multihop 3
neighbor pg-dn42 capability dynamic
neighbor pg-dn42 capability extended-nexthop
neighbor fe80::1801 peer-group pg-dn42
neighbor fe80::1801 interface as4242421801
neighbor fe80::1801 description 'https://as4242421801.example.net'
neighbor fe80::2301 peer-group pg-dn42
neighbor fe80::2301 interface as4242422301
neighbor fe80::2301 description 'https://as4242422301.example.net'
neighbor fd42:4242:2601:ac12::1 remote-as 4242422602
neighbor fd42:4242:2601:ac12::1 description 'https://lg.collector.dn42'
neighbor fd42:4242:2601:ac12::1 ebgp-multihop 10
!
address-family ipv4 unicast
network 192.0.2.0/28
redistribute static
neighbor pg-dn42 next-hop-self
neighbor pg-dn42 soft-reconfiguration inbound
neighbor pg-dn42 route-map rm-dn42-in-4 in
neighbor pg-dn42 route-map rm-dn42-out-4 out
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-in-4 in
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-out-4 out
exit-address-family
!
address-family ipv6 unicast
network 2001:db8::/48
redistribute static
neighbor pg-dn42 activate
neighbor pg-dn42 next-hop-self
neighbor pg-dn42 soft-reconfiguration inbound
neighbor pg-dn42 route-map rm-dn42-in-6 in
neighbor pg-dn42 route-map rm-dn42-out-6 out
neighbor fd42:4242:2601:ac12::1 activate
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-in-6 in
neighbor fd42:4242:2601:ac12::1 route-map rm-dn42-rs-out-6 out
exit-address-family
exit
!
ip prefix-list pl-our-4 seq 5 permit 192.0.2.0/24 le 32
ip prefix-list pl-dn42-4 seq 1 deny 172.22.166.0/24 le 32
ip prefix-list pl-dn42-4 seq 1001 permit 172.20.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1002 permit 172.21.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1003 permit 172.22.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1004 permit 172.23.0.0/24 ge 28 le 32
ip prefix-list pl-dn42-4 seq 1100 permit 172.20.0.0/14 ge 21 le 29
ip prefix-list pl-dn42-4 seq 2001 permit 10.100.0.0/14 le 32
ip prefix-list pl-dn42-4 seq 2002 permit 10.127.0.0/16 le 32
ip prefix-list pl-dn42-4 seq 2003 permit 10.0.0.0/8 ge 15 le 24
ip prefix-list pl-dn42-4 seq 3001 permit 172.31.0.0/16 le 32
!
ipv6 prefix-list pl-our-6 seq 5 permit 2001:db8::/32 le 128
ipv6 prefix-list pl-dn42-6 seq 10 deny fdd6:6302:7c04::/48 le 128
ipv6 prefix-list pl-dn42-6 seq 1001 permit fd00::/8 ge 44 le 64
ipv6 prefix-list pl-too-spec-6 seq 5 permit ::/0 ge 49 le 128
!
route-map rm-dn42-in-4 deny 10
match ip address prefix-list pl-our-4
exit
!
route-map rm-dn42-in-4 permit 40
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-in-4 deny 9999
exit
!
route-map rm-dn42-in-6 deny 10
match ipv6 address prefix-list pl-our-6
exit
!
route-map rm-dn42-in-6 permit 40
match ipv6 address prefix-list pl-dn42-6
set src 2001:db8::1
exit
!
route-map rm-dn42-in-6 deny 9999
exit
!
route-map rm-dn42-out-4 permit 10
match ip address prefix-list pl-our-4
set community 64511:41 64511:1756
exit
!
route-map rm-dn42-out-4 permit 40
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-out-4 deny 9999
exit
!
route-map rm-dn42-out-6 permit 10
match ipv6 address prefix-list pl-our-6
set community 64511:41 64511:1756
exit
!
route-map rm-dn42-out-6 permit 40
match ipv6 address prefix-list pl-dn42-6
exit
!
route-map rm-dn42-out-6 deny 9999
exit
!
route-map rm-dn42-rs-in-4 deny 10
exit
!
route-map rm-dn42-rs-out-4 permit 10
match ip address prefix-list pl-dn42-4
exit
!
route-map rm-dn42-rs-out-4 deny 9999
exit
!
route-map rm-dn42-rs-in-6 deny 10
exit
!
route-map rm-dn42-rs-out-6 permit 10
match ipv6 address prefix-list pl-dn42-6
exit
!
route-map rm-dn42-rs-out-6 deny 9999
exit
!
BGP debugging
All those commands are executed in vtysh. Which is the command line utility for zebra.
Get a first overview about the BGP peers, how many prefixes were recevied, how many sent and so on.
dn42.ams# sh bgp vrf inner summary
IPv4 Unicast Summary (VRF inner):
BGP router identifier 192.0.2.1, local AS number 4242422001 vrf-id 3
BGP table version 38403
RIB entries 1634, using 153 KiB of memory
Peers 4, using 80 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
fe80::1801 4 4242421801 82666 61819 38403 0 0 1d22h44m 649 856 N/A
fe80::2301 4 4242422301 99941 65486 38403 0 0 1d22h44m 847 856 N/A
Total number of neighbors 2
IPv6 Unicast Summary (VRF inner):
BGP router identifier 192.0.2.1, local AS number 4242422001 vrf-id 3
BGP table version 38252
RIB entries 1477, using 138 KiB of memory
Peers 4, using 80 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
fe80::1801 4 4242421801 82666 61819 38252 0 0 1d22h44m 769 768 N/A
fe80::2301 4 4242422301 99941 65486 38252 0 0 1d22h44m 760 768 N/A
Total number of neighbors 2
Get deeper insights how many prefixes were received, accepted, dropped, and so on.
dn42.ams# sh bgp vrf inner ipv4 neighbors fe80::1801 prefix-counts
Prefix counts for fe80::1801, IPv4 Unicast
PfxCt: 649
Counts from RIB table walk:
Adj-in: 649
Damped: 0
Removed: 0
History: 0
Stale: 0
Valid: 649
All RIB: 649
PfxCt counted: 649
PfxCt Best Selected: 46
Useable: 649
List the hole routing table of VRF inner may help in some cases. Also which routing protocol installed them to the kernel, where are they originating from and so on.
dn42.ams# sh ipv6 route vrf inner
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF inner:
B>* fd00:21:259e::/48 [20/0] via fe80::3088:204, as4242423088, weight 1, 1d22h47m
B>* fd00:bb:5bf3::/48 [20/0] via fe80::3088:204, as4242423088, weight 1, 1d22h47m
[...]
Get a specific route and list where it originates from.
dn42.ams# sh ipv6 route vrf inner fd42:d42:d42:54::1
Routing entry for fd42:d42:d42:54::/64
Known via "bgp", distance 20, metric 0, vrf inner, best
Last update 1d22h48m ago
* fe80::3088:204, via as4242423088, weight 1