A Look at the Cluster Platform Layer 2 Overlay Network SystemLet's bring back networking simplicity!
Last edit: 1/13/2020
In the ongoing effort to write Cluster Platform, the lean distributed computing system, I hit a challenge: networking.
The Cluster Platform has some really unique networking requirements. As a distributed system it has to be able to work both completely off-grid, accross globally distributed datacenters and low-power IoT devices. Ultimately, the Cluster Platform should be able to serve as a replacement for the internet, while at the same time being able to run on it.
Furthermore, as it is the case with the internet itself, decentralization is necessary for both speed and security. One datacenter of a Cluster Platform cluster might be connected to another datacenter over a low-bandwidth OpenSNET link but has a 10G link internally; it would be awesome if the high internal speed could be leveraged while not breaking inter-datacenter networking, which is only possible with a decentral system.
Additionally, the networking stack must be implemented as a "cloud-native" microservice system. This is the case, amongst many other reasons, due to the fact that datacenters must be able to be administered remotely. Off-grid parts of a Cluster Platform cluster could be lights off for multiple months; there must be no single point of failure, neither for the cluster as a collective, nor for the node as an individual.
Using these two services, it is possible to create decentralized high-speed P2P OSI layer 2 overlay networks, spanning everything from multiple cloud hosts in multiple datacenters around the globe to your home network. Furthermore it is possible for a node (called "edge" from now on) to join many different overlay networks; for example, if you want to run a Minecraft server on one of your cloud hosts and play a game with some of your friends, you can simply add another network just for you and your peers without giving them access to your production CI/CD system.
Within each overlay network (called "community" from now on), all edges talk in a peer-to-peer fashion with strong AES256 encryption. Each community can be thought of as an ethernet switch; there is full DHCP and broad/multicast support. A central "supernode" allows for the edges which can't talk P2P (if for example NAT hole punching doesn't work) to still communicate with the other edges; however, due to the fact that the edges can route traffic through the network like relays, such cases due rarely occur.
As I've lined out before, this entire system is architected in such a way that there are remote management capabilities. This is achieved by splitting each subsystem into a client-server architecture; this way, one has to only deploy the gRPC server (daemon) to the host which should have the subsystem's capabilities, but can manage it remotely from a management machine using the client CLIs. PWA management UIs are also planned for the future.
For the following, I will use three cloud which I'll be administering them from my laptop. The first cloud host is the only one that technically needs to be publically reachable; it will host the supernode daemon. For the sake of simplicity, I will also deploy a DHCP server daemon and a corresponding edge daemon on it, but the latter could technically run on any edge. The other two hosts will both run an edge and a DHCP client daemon.
Building the Binaries
Because there is currently no CI/CD system set up, you will have to build the binaries yourself for now. But no worries; everything is written in Go, so this process should be simple enough.
Let's start with
% git clone https://github.com/pojntfx/gon2n.git % cd gon2n % go generate ./... % go build -o supernoded cmd/supernoded/main.go % go build -o edged cmd/edged/main.go % go build -o supernodectl cmd/supernodectl/main.go % go build -o edgectl cmd/edgectl/main.go
Now, let's do the same for
% git clone https://github.com/pojntfx/go-isc-dhcp.git % cd go-isc-dhcp % go generate ./... % go build -o dhcpdd cmd/dhcpdd/main.go % go build -o dhclientd cmd/dhclientd/main.go % go build -o dhcpdctl cmd/dhcpdctl/main.go % go build -o dhclientctl cmd/dhclientctl/main.go
Distributing the Daemons
Let's distribute the daemons you've just built. To do so, let's utilize
scp; be sure to use your own hostnames:
# For the first host % scp gon2n/supernoded gon2n/edged go-isc-dhcp/dhcpd root@host-1:/usr/local/sbin # Repeat the following for the remaining hosts % scp gon2n/edged go-isc-dhcp/dhclientd root@host-1:/usr/local/sbin
Installing the Clients
% sudo install gon2n/supernodectl gon2n/edgectl go-isc-dhcp/dhcpdctl go-isc-dhcp/dhclientctl /usr/local/bin
Editors note: Single binaries are awesome.
Daemon Setup for the First Host
All daemons (and clients) can be configured via either environment variables, command line flags, configuration files or a combination of the three. For this demonstration, I'll be using the third option; configuration files.
First, let's save the following configuration files on the first host; be sure to use your own hostnames and open up the ports on the firewall:
# /etc/supernoded.yaml supernoded: listenHostPort: host-1:1236
# /etc/edged.yaml edged: listenHostPort: host-1:1235
# /etc/dhcpdd.yaml dhcpdd: listenHostPort: host-1:1240
That's all the server-side setup you need; all of the rest will be done client-side.
Let's start the daemons (on the first host):
% supernoded -f /etc/supernoded.yaml # In another terminal % sudo edged -f /etc/edged.yaml # This needs to manage TUN/TAP interfaces, so CAP_NET_ADMIN is required # In another terminal % sudo dhcpdd -f /etc/dhcpdd.yaml # This needs to bind to port 67, so root priviledges are required
Daemon Setup for the Remaining Hosts
First, let's save the following configuration files on the remaining hosts; be sure to use your own hostnames and open up the ports on the firewall:
# /etc/edged.yaml edged: listenHostPort: host-n:1235
# /etc/dhclientd.yaml dhclientd: listenHostPort: host-n:1241
That's all the server-side setup you need; all of the rest will be done client-side.
Let's start the daemons (on the remaining hosts):
% sudo edged -f /etc/edged.yaml # This needs to manage TUN/TAP interfaces, so CAP_NET_ADMIN is required # In another terminal % sudo dhclientd -f /etc/dhclientd.yaml # This needs to manage network interfaces, so CAP_NET_ADMIN is required
Configuring the Daemons
As I've outlined before, almost all configuration is done with the client CLIs, so now that the clients are installed and the daemons are running, let's go ahead!
First, create the following configuration files (be sure to use your own hostnames, unique MAC addresses and open up the ports on the firewall):
# supernode.yaml supernode: listenPort: 1234 managementPort: 5645
# edge-host-1.yaml edge: allowP2P: true allowRouting: true communityName: mynetwork disablePMTUDiscovery: false disableMulticast: false dynamicIPMode: false encryptionKey: mysecretkey localPort: 0 managementPort: 5644 registerInterval: 1 registerTTL: 1 supernodeHostPort: host-1:1234 typeOfService: 16 encryptionMethod: 2 deviceName: edge0 addressMode: static deviceIP: 192.168.1.1 deviceNetmask: 255.255.255.0 deviceMACAddress: DE:AD:BE:EF:01:10 MTU: 1500
# edge-host-n.yaml edge: allowP2P: true allowRouting: true communityName: mynetwork disablePMTUDiscovery: false disableMulticast: false dynamicIPMode: true encryptionKey: mysecretkey localPort: 0 managementPort: 5645 registerInterval: 1 registerTTL: 1 supernodeHostPort: host-1:1234 typeOfService: 16 encryptionMethod: 2 deviceName: edge0 addressMode: dhcp deviceIP: 0.0.0.0 deviceNetmask: 255.255.255.0 deviceMACAddress: DE:AD:BE:EF:01:11 MTU: 1500
# dhcpd.yaml dhcpd: device: edge0 subnets: - netmask: 255.255.255.0 network: 192.168.1.0 range: start: 192.168.1.10 end: 192.168.1.100
# dhclient.yaml dhclient: device: edge0
Pretty simple, right? These configuration files can be thought of as "Kubernetes objects"; in the next step we'll
# For the first host % supernodectl -s host-1:1236 -f supernode.yaml % edgectlctl -s host-1:1235 -f edge-host-1.yaml % dhcpdctl -s host-1:1240 -f dhcpd.yaml # Repeat the following for the remaining hosts % edgectlctl -s host-n:1235 -f edge-host-n.yaml % dhclientctl -s host-n:1241 -f dhclient.yaml
You may now
delete them as well, please see the
go-isc-dhcp documentation for usage instructions or just try it out:
% supernodectl get -s host-1:1236 ID LISTEN PORT 915adb75-73f0-41e3-81ae-caa238c329ef 1234
That's it! You've got a fully functional peer-to-peer OSI layer 2 overlay network with speed of up to 500 megabytes/second (or more; this is limited only by the encryption speed).
go-isc-dhcp projects will serve as a fundamental part of the future Cluster Platform. But even right now they are useful, and if you want a really fast layer 2 VPN, feel free to use these components; they are free/libre and open source software!