Flow and Retransmission Control Protocol (FRCP) implementation
With the longer weekend I had some fun implementing (parts of) the Flow and Retransmission Control Protocol (FRCP) to the point that it’s stable enough to bring you a very quick demo of it.
FRCP is the Ouroboros alternative to TCP / QUIC / LLC. It assures delivery of packets when the network itself isn’t very reliable.
The setup is simple: we run Ouroboros over the Ethernet loopback adapter lo,
systemctl restart ouroboros
irm i b t eth-dix l dix n dix dev lo
to which we add some impairment qdisc:
$ sudo tc qdisc add dev lo root netem loss 8% duplicate 3% reorder 10% delay 1
This causes the link to lose, duplicate and reorder packets.
We can use the oping tool to uses different QoS specs and watch the behaviour. Quality-of-Service (QoS) specs are a technology-agnostic way to request a network service (current status - not finalized yet). I’ll also capture tcpdump output.
We start an oping server and tell Ouroboros for it to listen to the name “oping”:
#bind the program oping to the name oping
irm b prog oping n oping
#register the name oping in the Ethernet layer that is attached to the loopback
irm n r oping l dix
#run the oping server
oping -l
We’ll now send 20 pings. If you try this, it can be that the flow allocation fails, due to the loss of a flow allocation packet (a bit similar to TCP losing the first SYN). The oping client currently doesn’t retry flow allocation. The default payload for oping is 64 bytes (of zeros); oping waits 2 seconds for all packets it has sent. It doesn’t detect duplicates.
Let’s first look at the raw QoS cube. That’s like best-effort UDP/IP. In Ouroboros, however, it doesn’t require a packet header at all.
First, the output of the client using a raw QoS cube:
$ oping -n oping -c 20 -i 200ms -q raw
Pinging oping with 64 bytes of data (20 packets):
64 bytes from oping: seq=0 time=0.880 ms
64 bytes from oping: seq=1 time=0.742 ms
64 bytes from oping: seq=4 time=1.303 ms
64 bytes from oping: seq=6 time=0.739 ms
64 bytes from oping: seq=6 time=0.771 ms [out-of-order]
64 bytes from oping: seq=6 time=0.789 ms [out-of-order]
64 bytes from oping: seq=7 time=0.717 ms
64 bytes from oping: seq=8 time=0.759 ms
64 bytes from oping: seq=9 time=0.716 ms
64 bytes from oping: seq=10 time=0.729 ms
64 bytes from oping: seq=11 time=0.720 ms
64 bytes from oping: seq=12 time=0.718 ms
64 bytes from oping: seq=13 time=0.722 ms
64 bytes from oping: seq=14 time=0.700 ms
64 bytes from oping: seq=16 time=0.670 ms
64 bytes from oping: seq=17 time=0.712 ms
64 bytes from oping: seq=18 time=0.716 ms
64 bytes from oping: seq=19 time=0.674 ms
Server timed out.
--- oping ping statistics ---
20 packets transmitted, 18 received, 2 out-of-order, 10% packet loss, time: 6004.273 ms
rtt min/avg/max/mdev = 0.670/0.765/1.303/0.142 ms
The netem did a good job of jumbling up the traffic! There were a couple out-of-order, duplicates, and quite some packets lost.
Let’s dig into an Ethernet frame captured from the “wire”. The most interesting thing its small total size: 82 bytes.
13:37:25.875092 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xa000), length 82:
0x0000: 0042 0040 0000 0001 0000 0011 e90c 0000 .B.@............
0x0010: 0000 0000 203f 350f 0000 0000 0000 0000 .....?5.........
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000
The first 12 bytes are the two MAC addresses (all zeros), then 2 bytes for the “Ethertype” (the default for an Ouroboros layer is 0xa000, so you can create more layers and seperate them by Ethertype1. The Ethernet Payload is thus 68 bytes. The Ouroboros ipcpd-eth-dix adds and extra header of 4 bytes with 2 extra “fields”. The first field we needed to take over from our Data Transfer protocol: the Endpoint Identifier that identifies the flow. The ipcpd-eth-dix has two endpoints, one for the client and one for the server. 0x0042 (66) is the destination EID of the server, 0x0043 (67) is the destination EID of the client. The second field is the length of the payload in octets, 0x0040 = 64. This is needed because Ethernet II has a minimum frame size of 64 bytes and pads smaller frames (called runt frames)2. The remaining 64 bytes are the oping payload, giving us an 82 byte packet.
That’s it for the raw QoS. The next one is voice. A voice service usually requires packets to be delivered with little delay and jitter (i.e. ASAP). Out-of-order packets are rejected since they cause artifacts in the audio output. The voice QoS will enable FRCP, because it needs to track sequence numbers.
$ oping -n oping -c 20 -i 200ms -q voice
Pinging oping with 64 bytes of data (20 packets):
64 bytes from oping: seq=0 time=0.860 ms
64 bytes from oping: seq=2 time=0.704 ms
64 bytes from oping: seq=3 time=0.721 ms
64 bytes from oping: seq=4 time=0.706 ms
64 bytes from oping: seq=5 time=0.721 ms
64 bytes from oping: seq=6 time=0.710 ms
64 bytes from oping: seq=7 time=0.721 ms
64 bytes from oping: seq=8 time=0.691 ms
64 bytes from oping: seq=10 time=0.691 ms
64 bytes from oping: seq=12 time=0.702 ms
64 bytes from oping: seq=13 time=0.730 ms
64 bytes from oping: seq=14 time=0.716 ms
64 bytes from oping: seq=15 time=0.725 ms
64 bytes from oping: seq=16 time=0.709 ms
64 bytes from oping: seq=17 time=0.703 ms
64 bytes from oping: seq=18 time=0.693 ms
64 bytes from oping: seq=19 time=0.666 ms
Server timed out.
--- oping ping statistics ---
20 packets transmitted, 17 received, 0 out-of-order, 15% packet loss, time: 6004.243 ms
rtt min/avg/max/mdev = 0.666/0.716/0.860/0.040 ms
As you can see, packets are delivered in-order, and some packets are missing. Nothing fancy. Let’s look at a data packet:
14:06:05.607699 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xa000), length 94:
0x0000: 0045 004c 0100 0000 eb1e 73ad 0000 0000 .E.L......s.....
0x0010: 0000 0000 0000 0012 a013 0000 0000 0000 ................
0x0020: 705c e53a 0000 0000 0000 0000 0000 0000 p\.:............
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
The same 18-byte header is present. The flow endpoint ID is a different one, and the length is also different. The packet is 94 bytes, the payload length for the _ipcp-ethdix is 0x004c = 76 octets. So the FRCP header adds 12 bytes, the total overhead is 30 bytes. Maybe a bit more detail on the FRCP header contents (more depth is available the protocol documentation). The first 2 bytes are the FLAGS (0x0100). There are only 7 flags, it’s 16 bits for memory alignment. This packet only has the DATA bit set. Then follows the flow control window, which is 0 (not implemented yet). Then we have a 4 byte sequence number (eb1e 73ae = 3944641454)3 and a 4 byte ACK number, which is 0. The remaining 64 bytes are the oping payload.
Next, the data QoS:
$ oping -n oping -c 20 -i 200ms -q data
Pinging oping with 64 bytes of data (20 packets):
64 bytes from oping: seq=0 time=0.932 ms
64 bytes from oping: seq=1 time=0.701 ms
64 bytes from oping: seq=2 time=200.949 ms
64 bytes from oping: seq=3 time=0.817 ms
64 bytes from oping: seq=4 time=0.753 ms
64 bytes from oping: seq=5 time=0.730 ms
64 bytes from oping: seq=6 time=0.726 ms
64 bytes from oping: seq=7 time=0.887 ms
64 bytes from oping: seq=8 time=0.878 ms
64 bytes from oping: seq=9 time=0.883 ms
64 bytes from oping: seq=10 time=0.865 ms
64 bytes from oping: seq=11 time=401.192 ms
64 bytes from oping: seq=12 time=201.047 ms
64 bytes from oping: seq=13 time=0.872 ms
64 bytes from oping: seq=14 time=0.966 ms
64 bytes from oping: seq=15 time=0.856 ms
64 bytes from oping: seq=16 time=0.849 ms
64 bytes from oping: seq=17 time=0.843 ms
64 bytes from oping: seq=18 time=0.797 ms
64 bytes from oping: seq=19 time=0.728 ms
--- oping ping statistics ---
20 packets transmitted, 20 received, 0 out-of-order, 0% packet loss, time: 4004.491 ms
rtt min/avg/max/mdev = 0.701/40.864/401.192/104.723 ms
With the data spec, we have no packet loss, but some packets have been retransmitted (hence the higher latency). The reason for the very high latency is that the current implementation only ACKs on data packets, this will be fixed soon.
Looking at an Ethernet frame, it’s again 94 bytes:
14:35:42.612066 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xa000), length 94:
0x0000: 0044 004c 0700 0000 81b8 0259 e2f3 eb59 .D.L.......Y...Y
0x0010: 0000 0000 0000 0012 911a 0000 0000 0000 ................
0x0020: 86b3 273b 0000 0000 0000 0000 0000 0000 ..';............
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
The main difference is that it has 2 flags set (DATA + ACK), and it thus contains both a sequence number (81b8 0259) and an acknowledgement (e2f3 eb59).
That’s about it for now. More to come soon.
Dimitri
- Don’t you love standards? One of the key design objectives for Ouroboros is exactly to avoid such shenanigans. Modify/abuse a header and Ouroboros should reject it because it cannot work, not because some standard says one shouldn’t do it. [return]
- Lesser known fact: Gigabit Ethernet has a 512 byte minimum frame size; but carrier extension handles this transparently. [return]
- In network byte order. [return]