Ouroboros Tutorial 10
Tutorial 08 showed how FRCT repairs a lossy flow, end to end, in the two processes. Tutorial 09 showed how the IPCPs keep the flows of a Layer from overrunning a link, per path. This tutorial puts the two on one link at once: a 1 GiB transfer over a 100 Mbit/s link with 10 ms of delay, which drops 1 frame in 100. The Ouroboros Metrics page shows this transfer in Grafana. Here we run it, and look at what each side sees: the loss is the business of FRCT, the queue is the business of the IPCP, and neither reads the other's signal.
We use the debug build of the previous tutorials with a larger FRCT window, explained below, and oftp, the Ouroboros file transfer tool, which sends a byte stream over a stream QoS flow.
Loss and congestion
TCP uses a lost packet for two things at once: it retransmits the packet, and it takes the loss as a sign of congestion and cuts its rate. On a link that drops packets for other reasons, a radio link or a lossy cable, it cuts its rate for nothing. The two jobs are welded together because, in the Internet, the only place with state about a connection is the connection itself.
In Ouroboros, the two jobs live in different places, and each has its own signal. FRCT, in the processes at the ends of a flow, numbers the packets and repairs the holes that the receiver reports in its selective acknowledgments. It never reads the ECN byte. The congestion controller, in the IPCP, paces the flows of a path at a rate it steers from the depth of the queues along the path, which the IPCPs write into the ECN byte of every packet. It never counts a lost packet.
The setup
The window
FRCT keeps a window of packets in flight: the receiver accepts up to the size of its reordering queue beyond the packet it is waiting for, 128 packets in the default build, as we saw in Tutorial 08. On a link with a round-trip time of 30 ms, 128 packets of 1400 bytes per round trip is 46 Mbit/s, less than half of our link. That is flow control: the third thing, after the rate of the path and the repair of losses, that can limit a transfer. For this tutorial we build with a reordering queue of 1024 packets. The queue has to fit in the packet pools, so these grow with it:
$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DFRCT_REORDER_QUEUE_SIZE=1024 -DFRCT_START_WINDOW=1024 \
> -DSSM_PUP_256_BLOCKS=2048 -DSSM_PUP_512_BLOCKS=2048 -DSSM_PUP_1K_BLOCKS=2048 -DSSM_PUP_2K_BLOCKS=2048 \
> -DSSM_GSPP_256_BLOCKS=2048 -DSSM_GSPP_512_BLOCKS=2048 -DSSM_GSPP_1K_BLOCKS=2048 -DSSM_GSPP_2K_BLOCKS=2048In the container of the tutorials page, pass these options with --build-arg CMAKE_ARGS="..." when building the image. At the end of the tutorial, we come back to what the default window does to this transfer.
The link
In the previous tutorials, both IPCPs were attached to lo. That will not do here. The IPCP reads the depth of the queue in front of its device from the kernel, and on lo, a netem delay would sit in that same queue: every frame in the 10 ms delay line would count as queued, and the controller would cut the rate to get rid of a queue that is not there. So the delay and the loss go on a device the IPCPs do not own. We give each IPCP a veth pair, and join the other ends of the pairs with a bridge. The shaper goes on the IPCP's end of its pair, where the IPCP measures it, and netem on the bridge's end, out of its sight. This is also how the integration test suite shapes its links.
$ sudo ip link add veth0 type veth peer name veth0b
$ sudo ip link add veth1 type veth peer name veth1b
$ sudo ip link add br0 type bridge
$ sudo ip link set veth0b master br0
$ sudo ip link set veth1b master br0
$ for d in veth0 veth0b veth1 veth1b br0; do sudo ip link set $d up; done
The configuration file is the one of Tutorial 04, with uni-1 on veth1, uni-2 on veth0, and the name oftp for the transfer. Save it as /etc/ouroboros/tut10.conf:
### Ouroboros Tutorial 10
[name.oftp]
prog=["/usr/bin/oftp"]
[unicast.uni-1]
bootstrap="unicast-layer"
autobind=true
eth=["veth1"]
reg=["oftp"]
[unicast.uni-2]
enrol={dst="unicast-layer", eth={}}
autobind=true
eth=["veth0"]
conn=[{dst="uni-1", eth={}}]
Create the client security configuration for the three names, and start the IRMd:
$ for n in unicast-layer uni-1 oftp; do
> sudo mkdir -p /etc/ouroboros/security/client/$n
> printf 'auth=optional\nencryption=none\n' | sudo tee /etc/ouroboros/security/client/$n/sec.conf > /dev/null
> done
$ sudo irmd --stdout --config /etc/ouroboros/tut10.confThe Layer comes up as in Tutorial 04, with the name query crossing the bridge. The PoAs now have the MAC addresses of the veth devices:
$ irm ipcp poa list name uni-1
$ irm ipcp poa list name uni-2
eth veth1 0xA000 ee:ae:77:e7:fe:d5
eth veth0 0xA000 0e:9c:ab:dd:df:3eNow shape the link. The shaper on the IPCP's device is the HTB of Tutorial 09, on both devices, so that both directions are limited to 100 Mbit/s.[1] The delay goes on the bridge's devices, 10 ms in each direction, with a queue deep enough to never drop a frame on its own:
$ for d in veth0 veth1; do
> sudo tc qdisc add dev $d root handle 1: htb default 1 r2q 625
> sudo tc class add dev $d parent 1: classid 1:1 htb rate 100mbit
> done
$ for d in veth0b veth1b; do
> sudo tc qdisc add dev $d root netem limit 65536 delay 10ms
> doneA clean transfer
The oftp server writes what it receives to its standard output, or to a file with -o. We do not need the data, only to know that it arrived intact, and oftp computes a checksum of the stream at both ends. Start the server as root, and discard its output:
$ sudo oftp --listen > /dev/null
oftp: listening...The client reads its standard input, or a file with -i, and -N stops it after a number of bytes. Send 1 GiB of random data as a regular user:
$ oftp -n oftp -N 1G < /dev/urandom
oftp: sent 1073741824 bytes in 93.135 s (10.99 MiB/s) crc64=f54f88c5ec0698f4The server reports the same number of bytes and the same checksum, and exits; start it again before every transfer:
oftp: received 1073741824 bytes in 93.144 s (10.99 MiB/s) crc64=f54f88c5ec0698f4The transfer takes 93 seconds, 92.2 Mbit/s of payload, about 96 Mbit/s on the wire with the headers counted. The rest is the headroom of the congestion controller: it cuts the rate as soon as the queue in front of the link grows, and climbs back slowly, so the link idles for a moment after every cut. The shaper on veth0 held back a frame every time the link was busy (overlimits) and dropped none:
$ tc -s qdisc show dev veth0
qdisc htb 1: root refcnt 33 r2q 625 default 0x1 direct_packets_stat 0 direct_qlen 1000
Sent 1120565050 bytes 790571 pkt (dropped 0, overlimits 312285 requeues 0)
backlog 0b 0p requeues 0While the transfer runs, the flow allocator of uni-2 shows the controller of the path, as in Tutorial 09:
$ sudo grep -E 'Paced|capacity|regime|Upstream|Signal' /tmp/ouroboros/uni-2/flow-allocator/*
Upstream congestion level: 0
Paced rate (bytes/s): 11871796
Current congestion regime: Additive inc
Path capacity (bytes/s): 11862016
Signal-loss cuts (count): 0The paced rate sits at the capacity of the link, and the controller alternates between increasing it and cutting it back, keeping a few packets in the queue of the shaper. The statistics of FRCT, on the client's side of the flow, show the other loop. Read them a minute into the transfer:
$ grep -E 'rtt|RTO|RTT|window edge|RXM \(|SACKs rec|ACK packets rec|Tx ring' /tmp/ouroboros/proc.$(pgrep -u $USER -x oftp)/*/frct
Smoothed rtt (ns): 35584591
Retransmit timeout RTO (ns): 71169182
Sender left window edge: 2161035556
Sender right window edge: 2161036580
RXM (RTO-driven) sent: 0
RXM (SACK mechanism) sent: 0
RXM (RACK-driven) sent: 0
RXM (zero reorder wnd) sent: 0
ACK packets received: 5705
OOO arrivals: 0
SACKs received: 0
D-SACKs received: 0
RTT estimator samples: 5705
Tx ring drops (any cause): 6The round-trip time is the 20 ms of the link, plus the time a packet waits in the queue of the shaper, plus the time the receiver holds an acknowledgment before sending it, up to 10 ms, as we saw in Tutorial 08; the retransmission timeout is twice that. The window edges are 1024 apart, the queue we built with. The sender received about a hundred acknowledgments per second, and not a single selective acknowledgment or retransmission: with no loss on the link, FRCT has nothing to repair.
Adding loss
Now make the bridge drop 1 frame in 100, in both directions:
$ for d in veth0b veth1b; do
> sudo tc qdisc change dev $d root netem limit 65536 delay 10ms loss 1%
> doneStart the server again, and send another 1 GiB:
$ oftp -n oftp -N 1G < /dev/urandom
oftp: sent 1073741824 bytes in 93.436 s (10.96 MiB/s) crc64=c3cbef35f0110c4boftp: received 1073741824 bytes in 93.458 s (10.96 MiB/s) crc64=c3cbef35f0110c4bThe transfer takes 93 seconds, 0.3 seconds longer than without loss: 91.9 Mbit/s, with the same checksum at both ends. netem dropped 7,885 of the frames on their way to uni-1, and 2,617 acknowledgments on their way back:
$ tc -s qdisc show dev veth1b
qdisc netem 8012: root refcnt 33 limit 65536 delay 10ms loss 1% seed 5611493419457351079
Sent 2241136160 bytes 1581185 pkt (dropped 7885, overlimits 0 requeues 0)
backlog 0b 0p requeues 0Every dropped frame left a hole at the receiver, and FRCT repaired every one of them. The statistics of the client, a minute into the transfer, count the repairs:
$ grep -E 'rtt|RTO|RTT|window edge|RXM \(|SACKs rec|ACK packets rec|Tx ring' /tmp/ouroboros/proc.$(pgrep -u $USER -x oftp)/*/frct
Smoothed rtt (ns): 30778686
Retransmit timeout RTO (ns): 61557372
Sender left window edge: 819241679
Sender right window edge: 819242703
RXM (RTO-driven) sent: 1
RXM (SACK mechanism) sent: 5047
RXM (RACK-driven) sent: 1
RXM (zero reorder wnd) sent: 5046
ACK packets received: 160118
OOO arrivals: 0
SACKs received: 157247
D-SACKs received: 0
RTT estimator samples: 157607
Tx ring drops (any cause): 2In the first minute, the client retransmitted 5,048 packets, about 1% of what it sent. All but two of them are repairs of holes that a selective acknowledgment reported, with the reordering window collapsed to zero: on a link that never reorders, FRCT repairs a hole as soon as a packet behind it is acknowledged, as in the delay phase of Tutorial 08. The receiver now answers nearly every packet with a selective acknowledgment, as there is nearly always a hole waiting for its repair. The retransmission timer fired once. The round-trip time came down a little, as the selective acknowledgments are sent at once and the plain ones are held for up to 10 ms, and the timeout followed it.
And the IPCP? Its controller saw nothing happen:
$ sudo grep -E 'Paced|capacity|regime|Upstream|Signal' /tmp/ouroboros/uni-2/flow-allocator/*
Upstream congestion level: 0
Paced rate (bytes/s): 11557717
Current congestion regime: Proportional dec
Path capacity (bytes/s): 11862016
Signal-loss cuts (count): 0The paced rate is still that of the link, the regime still alternates between increase and decrease, and the signal-loss cuts stay at zero: the controller only cuts the rate when its feedback stops arriving, and 1% of the flow updates going missing is far from that. A dropped frame costs the flow one retransmission, and nothing else. The link stays full, because the signal that steers the rate, the queue in front of the shaper, is as full as it was without loss.
The same transfer with TCP
For comparison, let's run a TCP transfer over a link with the same shaping. TCP does not use the raw sockets of the Ethernet PoA, so a simpler setup works: a veth pair with one end in a network namespace, and the rate, delay and loss chained on each end. Linux will not route traffic between two local addresses over a veth pair, which is what the namespace is for:
$ sudo ip netns add tcp
$ sudo ip link add vt0 type veth peer name vt1
$ sudo ip link set vt1 netns tcp
$ sudo ip addr add 10.0.0.1/24 dev vt0
$ sudo ip link set vt0 up
$ sudo ip netns exec tcp ip addr add 10.0.0.2/24 dev vt1
$ sudo ip netns exec tcp ip link set vt1 up
$ sudo ip netns exec tcp ip link set lo up
$ sudo tc qdisc add dev vt0 root handle 1: htb default 1 r2q 625
$ sudo tc class add dev vt0 parent 1: classid 1:1 htb rate 100mbit
$ sudo tc qdisc add dev vt0 parent 1:1 handle 10: netem limit 65536 delay 10ms loss 1%
$ sudo ip netns exec tcp tc qdisc add dev vt1 root handle 1: htb default 1 r2q 625
$ sudo ip netns exec tcp tc class add dev vt1 parent 1: classid 1:1 htb rate 100mbit
$ sudo ip netns exec tcp tc qdisc add dev vt1 parent 1:1 handle 10: netem limit 65536 delay 10ms loss 1%Start an iperf3 server in the namespace. Before adding the loss, run a client for a minute with CUBIC, the default congestion control of Linux, to see what TCP makes of this link on its own:
$ sudo ip netns exec tcp iperf3 -s -D
$ iperf3 -c 10.0.0.2 -t 60 -C cubic
[ 5] 0.00-60.00 sec 686 MBytes 95.9 Mbits/sec 0 sender
[ 5] 0.00-60.29 sec 686 MBytes 95.5 Mbits/sec receiverCUBIC fills the link as well as our transfer did: 95.5 Mbit/s. Now add the loss to both ends:
$ sudo tc qdisc change dev vt0 parent 1:1 handle 10: netem limit 65536 delay 10ms loss 1%
$ sudo ip netns exec tcp tc qdisc change dev vt1 parent 1:1 handle 10: netem limit 65536 delay 10ms loss 1%and run a client with CUBIC, and then with Reno:
$ iperf3 -c 10.0.0.2 -t 60 -C cubic
[ 5] 0.00-60.00 sec 68.5 MBytes 9.58 Mbits/sec 509 sender
[ 5] 0.00-60.02 sec 67.0 MBytes 9.36 Mbits/sec receiver
$ iperf3 -c 10.0.0.2 -t 60 -C reno
[ 5] 0.00-60.00 sec 71.9 MBytes 10.0 Mbits/sec 507 sender
[ 5] 0.00-60.02 sec 69.6 MBytes 9.73 Mbits/sec receiverCUBIC gets 9.4 Mbit/s out of the link and Reno 9.7, with 1% of the segments retransmitted, the same loss our transfer saw. The numbers are no surprise. The Mathis formula gives the throughput of a loss-based congestion controller as about 1.22 times the segment size, divided by the round-trip time and by the square root of the loss rate; with segments of 1448 bytes, a round-trip time of 20 ms and a loss rate of 1%, that is 7 Mbit/s. Every lost segment is a sign of congestion to TCP, and at 1% loss it spends its time recovering from congestion that does not exist.
The congestion controls that came after Reno and CUBIC took two different ways out. DCTCP, the TCP of the data centre, reads the ECN mark of every packet and cuts its window in proportion to the fraction of packets that were marked. The reaction to a lost packet, however, is still the one of Reno, and our link marks nothing:
$ iperf3 -c 10.0.0.2 -t 60 -C dctcp
[ 5] 0.00-60.00 sec 76.9 MBytes 10.7 Mbits/sec 578 sender
[ 5] 0.00-60.05 sec 73.6 MBytes 10.3 Mbits/sec receiverA better congestion signal does not help as long as loss is still read as one.[2] BBR took the other way: it does not read loss as congestion at all. It estimates the bandwidth and the base round-trip time of the path from the acknowledgments, paces at that bandwidth, and retransmits what is lost without slowing down. The kernel of this machine has the first version of BBR:
$ iperf3 -c 10.0.0.2 -t 60 -C bbr
[ 5] 0.00-60.00 sec 673 MBytes 94.0 Mbits/sec 4909 sender
[ 5] 0.00-60.02 sec 670 MBytes 93.7 Mbits/sec receiverWith 1% of its segments retransmitted, BBR gets 93.7 Mbit/s: the link, like our transfer.[3] Its per-second output shows the price of estimating the path from the sender. About every ten seconds, BBR holds four packets in flight for 200 ms to see the round-trip time of the path without its own queue in it: a window of 5.66 KB, four segments, and a second at 81 Mbit/s:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 9.00-10.00 sec 11.1 MBytes 93.3 Mbits/sec 42 537 KBytes
[ 5] 10.00-11.00 sec 9.62 MBytes 80.7 Mbits/sec 72 5.66 KBytes
[ 5] 11.00-12.00 sec 9.75 MBytes 81.8 Mbits/sec 70 535 KBytes
[ 5] 12.00-13.00 sec 12.6 MBytes 106 Mbits/sec 78 537 KBytes
ss -ti on the iperf3 socket. For Ouroboros the solid bar is the round trip the controller in uni-2 measures with its heartbeats, the path and its queue, and the dashed bar extends it to what FRCT in the oftp client measures, with the tx ring and the held acknowledgments in it. The transports that read loss as congestion lose the link to their congestion control. The ones that do not, Ouroboros and BBR, lose almost nothing to the loss; what separates them from the link rate is the headroom their controllers keep, and the round-trip time shows what that headroom buys: a path kept clear of queue.The comparison is about the shape, not the numbers: iperf3 and oftp are different tools. That our transfer sits a few Mbit/s under CUBIC without loss is tuning. CUBIC takes the link to 100 Mbit/s on the wire by filling the queue in front of it until something gives, and on this link nothing gives but the receiver's window: netem holds 65,536 frames, and the round-trip time of the transfer averages 511 ms, the 20 ms of the path and 491 ms of queue, some 6 MB waiting in front of a 100 Mbit/s link. The controller in uni-2 keeps that queue short and cuts the rate at the first sign of it growing, and the link idles for a moment after every cut; the round trip of our transfer averages 41 ms, and little of that is queue, as we will see. Its constants could let it sit closer to the link, at the price of a slightly longer queue; that trade-off is the controller's to make, and not where the two architectures differ. With loss, the queue is empty: CUBIC, Reno and DCTCP never get near the link, and their round trip is the 21 ms of the path. BBR, which fills the link, keeps its probing in the queue, at 25 ms.
The two round-trip times are not the same measurement, as the two figures show. Of our 41 ms, 10 are the queue the tx ring may hold in front of the IPCP, and 10 the time the receiver holds an acknowledgment; with loss the selective acknowledgments go out at once, and the 30 ms are the path, the ring, and a fraction of a millisecond of queue. TCP's clock starts at the queueing discipline, so the 21 ms of CUBIC with loss are the bare path, and the 511 ms without loss are the path and the queue it fills. The controller in uni-2 has a clock of its own: about once a round trip it sends a heartbeat to uni-1, straight from the IPCP into the shaper, and uni-1 answers at once; the round trip is folded into the ramp time constant of mb-ecn, which the RIB shows at twice its value. It read 20.2 ms without loss and 20.2 ms with: the path, and a few hundred microseconds of queue in front of the link. That is the division of labour between the two loops. The controller keeps the latency of the path low: a few hundred microseconds of queue, against the 491 ms that CUBIC piles up in front of the same link. FRCT's round-trip time is not a measure of the path but the clock for its retransmissions, and the ring and the hold in it cost the path nothing: the transfer carries them at 92.2 Mbit/s. Both of our 10 ms are settings, not properties of the protocol: the acknowledgment is held for two ticks of FRCT, FRCT_TICK_TIME at build time, and the queueing delay of the tx ring is SSM_RBUFF_TXQ_DELAY at build time, or per flow with fccntl(3) and FLOWSTXQDLY.
Where they differ is in the mechanism. CUBIC and Reno have one signal, loss, and read every lost packet as a queue that overflowed; on this link they slow down at every loss for a queue that never was. DCTCP has a finer signal for the queue, and the same reading of loss. BBR stopped reading loss and estimates from the acknowledgments what the path can carry. The IPCP does not have to estimate it: it stands at the queue, reads it, and shares the rate it finds between every flow of the path, while FRCT repairs the loss at the ends, and neither loop sees the other's signal. Every transport here repairs the same 1% of losses, and only the ones that also take those losses as an order to slow down pay for it.
BBR gets the transfer through this link as well as we do, so what does reading the queue buy over estimating it? BBR has to stop to measure, as the dip above shows, and its estimate depends on where it is made. A BBR flow keeps twice its bandwidth-delay product in flight, so on a shared bottleneck the flow with the longer round trip takes the larger share, the reverse of Reno, and together the flows put a round trip's worth of data into the queue, more than a shallow buffer holds. Fairness between BBR flows is not something the algorithm aims for but a side effect of the probing; it is slow, and it only holds for flows with similar round-trip times. The rate law of mb-ecn has no round-trip time in it: the controllers of every path through a bottleneck read the same queue depth and respond to it in the same way, whatever their distance from it. The integration suite checks that with two sources about ten times apart in round-trip time, which must share a bottleneck within 25% of each other.
The window again
If you run this tutorial with the default build, the same transfers come out differently. Without loss, the transfer takes 185 seconds, 46.5 Mbit/s: the client sends its window of 128 packets in 15 ms, and then waits for the rest of the 31 ms round trip for the acknowledgments, while the link sits idle. With loss, it takes 292 seconds, 29.4 Mbit/s: a hole stops the window until it is repaired, a round trip later, so every lost frame now costs a round trip, on top of the retransmission.
That is still a limit of the two ends, not of the path. The IPCP keeps raising the paced rate above what the flow uses, and never sees a queue: the controller is not what holds the flow back. Three things can limit a transfer, and they are kept apart: the window, which is the receiver's buffer; the rate, which is the path's capacity; and the repair of losses, which only costs the retransmissions. TCP keeps all three in one loop, and that loop is what we measured above.
Watching it live
With the metrics exporter running during the two transfers, the FRCT rows of the Grafana dashboard show the loop of the two ends, and the IPCP rows the loop of the path. These panels come from a run of the two transfers with the exporter sampling every half second:
When you are done, remove the link and the namespace:
$ sudo ip netns del tcp
$ sudo ip link del vt0
$ sudo ip link del veth0
$ sudo ip link del veth1
$ sudo ip link del br0Notes
- ↑ The r2q value only silences a warning of tc about the quantum of the class, which does not matter with a single class.
- ↑ That is also where L4S stands, the IETF's low-latency architecture (RFC 9330 to 9332). It builds on DCTCP: the network marks at a queue of about a millisecond, and TCP Prague, its congestion control, cuts in proportion to the fraction of marked packets. Loss stays a congestion signal by requirement, Prague must react to it like Reno, so it would share the fate of CUBIC and Reno on this link. One bit per packet carries the mark and the level is the fraction over a round trip; Prague behaves as if the round trip were at least 25 ms, to temper the round-trip bias of a controller of that kind. L4S flows get a queue of their own, coupled to the classic one, to live next to CUBIC. That queue, DualPI2, is in Linux since 6.17; TCP Prague waits for Accurate ECN, partly merged in 6.18, and is not in this kernel.
- ↑ The later versions of BBR brought loss back as a signal, in a bounded way: a loss rate above about 2% in a round trip is read as congestion, and the ceiling on the data in flight is cut by 30%. At the 1% of this link, they still ignore it.