Congestion avoidance: Difference between revisions
No edit summary |
|||
| (6 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast | mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast | ||
IPCP. | IPCP. Reliability, ordering and flow control are FRCP's job, end-to-end | ||
and per flow; congestion avoidance is a separate concern, and it runs | |||
end-to-end; congestion avoidance is | in the IPCP, once per layer. Within a layer the IPCP controls | ||
layer. Within a layer | '''aggregates''': one control loop per (destination address, QoS cube), | ||
per (destination address, QoS cube), shared by every flow | shared by every flow to that destination. | ||
destination | |||
This document | This document is about the algorithm: what it does and why. The | ||
code follows it; the [[#References|reference section]] maps the pieces | |||
to source files. It doesn't oversell: the [[#Limitations|limitations]] | |||
section says plainly what it can't do. | |||
"MUST", "SHOULD", "MAY" and friends mean what BCP 14 (RFC 2119, | |||
RFC 8174) says, but only where they appear in capitals. | |||
Note: Co-authered by Anthropic Claude Fable 5. | |||
== Notation == | == Notation == | ||
{| class="wikitable" | |||
! Symbol !! Units !! Meaning | |||
|- | |||
| <code>r</code> || B/s || Paced send rate | |||
|- | |||
| <code>C</code> || B/s || Bottleneck capacity | |||
|- | |||
| <code>S</code> || bytes || Packet size | |||
|- | |||
| <code>q</code> || packets || Standing queue occupancy at the bottleneck | |||
|- | |||
| <code>Q</code> || packets || Marking quantum: packets per mark unit (4) | |||
|- | |||
| <code>ecn</code> || 8-bit code || Per-packet congestion mark set by a forwarder | |||
|- | |||
| <code>ece</code> || 1/32 <code>ecn</code> || Receiver-side congestion estimate: 32 × time-mean <code>ecn</code> | |||
|- | |||
| <code>cap8</code> || 8-bit code || Path capacity, quarter-log2: <code>C ≈ 2^(cap8 / 4)</code> B/s, 0 = unknown | |||
|- | |||
| <code>W</code> || ns || Receiver averaging window | |||
|- | |||
| <code>a</code> || B/s^2 || Additive-increase slope | |||
|- | |||
| <code>M</code> || <code>ece</code> units || Full-congestion reference level (512) | |||
|- | |||
| <code>Δt</code> || ns || Elapsed wall-clock time between two events | |||
|- | |||
| <code>τ</code> || n/a || Path round-trip time (feedback delay); the controller does not measure it | |||
|} | |||
Rate is the control variable throughout. | Rate is the control variable throughout. | ||
Every concrete figure in this document (window bounds, horizons, | |||
rate thresholds | rate thresholds) is just an instance of the parameter values in the | ||
[[# | [[#Parameters|parameters table]]. What the algorithm fixes is the | ||
between parameters | mechanisms and the relations between parameters; retune the parameters | ||
and the figures move together. | |||
__TOC__ | __TOC__ | ||
== | == Overview == | ||
mb-ecn is a rate-based, ECN-driven controller with three moving parts, | mb-ecn is a rate-based, ECN-driven controller with three moving parts, | ||
| Line 85: | Line 82: | ||
</pre> | </pre> | ||
# The '''forwarder''' marks each packet with a ''multi-bit'' magnitude | # The '''forwarder''' marks each packet with a ''multi-bit'' magnitude proportional to its standing queue, and MIN-stamps its measured outgoing-link capacity into the same PCI. | ||
# The '''receiver''' turns the stream of marks into a time-averaged congestion estimate <code>ece</code> and feeds it back to the sender, together with the window minimum of the path-capacity codes. | |||
# The '''sender''' paces its rate with a start-time fair-queuing (SFQ) virtual clock, and adjusts that rate with additive-increase / multiplicative-decrease (AIMD) driven by <code>ece</code>; the rate floor and the additive slope derive from the fed-back path capacity. | |||
# The '''receiver''' turns the stream of marks into a time-averaged | |||
# The '''sender''' paces its rate with a start-time fair-queuing (SFQ) | |||
There's no timer thread: the controller runs when a packet goes out or | |||
feedback comes back. Every step is scaled by elapsed wall-clock time, | |||
wall-clock time, | and that is what makes the ''allocation'' independent of RTT (see | ||
RTT (see [[# | [[#RTT behaviour|RTT behaviour]]). | ||
The unit of control is the '''aggregate''' | The unit of control is the '''aggregate''': one per | ||
<code>(destination address, QoS cube)</code> | <code>(destination address, QoS cube)</code>, shared by every flow to | ||
destination at that QoS | that destination at that QoS. They share one controller and one rate. | ||
flow joins the aggregate | A new flow joins at whatever rate the aggregate is already running; when | ||
leaves | a flow leaves, its share goes to the others. The pacer (SFQ) splits | ||
fairly across the member flows | that rate fairly across the member flows, so per-flow fairness is the | ||
scheduler's doing, not the rate law's. | |||
== | == Forwarder marking == | ||
A forwarder marks a packet | A forwarder marks a packet based on how many packets are already queued | ||
on the outgoing link: | on the outgoing link: | ||
| Line 119: | Line 108: | ||
ecn = MAX(ecn, mark) (saturating, 8-bit field) | ecn = MAX(ecn, mark) (saturating, 8-bit field) | ||
<code>Q</code> is fixed for the whole layer, not tuned per hop: the marks | |||
get MAX-combined across hops, so every forwarder MUST use the same | |||
quantum or the numbers don't compare. | |||
Two | Two things worth noting: | ||
* ''' | * '''It's a magnitude, not a bit.''' The mark is an integer that tracks queue depth, in steps of <code>Q</code> packets: a coarse read of the standing queue. | ||
* '''MAX along the path.''' Cross several forwarders and the packet carries the largest mark, i.e. the deepest queue it passed. Taking the MAX (not a sum) keeps the number bounded and monotone across hops. | |||
* '''MAX along the path.''' | |||
There | There's a '''dead zone''': below <code>Q</code> queued packets the mark is 0. | ||
=== Capacity stamping === | === Capacity stamping === | ||
Each forwarder also measures | Each forwarder also measures its outgoing link's capacity and stamps it | ||
stamps it into the packet. The | into the packet. The measurement is a busy-period drain rate: look at | ||
the queue | the queue at most once a millisecond, and keep a window open until 16 | ||
window | packets have drained. So the window '''sizes itself to the link''': | ||
''' | the once-a-millisecond cadence caps it on fast links, the 16-packet | ||
drain time on slow ones (~19 ms at 10 Mbit/s). If more than 1/8 of the | |||
(~19 ms at 10 Mbit/s). | arrivals hit an empty queue, throw the window out: the link | ||
the | wasn't saturated. The odd empty sample (a token-bucket shaper grazing | ||
zero) is fine. A saturated queue drains at the link rate, so a max | |||
filter that decays slowly (1/16 per window) creeps up to it from below. | |||
filter | A window that started or ended on an empty queue might have drained | ||
window that | into buffers downstream faster than the wire, so it may pull the | ||
buffers | estimate down but never up. On the wire the code is quarter-log2 | ||
never | (<code>cap8 = 4 log2 C</code>, ~19% a step, 0 = unknown), and each hop | ||
(<code>cap8 = 4 log2 C</code>, ~19% | MIN-combines its own into the byte, so the packet arrives carrying the | ||
MIN-combines its own | slowest hop's rate. A hop that has never backed up stamps nothing, | ||
which is fine, since only backed-up hops matter. | |||
== | == Receiver estimate == | ||
The receiver | The receiver turns the marks into a smoothed estimate <code>ece</code>, | ||
averaged over a window <code>W</code> that '''tracks the incoming byte rate'''. | |||
a | It's a plain time-average over that window (a box-car, not an EWMA). | ||
<code>Δt</code> the gap since the | Per packet, with <code>Δt</code> the gap since the last one: | ||
A += ecn * MIN(Δt, W) dwell-weighted; one packet | A += ecn * MIN(Δt, W) dwell-weighted; one packet | ||
| Line 176: | Line 159: | ||
A = 0 ; B = 0 hard reset | A = 0 ; B = 0 hard reset | ||
When a window closes, the next one is nudged (a quarter-weight | |||
average toward the size that would hold about 16 packets at the | moving average) toward the size that would hold about 16 packets | ||
at the current byte rate, starting from ~67 ms. The target is in bytes | |||
"16 packets" is exact | (<code>B_target</code> = 16,000), so "16 packets" is only exact at | ||
~1000-byte packets, and it takes a few closes to settle. The mean | |||
''' | always divides by the window that '''actually''' elapsed, so a change | ||
estimate | in rate leaves the current estimate correct and only resizes the | ||
''next'' one. Net effect: a roughly constant ~16 samples a window from | |||
about 30 kbit/s to 122 Mbit/s. Above that the window bottoms out at | |||
thousands of samples at 10–100 GbE) | ~1 ms (still thousands of samples a window at 10–100 GbE); below | ||
~30 kbit/s it hits the ~4.3 s ceiling. The averaging clock stretches | |||
the flow the way TCP's ACK clock stretches with the RTT. A speed-up | with the flow, the way TCP's ACK clock stretches with the RTT. A | ||
sudden speed-up doesn't wait out a stretched window: once twice the | |||
target bytes have arrived it closes anyway (the ~1 ms floor keeps that | |||
at high rate). | from running away at high rate). So the <code>ece</code> the sender sees | ||
staircase | is a staircase: one step per window, steps getting shorter as | ||
the rate climbs. Two edge cases: | |||
* '''Onset''' | * '''Onset.''' The first mark after an idle stretch is passed through as-is, so a queue that's just starting to build shows up without waiting out a whole window. | ||
* '''Gap.''' A silence longer than 4 windows resets and sends the raw sample. "Long" is '''measured in the flow's own windows''', so a slow flow's normal spacing never reads as idle; and the dwell clamp above limits what a real pause can add to the mean before the reset fires. | |||
* '''Gap''' longer than 4 windows resets | |||
Alongside the | Alongside the marks, the receiver keeps the smallest non-zero capacity | ||
code it saw this window and ships it with each <code>ece</code> (on an | |||
onset/gap restart, just the current packet's code). Then it resets, so | |||
if the path reroutes onto something faster the fed-back capacity can | |||
fed-back capacity within one window. | climb within one window. | ||
The estimate | The estimate rides back to the sender on the reverse flow. | ||
== | == Sender pacer == | ||
The sender paces | The sender paces off an SFQ virtual clock. The aggregate keeps a | ||
virtual time <code>V</code>; each flow keeps a finish tag <code>F</code>: | virtual time <code>V</code>; each flow keeps a finish tag <code>F</code>: | ||
| Line 221: | Line 199: | ||
wait = (s - V) / r time until this packet may go | wait = (s - V) / r time until this packet may go | ||
That <code>wait</code> is the flow's scheduling deadline. A packet | |||
that's behind the virtual clock goes out now (<code>wait = 0</code>); | |||
(<code>wait = 0</code>); one ahead waits, | one that's ahead waits, and only that flow waits, nobody else. | ||
Across an idle gap longer than 50 ms the clock credits at most the | |||
flow's own owed lead plus one burst (50 ms of service, at least one | |||
least one packet) | packet). So a flow paced slower than a packet per 50 ms still gets its | ||
'''real''' elapsed service (capping the credit by time would | |||
starve it) while an idle flow still can't bank an unbounded | |||
burst. | |||
<code>V</code> is shared by every flow in the aggregate; the finish | |||
the finish tags are | tags are not. Since every flow measures its start tag <code>s</code> | ||
tag <code>s</code> against the same <code>V</code>, a flow that | against the same <code>V</code>, a flow that just sent is now ahead of | ||
<code>V</code> and has to wait, while an idle one (tag at or behind | |||
( | <code>V</code>) goes right away. That is what splits the aggregate rate | ||
what | <code>r</code> fairly among them. | ||
== | == Rate control (AIMD + PD) == | ||
The rate law runs on packet sends, at most once | The rate law runs on packet sends, at most once a millisecond, and | ||
also | also when feedback arrives, so a sender with no traffic of its | ||
still reacts. | own still reacts. The increase terms and the proportional decrease are | ||
scaled by elapsed | each scaled by elapsed time, but not by the same clock: the increases | ||
use banked time '''capped''' at 50 ms (an idle gap shouldn't buy an | |||
unbounded ramp), while the decrease uses the '''uncapped''' elapsed | |||
the '''uncapped''' elapsed time (a starved sender still cuts by the | time (a starved sender still cuts by the right amount). The one-sided | ||
right amount). The one-sided derivative ( | derivative (see the | ||
[[#Multiplicative decrease (proportional + derivative)|decrease]]) is | |||
not scaled by time at all; it is per-step. | |||
=== | === Slow start === | ||
Before the first congestion signal the rate ramps exponentially with a | Before the first congestion signal, the rate ramps up exponentially | ||
20 ms time constant: | with a 20 ms time constant: | ||
r += r * Δt / T_ss (T_ss = 20 ms) | r += r * Δt / T_ss (T_ss = 20 ms) | ||
Slow start ends the first time a congestion signal | Slow start ends the first time a congestion signal shows up, whether | ||
fed back from the receiver or | fed back from the receiver or seen at the sender's own first hop, and | ||
it never comes back. | |||
Slow start belongs to the aggregate, so it | Slow start belongs to the aggregate, so it happens | ||
(destination, QoS cube)''', on the first flow. A flow | '''once per (destination, QoS cube)''', on the first flow. A flow that | ||
shows up later, at an aggregate that's already running, rides the | |||
existing | |||
the pacer | estimate: it starts at the current rate and takes its share through | ||
the [[#Sender pacer|pacer]], without probing of its own. | |||
=== | === Increase: additive plus proportional probe (always on) === | ||
Past slow start, two increase terms | Past slow start, two increase terms run every step, congested or not: | ||
r += a * Δt additive; a defaults to 2^16 B/s^2 | r += a * Δt additive; a defaults to 2^16 B/s^2 | ||
r += r * Δt / T_probe proportional; T_probe = 8 s | r += r * Δt / T_probe proportional; T_probe = 8 s | ||
The additive term is a fixed | The additive term is a fixed slope (bytes/s per second), sized to the | ||
path capacity (see | |||
[[#Capacity-derived floor and slope|the capacity floor]]). The | |||
<code>T_probe</code>): a flow | proportional probe adds a fixed fraction of the rate per unit time (it | ||
link rate; | e-folds over <code>T_probe</code>): that is what lets a flow recover in | ||
fast link. | the same number of steps whatever the link rate; a fixed additive step | ||
pressure the multiplicative decrease balances at equilibrium | is negligible next to a fast link. Together they are the | ||
probe | upward pressure the multiplicative decrease balances at equilibrium. | ||
The probe is not free: it costs a standing queue that does not shrink | |||
with rate (see [[#Fairness|fairness]]). | |||
=== | === Multiplicative decrease (proportional + derivative) === | ||
With congestion level <code>m</code> (the fed-back <code>ece</code>, or | With congestion level <code>m</code> (the fed-back <code>ece</code>, or | ||
first-hop mark | the local first-hop mark if there's no feedback yet), the cut is a | ||
term | proportional term plus a one-sided derivative term, a PD controller on | ||
congestion signal: | the congestion signal: | ||
mark = MIN(m, M) M = 512 | mark = MIN(m, M) M = 512 | ||
| Line 302: | Line 283: | ||
m_prev = m | m_prev = m | ||
* The '''proportional''' term | * The '''proportional''' term uses the real elapsed milliseconds, so a starved sender that hasn't run the controller in a while still cuts by the right amount. | ||
* The '''derivative''' term (<code>rise</code>) only fires when the congestion level ''goes up'' between samples: one-sided, per-step. It sharpens the response right as congestion starts. | |||
* The cut is capped at <code>r/2</code>, so one step can at most halve the rate. | |||
* The '''derivative''' term (<code>rise</code>) | |||
* The cut is | |||
<code>M = 512 = 32 × 16</code> | <code>M = 512 = 32 × 16</code> pins "full congestion" at a mean | ||
<code>ecn</code> of 16, i.e. a mean queue of <code>16 × Q = 64</code> packets. | <code>ecn</code> of 16, i.e. a mean queue of <code>16 × Q = 64</code> | ||
packets. | |||
=== | === Staleness === | ||
Feedback | Feedback comes once per receiver window, and that window stretches with | ||
the flow's byte rate (see | |||
[[#Receiver estimate|the receiver estimate]]), so the staleness horizon | |||
feedbacks and the rate would ramp | has to stretch with it too, or a slow flow's mark would expire | ||
sender mirrors the receiver: the horizon is four target windows of | ''between'' feedbacks and the rate would ramp straight into a congested | ||
bytes at the current rate, floored at ~268 ms so fast flows keep a | path. The sender mirrors the receiver: the horizon is four target | ||
fixed | windows' worth of bytes at the current rate, floored at ~268 ms so fast | ||
than | flows keep a fixed one; at the default rate floor it's ~8 s. Once a | ||
local mark), | signal is older than that it's dropped (both the fed-back estimate and | ||
rate | the local mark), which frees the rate to climb again after congestion | ||
<code>[r_min, r_max]</code> | clears. Every step clamps the rate to <code>[r_min, r_max]</code>. | ||
=== | === Capacity-derived floor and slope === | ||
The fed-back capacity code sets the two rate-scale constants | The fed-back capacity code sets the controller's two rate-scale | ||
constants, per aggregate: | |||
target = decode(cap8) / 32 clamped to [2^13, 2^32] B/s | target = decode(cap8) / 32 clamped to [2^13, 2^32] B/s | ||
| Line 337: | Line 314: | ||
a = r_min | a = r_min | ||
On a path that never reports a capacity, both fall back to fixed | |||
<code>a = 2^16 B/s^2</code>) | defaults (<code>r_min = 2^13 B/s</code>, <code>a = 2^16 B/s^2</code>). | ||
When the estimator is live, the fairness floor and the post-cut | |||
slope scale with the bottleneck, and capacity | recovery slope scale with the bottleneck instead, and capacity kicks in | ||
<code>C/32</code> | wherever <code>C/32</code> beats the default floor, roughly above | ||
capacity older than 16 staleness horizons (~4.3 s for fast flows) | 2 Mbit/s. A capacity older than 16 staleness horizons (~4.3 s for fast | ||
reverts both to the defaults. That horizon deliberately | flows) reverts both to the defaults. That horizon is deliberately | ||
feedback horizon: feedback stops the | longer than the feedback horizon: feedback stops the instant the marks | ||
exactly when the recovery slope | clear, which is exactly when you need the recovery slope, and the | ||
onset-fresh capacity (see | |||
[[#Receiver estimate|the receiver estimate]]) re-seeds it on the first | |||
mark of the next episode anyway. | |||
=== | === Ramp and recovery === | ||
Three mechanisms set how fast an aggregate reaches a fast link's | Three mechanisms set how fast an aggregate reaches a fast link's | ||
rate, each owning one regime: | rate, each owning one regime: | ||
* '''From scratch: slow start.''' The exponential ramp | * '''From scratch: slow start.''' The exponential ramp hits any real link rate in well under a second (~200 ms from the seed to 10 Gbit/s; every further doubling of link speed adds one ~14 ms doubling). Nothing marks on an uncongested path, so slow start just runs until the aggregate reaches its bottleneck. | ||
* '''After a cut: the probe.''' A cut is at most a halving, and the mark expires on the rate-relative horizon once congestion clears (see [[#Staleness|staleness]]); the probe then climbs back from a halving in <code>T_probe · ln 2 ≈ 5.5 s</code>, whatever the link rate. | |||
* '''At the bottom: the capacity floor.''' The rate never sits more than 32× below a measured bottleneck, and the additive slope refills <code>C/32</code> a second, so the worst hole is bounded: floor to full capacity in tens of seconds, a halving in seconds. | |||
* '''After a cut: the probe.''' A | |||
* '''At the bottom: the capacity floor.''' The rate sits | |||
These | These fit together because a deep cut needs sustained congestion, | ||
sustained congestion | sustained congestion backs up the bottleneck queue, and a backed-up | ||
queue is exactly when its capacity | queue is exactly when its capacity gets measured and stamped (see | ||
[[#Forwarder marking|forwarder marking]]). So whenever the controller | |||
live. The default floor only | has been cut deep, the scaled floor is already live. The default floor | ||
only ever applies to paths that never congested the sender in the first | |||
slow start can | place, where there's nothing to recover from. That's why slow start can | ||
quit for good: the floor and the probe handle every recovery after it. | |||
=== | === Regimes === | ||
<pre> | <pre> | ||
| Line 385: | Line 352: | ||
| r *= e^(t/T)| | avoidance | | | r *= e^(t/T)| | avoidance | | ||
+-------------+ | | | +-------------+ | | | ||
| m == 0 -> | | m == 0 ---> additive increase | ||
| m > 0 -> | | m > 0 ---> AI + PD decrease | ||
+----------------+ | +----------------+ | ||
m = fed-back ece, or the local first-hop mark when ece == 0 | m = fed-back ece, or the local first-hop mark when ece == 0 | ||
</pre> | </pre> | ||
Because every increase and decrease is scaled by wall-clock time, | == RTT behaviour == | ||
Because every increase and decrease is scaled by wall-clock time, two | |||
flows with different RTTs sharing a bottleneck follow the ''same'' rate | |||
law and settle at the ''same'' rate. So the steady-state | |||
'''allocation is RTT-independent'''. | '''allocation is RTT-independent'''. | ||
That is a property of the equilibrium, not the dynamics. The feedback | |||
delay in the loop ''is'' the path RTT <code>τ</code>, and the | |||
controller | controller neither measures nor compensates for it. At low rate the | ||
receiver window (up to its ~4.3 s ceiling) is the | receiver window (up to its ~4.3 s ceiling) is the biggest lag: the loop | ||
the loop is slow there in proportion to how slow the flow is, | is slow there in proportion to how slow the flow is, just like a | ||
long-RTT TCP. As the rate climbs the window shrinks toward its ~1 ms | |||
toward its ~1 ms floor, so | floor, so its share of the loop delay ''falls'' with capacity and the | ||
''falls'' with capacity and the path RTT <code>τ</code> becomes the | path RTT <code>τ</code> becomes the irreducible part. Two things keep | ||
irreducible | the loop damped: the adaptive window takes out the delay-dominated | ||
window | corner at low <code>τ</code>, and a bucketed | ||
'''rate-velocity damping''' (the washout: each ~31 ms bucket pulls the | |||
pulls the rate a quarter of the way back toward its value at the | rate a quarter of the way back toward its value at the previous bucket) | ||
previous bucket) supplies the damping | supplies the damping this otherwise near-double-integrator loop lacks, | ||
near-double-integrator lacks, which is what holds | which is what holds it together at high capacity and moderate | ||
high capacity and moderate <code>τ</code>. What | <code>τ</code>. What is left is the loop gain | ||
<code>√G ∝ √C</code>: at very high capacity ''and'' large | <code>√G ∝ √C</code>: at very high capacity ''and'' | ||
<code>τ</code> the <code>√G · τ</code> phase budget is the | large <code>τ</code> the <code>√G · τ</code> phase | ||
binding constraint, and no | budget is the binding constraint, and no amount of windowing or damping | ||
[[# | removes it (see [[#Limitations|Limitations]]). So, precisely: | ||
'''RTT-fair in equilibrium, and dynamically stable up to a | '''RTT-fair in equilibrium, and dynamically stable up to a''' | ||
capacity-dependent RTT limit set by <code>√G · τ</code>.''' | '''capacity-dependent RTT limit set by <code>√G · τ</code>.''' | ||
== | == Fairness == | ||
At equilibrium the two increase terms balance the proportional | At equilibrium the two increase terms balance the proportional | ||
decrease, <code>a + r/T_probe = r · m*/M</code>, which pins the | decrease, <code>a + r/T_probe = r · m*/M</code>, which pins the | ||
congestion level | congestion level at <code>m* = M · (a/r + 1/T_probe)</code>. Since | ||
<code>ece = 8q</code> (one mark unit per <code>Q = 4</code> packets, 32 fixed-point | <code>ece = 8q</code> (one mark unit per <code>Q = 4</code> packets, 32 | ||
steps per unit), a standing queue of | fixed-point steps per unit), that is a standing queue of | ||
q* = 64 * a / r + 64 / T_probe (packets, r in bytes/s) | q* = 64 * a / r + 64 / T_probe (packets, r in bytes/s) | ||
The additive part (<code>64 · a / r</code>) | The additive part (<code>64 · a / r</code>) fades to nothing with | ||
the default slope; with the capacity-derived slope (<code>a = C/32</code> | rate under the default slope; with the capacity-derived slope | ||
(<code>a = C/32</code>; see | |||
competing flow, rate | [[#Capacity-derived floor and slope|the capacity floor]]) it settles at | ||
(<code>64 / T_probe</code>) is a rate | <code>2 C / r</code>, about two packets per competing flow, whatever the | ||
8 s). That floor is | rate. The probe part (<code>64 / T_probe</code>) is a floor that does | ||
not move with rate (~8 packets at 8 s). That floor is what | |||
rate-independent convergence costs you, and it is what keeps | |||
<code>q*</code> above the marking threshold at high rate. | |||
Two flows at the same bottleneck see the same <code>m*</code>, so each | Two flows at the same bottleneck see the same <code>m*</code>, so each | ||
solves <code>r_i = a / (m*/M - 1/T_probe)</code> to the same rate. | solves <code>r_i = a / (m*/M - 1/T_probe)</code> to the same rate. In | ||
network-utility-maximization terms that is '''proportional fairness''' | |||
'''proportional fairness''' | with equal weights, which at a single bottleneck is just max-min fair. | ||
a | |||
Getting a good measurement at low rate is the estimator's job, not the | |||
floor's: the averaging window stretches with the flow (see | |||
[[#Receiver estimate|the receiver estimate]]), so a CA-limited flow | |||
still keeps ~16 packets a window down to ~30 kbit/s, and the | |||
rate-relative gap threshold keeps its normal spacing from reading as | |||
idle. The default floor (<code>2^13 B/s</code>) only bounds the extremes | |||
(window ceiling, staleness horizon, pacer arithmetic). The floor itself | |||
scales with the fed-back path capacity (<code>r_min = C/32</code>; see | |||
[[#Capacity-derived floor and slope|the capacity floor]]), so about 32 | |||
CA-limited flows fit above it on any class of link. Flows whose | |||
bottlenecks are in different classes get different slopes, so fairness | |||
between them is capacity-weighted, not equal; see | |||
[[#Limitations|Limitations]]. | |||
== Limitations == | |||
Things the algorithm '''doesn't''' do. Listed here so nobody credits it | |||
with more than it manages. | |||
* '''Queue-only signal, no rate term.''' <code>M</code> is a queue level, so mb-ecn is a standing-queue controller: it has to build a queue (<code>q* = 64 · a / r + 64 / T_probe</code>) to get any signal at all, and can't sit at full utilisation on an empty queue the way a controller with a rate estimate can. The probe floor (~8 packets) keeps <code>q*</code> above the marking threshold at every rate, but you pay for it with a permanent standing queue. | |||
* '''Aggregation is per destination, per QoS cube.''' The aggregate is a stand-in for the path: flows to the same destination are assumed to share a bottleneck, and flows to different destinations run separate loops even when they pile up at the same hop. Each loop still converges to its own fair share (see [[#Fairness|fairness]]), so the cost is duplicated state and probing at that shared queue, not unfairness. One day we could tag the queue in the packet: if a forwarder stamped a queue id next to its mark, senders could key their aggregates on the real bottleneck instead of the destination, and pool the estimate and the probing across everything sharing it. | |||
* '''Capacity is learned, and there are gaps.''' The floor and slope ride on an in-band path-capacity estimate (see [[#Forwarder marking|forwarder marking]] and [[#Capacity-derived floor and slope|the capacity floor]]), and that estimate has its own limits. A hop only learns its rate while its queue is backed up, so a rerouted or brand-new bottleneck reads ''unknown'' at first and the path MIN can briefly come from faster hops (the sender's smoothing and the next feedback bound the overshoot). The measurement is order-of-magnitude on purpose: quarter-log2 code (~19% a step), burst noise held down by a 16-packet minimum window and a max filter, and nothing learned below ~120 packets a second of busy drain (16 packets inside the ~134 ms staleness cap; at MTU that's ~1.4 Mbit/s, below which capacity/32 sits under the default floor anyway). The derived floor clamps to <code>[2^13, 2^32]</code> B/s, and flows with bottlenecks in different classes get capacity-weighted, not equal, shares at a common queue (see [[#Fairness|fairness]]). | |||
* '''The low end trades speed, not correctness.''' Below ~30 kbit/s the averaging window hits its ~4.3 s ceiling and holds fewer samples; below ~64 kbit/s of offered load an ''app-limited'' flow spaces its packets past the gap threshold and rides the raw onset branch, as it always did. At the floor, feedback and the staleness horizon are both measured in seconds, so a slow flow converges slowly, TCP-style. Nothing blows up; nothing's quick either. | |||
* '''No loss response.''' The controller only reacts to ECN marks. Loss is FRCP's problem, handled by retransmission, out of the CA's sight. At a bottleneck that drops instead of marking (say a plain drop-tail queue at the Ethernet or UDP shim), mb-ecn sees nothing and additive increase just keeps ramping. It's built for Ouroboros-native bottlenecks where the forwarder marks; there's no classic-bottleneck fallback. | |||
* '''Dynamic stability has a capacity-dependent RTT limit.''' As covered under [[#RTT behaviour|RTT behaviour]], the allocation is RTT-independent but the dynamics aren't: the binding constraint is <code>√G · τ</code> with <code>√G ∝ √C</code>. The adaptive window and the washout push that limit out a long way (a single flow at 10 GbE stays stable to ~50 ms RTT in the fluid model), but the far corner (100 GbE at ≥ 50 ms) is out of reach for a window/damping controller and would need explicit-rate signalling. | |||
== | == Architectural fit == | ||
Each | Each of these properties falls out of something structural in the | ||
recursive architecture. Here's the mapping. | |||
the mapping | |||
* ''' | * '''CA is fully separate from ARQ and flow control.''' Three concerns, three mechanisms, two scopes: FRCP does retransmission (ARQ) and flow control (the peer pacing the sender), end-to-end and per flow; CA runs in the IPCP, per aggregate. Each signal says one thing. A loss just means retransmit: a lossy link reads as lossy, and the congestion call is left to the marks (the [[#Limitations|limitations]] section's "no loss response" is this same separation from the other side). The peer's flow-control window just paces the endpoint: back-pressure from a slow receiver stays separate from congestion in the network. A retransmitted packet is ordinary traffic as far as the pacer cares. TCP jams all three into one window machine, where the receive window caps the congestion window and loss is both the reliability trigger and the congestion signal; here you can reason about, test, and change each one on its own. | ||
* '''CA sits below what the application picks.''' Every flow in the layer runs under the same rate law, whatever its QoS: a greedy raw sender shares a bottleneck fairly with a reliable stream because the control belongs to the layer, on the aggregate, not to the endpoint transport's good manners. | |||
* '''A layer owns its PCI, so the signal can be rich.''' Everyone in a layer enrolled into it, so the layer is one administrative domain by construction, at whatever scope it covers, and the header is the layer's own. That's why forwarders can write a multi-bit queue magnitude and a capacity byte straight into the packet. The datacenter schemes in [[#Heritage and positioning|heritage and positioning]] need exactly that kind of domain and only get it inside one operator's fabric; a recursive layer has it everywhere. | |||
* '''Layer-internal addressing makes the aggregate well-defined.''' Flows in a layer run between layer addresses, so the (destination, QoS cube) aggregate, which is RFC 3124's macroflow, drops straight out of the naming. Controller state scales with the number of destinations, and each QoS cube keeps its own loop, so service classes don't share a fate. | |||
* '''Flows are allocated, so feedback already has a channel.''' Every flow has state at both ends and a reverse direction, so the receiver's <code>ece</code> and the path capacity just ride the flow allocator's existing exchange. | |||
* '''There's no transport ack clock, and nothing leans on one.''' The layer has to handle raw flows, which carry no acknowledgements at all, so the controller paces on wall-clock time, and that constraint is exactly what buys the RTT-independent allocation under [[#RTT behaviour|RTT behaviour]]. A restriction turned into the feature. | |||
* '''Recursion scales it.''' Each layer handles congestion over its own scope and timescale; many N-flows ride one N-1 flow, so aggregation compounds down the stack. The capacity a layer sees at its egress queue is whatever the layer below is actually giving it (a paced, shared lower flow, not some nominal wire speed), so the signal means something at every level and back-pressure cascades down layer by layer. | |||
* ''' | |||
* '''A layer | |||
* '''Layer-internal addressing makes the aggregate well-defined.''' | |||
* '''Flows are allocated, so feedback has a channel.''' Every flow | |||
* '''There | |||
* '''Recursion scales | |||
There's a boundary: at a shim over legacy media the layer below neither | |||
enrolls nor marks, which is the deployment edge behind the | |||
[[#Limitations|limitations]] section's no-loss-response point. | |||
== | == Heritage and positioning == | ||
mb-ecn is | mb-ecn is basically a rate-based ECN controller in the DECbit / QCN | ||
line, adapted to a recursive layer with no acknowledgements. | |||
The core | The core goes back to the first Ouroboros CA policy (2020): a multi-bit | ||
queue-depth mark relayed by forwarders to the receiver, a smoothed | |||
relayed by forwarders to the receiver, a smoothed multi-bit | multi-bit <code>ece</code> fed back on the reverse flow at the same 1/32 | ||
<code>ece</code> fed back on the reverse flow at the same 1/32 fixed | fixed point used today, and a sender that paced packets from a time | ||
point used today, and a sender that paced packets from a time budget | budget and ran slow start, additive increase and multiplicative | ||
and ran slow start, additive increase and multiplicative decrease on | decrease on wall-clock time slots. Rate-based and acknowledgement-free | ||
wall-clock time slots | from the start, with RTT-independent allocation already the goal. The | ||
the start, with RTT-independent allocation already | |||
present algorithm keeps that skeleton and reworks every estimator | present algorithm keeps that skeleton and reworks every estimator | ||
around it: per-path aggregation with SFQ pacing, the rate-adaptive | around it: per-path aggregation with SFQ pacing, the rate-adaptive | ||
box-car receiver, elapsed-time-scaled AIMD with a derivative term and | |||
rate-velocity damping, and the in-band capacity signal. | rate-velocity damping, and the in-band capacity signal. | ||
* '''DECbit''' (Ramakrishnan & Jain, 1988; ToCS 1990) established | * '''DECbit''' (Ramakrishnan & Jain, 1988; ToCS 1990) established binary congestion feedback set by the switch on queue occupancy. mb-ecn keeps the switch-sets-on-queue idea and carries a ''magnitude''. | ||
* '''The Congestion Manager''' (Balakrishnan et al., SIGCOMM 1999; RFC 3124, 2001) aggregates congestion state at the end host: one controller per macroflow, a scheduler apportioning its rate across the member streams, and new streams joining at the ensemble's current state instead of probing from scratch. mb-ecn's (destination, QoS cube) aggregate is the same idea placed inside the IPCP: every layer manages its own macroflows, recursively, the SFQ pacer plays the CM scheduler's role (see [[#Sender pacer|the pacer]]), and a joining flow rides the aggregate's estimates (see [[#Slow start|slow start]]). The CM gathers its signal from transport feedback at the edge; mb-ecn reads it off the wire from the forwarders. | |||
* '''DCTCP''' (Alizadeh et al., SIGCOMM 2010) recovers a queue magnitude at the endpoint by averaging a single threshold bit. mb-ecn instead reads the magnitude off the wire, then time-averages it at the receiver. The additive-increase / proportional-decrease structure is shared. | |||
* '''QCN''' (IEEE 802.1Qau, 2010) and '''DCQCN''' (Zhu et al., SIGCOMM 2015) contribute the rate-based (not window-based) ECN control skeleton. mb-ecn's proportional decrease plus a probing increase mirrors that skeleton; the increase here is wall-clock scaled rather than byte-counter/timer based. | |||
* '''The Congestion Manager''' (Balakrishnan et al., SIGCOMM 1999; | * '''HPCC''' (Li et al., SIGCOMM 2019) and '''PowerTCP''' (Addanki et al., NSDI 2022) combine a queue term ''and'' a rate term (via in-band telemetry or delay). mb-ecn now carries a coarse in-band rate (the quarter-log2 path MIN; see [[#Forwarder marking|forwarder marking]]) but uses it only to scale the controller's gains; the congestion ''signal'' remains queue-only, which is still the main functional difference (see [[#Limitations|Limitations]]). | ||
* '''L4S / TCP Prague''' (RFC 9330–9332, 2023) make RTT-independence a requirement and use a high-frequency single-bit signal. mb-ecn shares the RTT-independent ''allocation'' goal but reaches it by wall-clock-scaled rate control, with a low-frequency multi-bit signal. | |||
* '''DCTCP''' (Alizadeh et al., SIGCOMM 2010) recovers a queue | |||
* '''QCN''' (IEEE 802.1Qau, 2010) and '''DCQCN''' (Zhu et al., SIGCOMM | |||
* '''HPCC''' (Li et al., SIGCOMM 2019) and '''PowerTCP''' (Addanki et | |||
* '''L4S / TCP Prague''' (RFC 9330–9332, 2023) make | |||
What's new is the pairing: a multi-bit magnitude on the wire with a | |||
wall-clock-scaled rate pacer, in a setting with no per-flow ack clock. | |||
The trade-off, | The trade-off, spelled out above, is a queue-only signal and a | ||
reactivity capped by the averaging window. | reactivity capped by the averaging window. | ||
== | == Parameters == | ||
These are the values every figure in this document | These are the values behind every figure in this document. They're | ||
'''coupled''': the mechanisms only fit together while a few relations | |||
hold | hold. The gap-restart horizon at the rate floor has to exceed a | ||
packet's service time (or a floor-rate flow | packet's service time (or a floor-rate flow loops at onset); a | ||
floor-rate flow's target window has to fit under the window ceiling; | |||
the control interval | the control interval has to sit under the damping bucket, which sits | ||
credit cap | under the idle credit cap; the rate floor mustn't exceed the slow-start | ||
the derived-floor ceiling | seed; and the derived-floor ceiling has to stay under the rate ceiling. | ||
them as a set, against those | Retune them as a set, against those constraints. | ||
{| class="wikitable" | {| class="wikitable" | ||
| Line 662: | Line 566: | ||
== | == References == | ||
=== | === Source === | ||
The implementation follows the algorithm above: | The implementation follows the algorithm above: | ||
* <code>src/ipcpd/unicast/ca/mb-ecn.c</code>, <code>mb-ecn.h</code> | * <code>src/ipcpd/unicast/ca/mb-ecn.c</code>, <code>mb-ecn.h</code>: receiver estimator, pacer and rate control. | ||
* <code>src/ipcpd/unicast/ca/ops.h</code>: CA policy interface. | |||
* <code>src/ipcpd/unicast/ca/ops.h</code> | * <code>src/ipcpd/unicast/cap.c</code>, <code>cap.h</code>: link-capacity estimator and quarter-log2 codec. | ||
* <code>src/ipcpd/unicast/cap.c</code>, <code>cap.h</code> | * <code>src/ipcpd/unicast/dt.c</code>, <code>fa.c</code>: per-hop marking and stamping, and the feedback carrying <code>ece</code> + <code>cap8</code>. | ||
* <code>src/ipcpd/unicast/psched.c</code>: consumes the pacing deadline. | |||
* <code>src/ipcpd/unicast/dt.c</code>, <code>fa.c</code> | |||
* <code>src/ipcpd/unicast/psched.c</code> | |||
=== | === Literature === | ||
* K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion | * K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990. | ||
* H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion Management Architecture for Internet Hosts", SIGCOMM 1999; H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001. | |||
* H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion | |||
* M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010. | * M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010. | ||
* Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" | * Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" (DCQCN), SIGCOMM 2015. | ||
* R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015. | * R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015. | ||
* Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019. | * Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019. | ||
* V. Addanki et al., "PowerTCP", NSDI 2022. | * V. Addanki et al., "PowerTCP", NSDI 2022. | ||
* K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and | * K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and Scalable Throughput (L4S)", RFC 9330/9331/9332. | ||
* F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; R. Srikant, "The Mathematics of Internet Congestion Control", 2004. | |||
* F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; | |||
Latest revision as of 22:23, 11 July 2026
mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast
IPCP. Reliability, ordering and flow control are FRCP's job, end-to-end
and per flow; congestion avoidance is a separate concern, and it runs
in the IPCP, once per layer. Within a layer the IPCP controls
aggregates: one control loop per (destination address, QoS cube),
shared by every flow to that destination.
This document is about the algorithm: what it does and why. The code follows it; the reference section maps the pieces to source files. It doesn't oversell: the limitations section says plainly what it can't do.
"MUST", "SHOULD", "MAY" and friends mean what BCP 14 (RFC 2119, RFC 8174) says, but only where they appear in capitals.
Note: Co-authered by Anthropic Claude Fable 5.
Notation
| Symbol | Units | Meaning |
|---|---|---|
r |
B/s | Paced send rate |
C |
B/s | Bottleneck capacity |
S |
bytes | Packet size |
q |
packets | Standing queue occupancy at the bottleneck |
Q |
packets | Marking quantum: packets per mark unit (4) |
ecn |
8-bit code | Per-packet congestion mark set by a forwarder |
ece |
1/32 ecn |
Receiver-side congestion estimate: 32 × time-mean ecn
|
cap8 |
8-bit code | Path capacity, quarter-log2: C ≈ 2^(cap8 / 4) B/s, 0 = unknown
|
W |
ns | Receiver averaging window |
a |
B/s^2 | Additive-increase slope |
M |
ece units |
Full-congestion reference level (512) |
Δt |
ns | Elapsed wall-clock time between two events |
τ |
n/a | Path round-trip time (feedback delay); the controller does not measure it |
Rate is the control variable throughout.
Every concrete figure in this document (window bounds, horizons, rate thresholds) is just an instance of the parameter values in the parameters table. What the algorithm fixes is the mechanisms and the relations between parameters; retune the parameters and the figures move together.
Overview
mb-ecn is a rate-based, ECN-driven controller with three moving parts, one at each point of the loop:
sender forwarder
+--------------------------------+ +--------------------------+
| SFQ virtual-clock pacer | | ecn = MAX(ecn, queue/Q) |
| rate r, paced per packet | data | (deepest queue wins) |
| AIMD + PD control on r |------>| cap8 = MIN(cap8, link) |
+--------------------------------+ +--------------------------+
^ |
| ece + cap8 feedback | marked data
| v
+--------------------------------+ +--------------------------+
| reads downstream ece, |<------| receiver |
| updates the rate on feedback, | ece | time-mean of ecn over |
| scales floor/slope to cap8 | cap8 | an adaptive window |
+--------------------------------+ +--------------------------+
- The forwarder marks each packet with a multi-bit magnitude proportional to its standing queue, and MIN-stamps its measured outgoing-link capacity into the same PCI.
- The receiver turns the stream of marks into a time-averaged congestion estimate
eceand feeds it back to the sender, together with the window minimum of the path-capacity codes. - The sender paces its rate with a start-time fair-queuing (SFQ) virtual clock, and adjusts that rate with additive-increase / multiplicative-decrease (AIMD) driven by
ece; the rate floor and the additive slope derive from the fed-back path capacity.
There's no timer thread: the controller runs when a packet goes out or feedback comes back. Every step is scaled by elapsed wall-clock time, and that is what makes the allocation independent of RTT (see RTT behaviour).
The unit of control is the aggregate: one per
(destination address, QoS cube), shared by every flow to
that destination at that QoS. They share one controller and one rate.
A new flow joins at whatever rate the aggregate is already running; when
a flow leaves, its share goes to the others. The pacer (SFQ) splits
that rate fairly across the member flows, so per-flow fairness is the
scheduler's doing, not the rate law's.
Forwarder marking
A forwarder marks a packet based on how many packets are already queued on the outgoing link:
mark = queued / Q (Q = 4 packets) ecn = MAX(ecn, mark) (saturating, 8-bit field)
Q is fixed for the whole layer, not tuned per hop: the marks
get MAX-combined across hops, so every forwarder MUST use the same
quantum or the numbers don't compare.
Two things worth noting:
- It's a magnitude, not a bit. The mark is an integer that tracks queue depth, in steps of
Qpackets: a coarse read of the standing queue. - MAX along the path. Cross several forwarders and the packet carries the largest mark, i.e. the deepest queue it passed. Taking the MAX (not a sum) keeps the number bounded and monotone across hops.
There's a dead zone: below Q queued packets the mark is 0.
Capacity stamping
Each forwarder also measures its outgoing link's capacity and stamps it
into the packet. The measurement is a busy-period drain rate: look at
the queue at most once a millisecond, and keep a window open until 16
packets have drained. So the window sizes itself to the link:
the once-a-millisecond cadence caps it on fast links, the 16-packet
drain time on slow ones (~19 ms at 10 Mbit/s). If more than 1/8 of the
arrivals hit an empty queue, throw the window out: the link
wasn't saturated. The odd empty sample (a token-bucket shaper grazing
zero) is fine. A saturated queue drains at the link rate, so a max
filter that decays slowly (1/16 per window) creeps up to it from below.
A window that started or ended on an empty queue might have drained
into buffers downstream faster than the wire, so it may pull the
estimate down but never up. On the wire the code is quarter-log2
(cap8 = 4 log2 C, ~19% a step, 0 = unknown), and each hop
MIN-combines its own into the byte, so the packet arrives carrying the
slowest hop's rate. A hop that has never backed up stamps nothing,
which is fine, since only backed-up hops matter.
Receiver estimate
The receiver turns the marks into a smoothed estimate ece,
averaged over a window W that tracks the incoming byte rate.
It's a plain time-average over that window (a box-car, not an EWMA).
Per packet, with Δt the gap since the last one:
A += ecn * MIN(Δt, W) dwell-weighted; one packet
weighs at most one window
B += S accumulate bytes this window
...
at window close (elapsed >= W, or B >= 2 * B_target with at
least the window floor elapsed):
ece = 32 * A / elapsed mean over the actual window
W += (B_target * elapsed / B - W) / 4 nudge toward the size
that holds the target
W = clamp(W, ~1.05 ms, ~4.3 s)
A = 0 ; B = 0 hard reset
When a window closes, the next one is nudged (a quarter-weight
moving average) toward the size that would hold about 16 packets
at the current byte rate, starting from ~67 ms. The target is in bytes
(B_target = 16,000), so "16 packets" is only exact at
~1000-byte packets, and it takes a few closes to settle. The mean
always divides by the window that actually elapsed, so a change
in rate leaves the current estimate correct and only resizes the
next one. Net effect: a roughly constant ~16 samples a window from
about 30 kbit/s to 122 Mbit/s. Above that the window bottoms out at
~1 ms (still thousands of samples a window at 10–100 GbE); below
~30 kbit/s it hits the ~4.3 s ceiling. The averaging clock stretches
with the flow, the way TCP's ACK clock stretches with the RTT. A
sudden speed-up doesn't wait out a stretched window: once twice the
target bytes have arrived it closes anyway (the ~1 ms floor keeps that
from running away at high rate). So the ece the sender sees
is a staircase: one step per window, steps getting shorter as
the rate climbs. Two edge cases:
- Onset. The first mark after an idle stretch is passed through as-is, so a queue that's just starting to build shows up without waiting out a whole window.
- Gap. A silence longer than 4 windows resets and sends the raw sample. "Long" is measured in the flow's own windows, so a slow flow's normal spacing never reads as idle; and the dwell clamp above limits what a real pause can add to the mean before the reset fires.
Alongside the marks, the receiver keeps the smallest non-zero capacity
code it saw this window and ships it with each ece (on an
onset/gap restart, just the current packet's code). Then it resets, so
if the path reroutes onto something faster the fed-back capacity can
climb within one window.
The estimate rides back to the sender on the reverse flow.
Sender pacer
The sender paces off an SFQ virtual clock. The aggregate keeps a
virtual time V; each flow keeps a finish tag F:
V += r * Δt virtual time advances by bytes served s = MAX(F, V) a packet behind the clock starts now F = s + S the one after it queues behind wait = (s - V) / r time until this packet may go
That wait is the flow's scheduling deadline. A packet
that's behind the virtual clock goes out now (wait = 0);
one that's ahead waits, and only that flow waits, nobody else.
Across an idle gap longer than 50 ms the clock credits at most the
flow's own owed lead plus one burst (50 ms of service, at least one
packet). So a flow paced slower than a packet per 50 ms still gets its
real elapsed service (capping the credit by time would
starve it) while an idle flow still can't bank an unbounded
burst.
V is shared by every flow in the aggregate; the finish
tags are not. Since every flow measures its start tag s
against the same V, a flow that just sent is now ahead of
V and has to wait, while an idle one (tag at or behind
V) goes right away. That is what splits the aggregate rate
r fairly among them.
Rate control (AIMD + PD)
The rate law runs on packet sends, at most once a millisecond, and also when feedback arrives, so a sender with no traffic of its own still reacts. The increase terms and the proportional decrease are each scaled by elapsed time, but not by the same clock: the increases use banked time capped at 50 ms (an idle gap shouldn't buy an unbounded ramp), while the decrease uses the uncapped elapsed time (a starved sender still cuts by the right amount). The one-sided derivative (see the decrease) is not scaled by time at all; it is per-step.
Slow start
Before the first congestion signal, the rate ramps up exponentially with a 20 ms time constant:
r += r * Δt / T_ss (T_ss = 20 ms)
Slow start ends the first time a congestion signal shows up, whether fed back from the receiver or seen at the sender's own first hop, and it never comes back.
Slow start belongs to the aggregate, so it happens once per (destination, QoS cube), on the first flow. A flow that shows up later, at an aggregate that's already running, rides the existing estimate: it starts at the current rate and takes its share through the pacer, without probing of its own.
Increase: additive plus proportional probe (always on)
Past slow start, two increase terms run every step, congested or not:
r += a * Δt additive; a defaults to 2^16 B/s^2 r += r * Δt / T_probe proportional; T_probe = 8 s
The additive term is a fixed slope (bytes/s per second), sized to the
path capacity (see
the capacity floor). The
proportional probe adds a fixed fraction of the rate per unit time (it
e-folds over T_probe): that is what lets a flow recover in
the same number of steps whatever the link rate; a fixed additive step
is negligible next to a fast link. Together they are the
upward pressure the multiplicative decrease balances at equilibrium.
The probe is not free: it costs a standing queue that does not shrink
with rate (see fairness).
Multiplicative decrease (proportional + derivative)
With congestion level m (the fed-back ece, or
the local first-hop mark if there's no feedback yet), the cut is a
proportional term plus a one-sided derivative term, a PD controller on
the congestion signal:
mark = MIN(m, M) M = 512 rise = MAX(m - m_prev, 0), capped at M one-sided derivative
cut = r * rise / (4 * M) derivative, gain 1/4 cut += r * mark * Δt_ms / (1000 * M) proportional cut = MIN(cut, r / 2) at most a halving
r -= cut m_prev = m
- The proportional term uses the real elapsed milliseconds, so a starved sender that hasn't run the controller in a while still cuts by the right amount.
- The derivative term (
rise) only fires when the congestion level goes up between samples: one-sided, per-step. It sharpens the response right as congestion starts. - The cut is capped at
r/2, so one step can at most halve the rate.
M = 512 = 32 × 16 pins "full congestion" at a mean
ecn of 16, i.e. a mean queue of 16 × Q = 64
packets.
Staleness
Feedback comes once per receiver window, and that window stretches with
the flow's byte rate (see
the receiver estimate), so the staleness horizon
has to stretch with it too, or a slow flow's mark would expire
between feedbacks and the rate would ramp straight into a congested
path. The sender mirrors the receiver: the horizon is four target
windows' worth of bytes at the current rate, floored at ~268 ms so fast
flows keep a fixed one; at the default rate floor it's ~8 s. Once a
signal is older than that it's dropped (both the fed-back estimate and
the local mark), which frees the rate to climb again after congestion
clears. Every step clamps the rate to [r_min, r_max].
Capacity-derived floor and slope
The fed-back capacity code sets the controller's two rate-scale constants, per aggregate:
target = decode(cap8) / 32 clamped to [2^13, 2^32] B/s r_min += (target - r_min) / 2 per feedback (moving average) a = r_min
On a path that never reports a capacity, both fall back to fixed
defaults (r_min = 2^13 B/s, a = 2^16 B/s^2).
When the estimator is live, the fairness floor and the post-cut
recovery slope scale with the bottleneck instead, and capacity kicks in
wherever C/32 beats the default floor, roughly above
2 Mbit/s. A capacity older than 16 staleness horizons (~4.3 s for fast
flows) reverts both to the defaults. That horizon is deliberately
longer than the feedback horizon: feedback stops the instant the marks
clear, which is exactly when you need the recovery slope, and the
onset-fresh capacity (see
the receiver estimate) re-seeds it on the first
mark of the next episode anyway.
Ramp and recovery
Three mechanisms set how fast an aggregate reaches a fast link's rate, each owning one regime:
- From scratch: slow start. The exponential ramp hits any real link rate in well under a second (~200 ms from the seed to 10 Gbit/s; every further doubling of link speed adds one ~14 ms doubling). Nothing marks on an uncongested path, so slow start just runs until the aggregate reaches its bottleneck.
- After a cut: the probe. A cut is at most a halving, and the mark expires on the rate-relative horizon once congestion clears (see staleness); the probe then climbs back from a halving in
T_probe · ln 2 ≈ 5.5 s, whatever the link rate. - At the bottom: the capacity floor. The rate never sits more than 32× below a measured bottleneck, and the additive slope refills
C/32a second, so the worst hole is bounded: floor to full capacity in tens of seconds, a halving in seconds.
These fit together because a deep cut needs sustained congestion, sustained congestion backs up the bottleneck queue, and a backed-up queue is exactly when its capacity gets measured and stamped (see forwarder marking). So whenever the controller has been cut deep, the scaled floor is already live. The default floor only ever applies to paths that never congested the sender in the first place, where there's nothing to recover from. That's why slow start can quit for good: the floor and the probe handle every recovery after it.
Regimes
+-------------+ first mark seen +----------------+
| slow start |--------------------->| congestion |
| r *= e^(t/T)| | avoidance |
+-------------+ | |
| m == 0 ---> additive increase
| m > 0 ---> AI + PD decrease
+----------------+
m = fed-back ece, or the local first-hop mark when ece == 0
RTT behaviour
Because every increase and decrease is scaled by wall-clock time, two flows with different RTTs sharing a bottleneck follow the same rate law and settle at the same rate. So the steady-state allocation is RTT-independent.
That is a property of the equilibrium, not the dynamics. The feedback
delay in the loop is the path RTT τ, and the
controller neither measures nor compensates for it. At low rate the
receiver window (up to its ~4.3 s ceiling) is the biggest lag: the loop
is slow there in proportion to how slow the flow is, just like a
long-RTT TCP. As the rate climbs the window shrinks toward its ~1 ms
floor, so its share of the loop delay falls with capacity and the
path RTT τ becomes the irreducible part. Two things keep
the loop damped: the adaptive window takes out the delay-dominated
corner at low τ, and a bucketed
rate-velocity damping (the washout: each ~31 ms bucket pulls the
rate a quarter of the way back toward its value at the previous bucket)
supplies the damping this otherwise near-double-integrator loop lacks,
which is what holds it together at high capacity and moderate
τ. What is left is the loop gain
√G ∝ √C: at very high capacity and
large τ the √G · τ phase
budget is the binding constraint, and no amount of windowing or damping
removes it (see Limitations). So, precisely:
RTT-fair in equilibrium, and dynamically stable up to a
capacity-dependent RTT limit set by √G · τ.
Fairness
At equilibrium the two increase terms balance the proportional
decrease, a + r/T_probe = r · m*/M, which pins the
congestion level at m* = M · (a/r + 1/T_probe). Since
ece = 8q (one mark unit per Q = 4 packets, 32
fixed-point steps per unit), that is a standing queue of
q* = 64 * a / r + 64 / T_probe (packets, r in bytes/s)
The additive part (64 · a / r) fades to nothing with
rate under the default slope; with the capacity-derived slope
(a = C/32; see
the capacity floor) it settles at
2 C / r, about two packets per competing flow, whatever the
rate. The probe part (64 / T_probe) is a floor that does
not move with rate (~8 packets at 8 s). That floor is what
rate-independent convergence costs you, and it is what keeps
q* above the marking threshold at high rate.
Two flows at the same bottleneck see the same m*, so each
solves r_i = a / (m*/M - 1/T_probe) to the same rate. In
network-utility-maximization terms that is proportional fairness
with equal weights, which at a single bottleneck is just max-min fair.
Getting a good measurement at low rate is the estimator's job, not the
floor's: the averaging window stretches with the flow (see
the receiver estimate), so a CA-limited flow
still keeps ~16 packets a window down to ~30 kbit/s, and the
rate-relative gap threshold keeps its normal spacing from reading as
idle. The default floor (2^13 B/s) only bounds the extremes
(window ceiling, staleness horizon, pacer arithmetic). The floor itself
scales with the fed-back path capacity (r_min = C/32; see
the capacity floor), so about 32
CA-limited flows fit above it on any class of link. Flows whose
bottlenecks are in different classes get different slopes, so fairness
between them is capacity-weighted, not equal; see
Limitations.
Limitations
Things the algorithm doesn't do. Listed here so nobody credits it with more than it manages.
- Queue-only signal, no rate term.
Mis a queue level, so mb-ecn is a standing-queue controller: it has to build a queue (q* = 64 · a / r + 64 / T_probe) to get any signal at all, and can't sit at full utilisation on an empty queue the way a controller with a rate estimate can. The probe floor (~8 packets) keepsq*above the marking threshold at every rate, but you pay for it with a permanent standing queue. - Aggregation is per destination, per QoS cube. The aggregate is a stand-in for the path: flows to the same destination are assumed to share a bottleneck, and flows to different destinations run separate loops even when they pile up at the same hop. Each loop still converges to its own fair share (see fairness), so the cost is duplicated state and probing at that shared queue, not unfairness. One day we could tag the queue in the packet: if a forwarder stamped a queue id next to its mark, senders could key their aggregates on the real bottleneck instead of the destination, and pool the estimate and the probing across everything sharing it.
- Capacity is learned, and there are gaps. The floor and slope ride on an in-band path-capacity estimate (see forwarder marking and the capacity floor), and that estimate has its own limits. A hop only learns its rate while its queue is backed up, so a rerouted or brand-new bottleneck reads unknown at first and the path MIN can briefly come from faster hops (the sender's smoothing and the next feedback bound the overshoot). The measurement is order-of-magnitude on purpose: quarter-log2 code (~19% a step), burst noise held down by a 16-packet minimum window and a max filter, and nothing learned below ~120 packets a second of busy drain (16 packets inside the ~134 ms staleness cap; at MTU that's ~1.4 Mbit/s, below which capacity/32 sits under the default floor anyway). The derived floor clamps to
[2^13, 2^32]B/s, and flows with bottlenecks in different classes get capacity-weighted, not equal, shares at a common queue (see fairness). - The low end trades speed, not correctness. Below ~30 kbit/s the averaging window hits its ~4.3 s ceiling and holds fewer samples; below ~64 kbit/s of offered load an app-limited flow spaces its packets past the gap threshold and rides the raw onset branch, as it always did. At the floor, feedback and the staleness horizon are both measured in seconds, so a slow flow converges slowly, TCP-style. Nothing blows up; nothing's quick either.
- No loss response. The controller only reacts to ECN marks. Loss is FRCP's problem, handled by retransmission, out of the CA's sight. At a bottleneck that drops instead of marking (say a plain drop-tail queue at the Ethernet or UDP shim), mb-ecn sees nothing and additive increase just keeps ramping. It's built for Ouroboros-native bottlenecks where the forwarder marks; there's no classic-bottleneck fallback.
- Dynamic stability has a capacity-dependent RTT limit. As covered under RTT behaviour, the allocation is RTT-independent but the dynamics aren't: the binding constraint is
√G · τwith√G ∝ √C. The adaptive window and the washout push that limit out a long way (a single flow at 10 GbE stays stable to ~50 ms RTT in the fluid model), but the far corner (100 GbE at ≥ 50 ms) is out of reach for a window/damping controller and would need explicit-rate signalling.
Architectural fit
Each of these properties falls out of something structural in the recursive architecture. Here's the mapping.
- CA is fully separate from ARQ and flow control. Three concerns, three mechanisms, two scopes: FRCP does retransmission (ARQ) and flow control (the peer pacing the sender), end-to-end and per flow; CA runs in the IPCP, per aggregate. Each signal says one thing. A loss just means retransmit: a lossy link reads as lossy, and the congestion call is left to the marks (the limitations section's "no loss response" is this same separation from the other side). The peer's flow-control window just paces the endpoint: back-pressure from a slow receiver stays separate from congestion in the network. A retransmitted packet is ordinary traffic as far as the pacer cares. TCP jams all three into one window machine, where the receive window caps the congestion window and loss is both the reliability trigger and the congestion signal; here you can reason about, test, and change each one on its own.
- CA sits below what the application picks. Every flow in the layer runs under the same rate law, whatever its QoS: a greedy raw sender shares a bottleneck fairly with a reliable stream because the control belongs to the layer, on the aggregate, not to the endpoint transport's good manners.
- A layer owns its PCI, so the signal can be rich. Everyone in a layer enrolled into it, so the layer is one administrative domain by construction, at whatever scope it covers, and the header is the layer's own. That's why forwarders can write a multi-bit queue magnitude and a capacity byte straight into the packet. The datacenter schemes in heritage and positioning need exactly that kind of domain and only get it inside one operator's fabric; a recursive layer has it everywhere.
- Layer-internal addressing makes the aggregate well-defined. Flows in a layer run between layer addresses, so the (destination, QoS cube) aggregate, which is RFC 3124's macroflow, drops straight out of the naming. Controller state scales with the number of destinations, and each QoS cube keeps its own loop, so service classes don't share a fate.
- Flows are allocated, so feedback already has a channel. Every flow has state at both ends and a reverse direction, so the receiver's
eceand the path capacity just ride the flow allocator's existing exchange. - There's no transport ack clock, and nothing leans on one. The layer has to handle raw flows, which carry no acknowledgements at all, so the controller paces on wall-clock time, and that constraint is exactly what buys the RTT-independent allocation under RTT behaviour. A restriction turned into the feature.
- Recursion scales it. Each layer handles congestion over its own scope and timescale; many N-flows ride one N-1 flow, so aggregation compounds down the stack. The capacity a layer sees at its egress queue is whatever the layer below is actually giving it (a paced, shared lower flow, not some nominal wire speed), so the signal means something at every level and back-pressure cascades down layer by layer.
There's a boundary: at a shim over legacy media the layer below neither enrolls nor marks, which is the deployment edge behind the limitations section's no-loss-response point.
Heritage and positioning
mb-ecn is basically a rate-based ECN controller in the DECbit / QCN line, adapted to a recursive layer with no acknowledgements.
The core goes back to the first Ouroboros CA policy (2020): a multi-bit
queue-depth mark relayed by forwarders to the receiver, a smoothed
multi-bit ece fed back on the reverse flow at the same 1/32
fixed point used today, and a sender that paced packets from a time
budget and ran slow start, additive increase and multiplicative
decrease on wall-clock time slots. Rate-based and acknowledgement-free
from the start, with RTT-independent allocation already the goal. The
present algorithm keeps that skeleton and reworks every estimator
around it: per-path aggregation with SFQ pacing, the rate-adaptive
box-car receiver, elapsed-time-scaled AIMD with a derivative term and
rate-velocity damping, and the in-band capacity signal.
- DECbit (Ramakrishnan & Jain, 1988; ToCS 1990) established binary congestion feedback set by the switch on queue occupancy. mb-ecn keeps the switch-sets-on-queue idea and carries a magnitude.
- The Congestion Manager (Balakrishnan et al., SIGCOMM 1999; RFC 3124, 2001) aggregates congestion state at the end host: one controller per macroflow, a scheduler apportioning its rate across the member streams, and new streams joining at the ensemble's current state instead of probing from scratch. mb-ecn's (destination, QoS cube) aggregate is the same idea placed inside the IPCP: every layer manages its own macroflows, recursively, the SFQ pacer plays the CM scheduler's role (see the pacer), and a joining flow rides the aggregate's estimates (see slow start). The CM gathers its signal from transport feedback at the edge; mb-ecn reads it off the wire from the forwarders.
- DCTCP (Alizadeh et al., SIGCOMM 2010) recovers a queue magnitude at the endpoint by averaging a single threshold bit. mb-ecn instead reads the magnitude off the wire, then time-averages it at the receiver. The additive-increase / proportional-decrease structure is shared.
- QCN (IEEE 802.1Qau, 2010) and DCQCN (Zhu et al., SIGCOMM 2015) contribute the rate-based (not window-based) ECN control skeleton. mb-ecn's proportional decrease plus a probing increase mirrors that skeleton; the increase here is wall-clock scaled rather than byte-counter/timer based.
- HPCC (Li et al., SIGCOMM 2019) and PowerTCP (Addanki et al., NSDI 2022) combine a queue term and a rate term (via in-band telemetry or delay). mb-ecn now carries a coarse in-band rate (the quarter-log2 path MIN; see forwarder marking) but uses it only to scale the controller's gains; the congestion signal remains queue-only, which is still the main functional difference (see Limitations).
- L4S / TCP Prague (RFC 9330–9332, 2023) make RTT-independence a requirement and use a high-frequency single-bit signal. mb-ecn shares the RTT-independent allocation goal but reaches it by wall-clock-scaled rate control, with a low-frequency multi-bit signal.
What's new is the pairing: a multi-bit magnitude on the wire with a wall-clock-scaled rate pacer, in a setting with no per-flow ack clock. The trade-off, spelled out above, is a queue-only signal and a reactivity capped by the averaging window.
Parameters
These are the values behind every figure in this document. They're coupled: the mechanisms only fit together while a few relations hold. The gap-restart horizon at the rate floor has to exceed a packet's service time (or a floor-rate flow loops at onset); a floor-rate flow's target window has to fit under the window ceiling; the control interval has to sit under the damping bucket, which sits under the idle credit cap; the rate floor mustn't exceed the slow-start seed; and the derived-floor ceiling has to stay under the rate ceiling. Retune them as a set, against those constraints.
| Parameter | Value | Meaning |
|---|---|---|
Marking quantum Q |
4 packets | Mark onset and step; layer-wide |
| Feedback resolution | 1/32 ecn unit |
Fixed-point LSB of ece
|
Full-congestion reference M |
512 | Mean queue of 64 packets |
| Receiver window target | 16 packets (16,000 bytes) | Samples per window |
| Initial receiver window | ~67 ms | Before rate is known |
| Receiver window bounds | ~1.05 ms – ~4.3 s | Clamp on W
|
| Window resize weight | 1/4 | Moving average toward target |
| Early window close | 2 × target bytes | Speed-up escape |
| Gap restart | 4 windows | Idle horizon (rate-relative) |
Additive slope a (default) |
2^16 B/s^2 | Lower clamp of derived slope |
Probe time constant T_probe |
8 s | Rate-proportional recovery |
| Derivative gain | 1/4 | Onset sharpening |
Rate floor r_min (default) |
2^13 B/s | Lower clamp of derived floor |
| Slow-start seed rate | 2^16 B/s | Initial rate |
| Slow-start time constant | 20 ms | Exponential ramp |
Rate ceiling r_max |
2^37 B/s | Upper clamp |
| Capacity scale | C / 32 |
Floor and slope from bottleneck |
| Capacity smoothing | 1/2 | Moving average per feedback |
| Capacity staleness | 16 × feedback horizon | Revert to defaults |
| Derived floor ceiling | 2^32 B/s | Upper clamp on derived floor |
| Control interval | 1 ms | Minimum spacing of rate updates |
| Idle credit cap | 50 ms | Burst allowance / increase banking |
| Feedback staleness floor | ~268 ms | Floor of rate-relative horizon |
| Damping bucket | ~31 ms | Rate-velocity washout cadence |
| Damping fraction | 1/4 | Pullback per bucket |
| Estimator fold spacing | 1 ms | Minimum measurement cadence |
| Estimator window closure | 16 packets drained | Self-scaling window |
| Estimator window staleness | ~134 ms | Discard across traffic gaps |
| Estimator idle tolerance | 1/8 of arrivals | Unsaturated-window rejection |
| Estimator decay | 1/16 per window | Max-filter convergence from below |
References
Source
The implementation follows the algorithm above:
src/ipcpd/unicast/ca/mb-ecn.c,mb-ecn.h: receiver estimator, pacer and rate control.src/ipcpd/unicast/ca/ops.h: CA policy interface.src/ipcpd/unicast/cap.c,cap.h: link-capacity estimator and quarter-log2 codec.src/ipcpd/unicast/dt.c,fa.c: per-hop marking and stamping, and the feedback carryingece+cap8.src/ipcpd/unicast/psched.c: consumes the pacing deadline.
Literature
- K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990.
- H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion Management Architecture for Internet Hosts", SIGCOMM 1999; H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001.
- M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010.
- Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" (DCQCN), SIGCOMM 2015.
- R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015.
- Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019.
- V. Addanki et al., "PowerTCP", NSDI 2022.
- K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and Scalable Throughput (L4S)", RFC 9330/9331/9332.
- F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; R. Srikant, "The Mathematics of Internet Congestion Control", 2004.