Congestion avoidance: Difference between revisions

From Ouroboros
Jump to navigation Jump to search
No edit summary
 
(5 intermediate revisions by the same user not shown)
Line 2: Line 2:


mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast
mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast
IPCP. It lives in the IPC Process: the flow-and-retransmission task
IPCP. Reliability, ordering and flow control are FRCP's job, end-to-end
(FRCP) provides reliability, in-order delivery and flow control
and per flow; congestion avoidance is a separate concern, and it runs
end-to-end; congestion avoidance is orthogonal and runs per IPCP
in the IPCP, once per layer. Within a layer the IPCP controls
layer. Within a layer, the IPCP controls '''aggregates''', one loop
'''aggregates''': one control loop per (destination address, QoS cube),
per (destination address, QoS cube), shared by every flow toward that
shared by every flow to that destination.
destination. The alternative policy is <code>nop</code> (no congestion
avoidance).


This document describes the algorithm: what the policy does and why.
This document is about the algorithm: what it does and why. The
The implementation follows it ([[#12. References|Section 12]] maps the
code follows it; the [[#References|reference section]] maps the pieces
pieces to source files). It does not claim properties the algorithm
to source files. It doesn't oversell: the [[#Limitations|limitations]]
does not have; [[#8. Limitations|Section 8]] states the known gaps
section says plainly what it can't do.
explicitly.


The keywords "MUST", "SHOULD", "MAY" etc. are used only where they
"MUST", "SHOULD", "MAY" and friends mean what BCP 14 (RFC 2119,
appear in all capitals, per BCP 14 (RFC 2119, RFC 8174).
RFC 8174) says, but only where they appear in capitals.


Note: Co-authered by Anthropic Claude Fable 5.


== Notation ==
== Notation ==


;<code>r</code>
{| class="wikitable"
:Paced send rate, bytes per second.
! Symbol !! Units !! Meaning
;<code>C</code>
|-
:Bottleneck capacity, bytes per second.
| <code>r</code> || B/s || Paced send rate
;<code>S</code>
|-
:Packet size in bytes.
| <code>C</code> || B/s || Bottleneck capacity
;<code>q</code>
|-
:Standing queue occupancy at the bottleneck, in packets.
| <code>S</code> || bytes || Packet size
;<code>Q</code>
|-
:Marking quantum, packets per mark unit (4).
| <code>q</code> || packets || Standing queue occupancy at the bottleneck
;<code>ecn</code>
|-
:Per-packet congestion mark set by a forwarder, an unsigned integer
| <code>Q</code> || packets || Marking quantum: packets per mark unit (4)
(on-wire field is 8-bit).
|-
;<code>ece</code>
| <code>ecn</code> || 8-bit code || Per-packet congestion mark set by a forwarder
:Receiver-side congestion estimate, fixed point with LSB = 1/32 of an
|-
<code>ecn</code> unit, i.e. <code>ece = 32 &times; (time-mean ecn)</code>.
| <code>ece</code> || 1/32 <code>ecn</code> || Receiver-side congestion estimate: 32 &times; time-mean <code>ecn</code>
;<code>cap8</code>
|-
:Path-capacity code carried per packet, quarter-log2:
| <code>cap8</code> || 8-bit code || Path capacity, quarter-log2: <code>C &asymp; 2^(cap8 / 4)</code> B/s, 0 = unknown
<code>C &asymp; 2^(cap8 / 4)</code> bytes/s, 0 = unknown.
|-
;<code>W</code>
| <code>W</code> || ns || Receiver averaging window
:Receiver averaging window, nanoseconds.
|-
;<code>a</code>
| <code>a</code> || B/s^2 || Additive-increase slope
:Additive-increase slope, bytes/s per second.
|-
;<code>M</code>
| <code>M</code> || <code>ece</code> units || Full-congestion reference level (512)
:Full-congestion reference level of <code>ece</code> (512).
|-
;<code>&Delta;t</code>
| <code>&Delta;t</code> || ns || Elapsed wall-clock time between two events
:Elapsed wall-clock time between two events, nanoseconds.
|-
;<code>&tau;</code>
| <code>&tau;</code> || n/a || Path round-trip time (feedback delay); the controller does not measure it
:Path round-trip time (feedback delay). The controller does not
|}
measure it.


Rate is the control variable throughout.
Rate is the control variable throughout.


All concrete figures in this document &mdash; window bounds, horizons,
Every concrete figure in this document (window bounds, horizons,
rate thresholds &mdash; are instances of the parameter values in
rate thresholds) is just an instance of the parameter values in the
[[#11. Parameters|Section 11]]. The mechanisms and the relations
[[#Parameters|parameters table]]. What the algorithm fixes is the
between parameters are what the algorithm fixes; retuning the
mechanisms and the relations between parameters; retune the parameters
parameters moves the figures together.
and the figures move together.


__TOC__
__TOC__
Line 85: Line 82:
</pre>
</pre>


# The '''forwarder''' marks each packet with a ''multi-bit'' magnitude
# The '''forwarder''' marks each packet with a ''multi-bit'' magnitude proportional to its standing queue, and MIN-stamps its measured outgoing-link capacity into the same PCI.
  proportional to its standing queue, and MIN-stamps its measured
# The '''receiver''' turns the stream of marks into a time-averaged congestion estimate <code>ece</code> and feeds it back to the sender, together with the window minimum of the path-capacity codes.
  outgoing-link capacity into the same PCI.
# The '''sender''' paces its rate with a start-time fair-queuing (SFQ) virtual clock, and adjusts that rate with additive-increase / multiplicative-decrease (AIMD) driven by <code>ece</code>; the rate floor and the additive slope derive from the fed-back path capacity.
# The '''receiver''' turns the stream of marks into a time-averaged
 
  congestion estimate <code>ece</code> and feeds it back to the sender,
There's no timer thread: the controller runs when a packet goes out or
  together with the window minimum of the path-capacity codes.
feedback comes back. Every step is scaled by elapsed wall-clock time,
# The '''sender''' paces its rate with a start-time fair-queuing (SFQ)
and that is what makes the ''allocation'' independent of RTT (see
  virtual clock, and adjusts that rate with additive-increase /
[[#RTT behaviour|RTT behaviour]]).
  multiplicative-decrease (AIMD) driven by <code>ece</code>; the rate floor
  and the additive slope derive from the fed-back path capacity.


The controller runs on packet sends and on feedback arrivals. All
The unit of control is the '''aggregate''': one per
increase/decrease steps are scaled by elapsed
<code>(destination address, QoS cube)</code>, shared by every flow to
wall-clock time, which is what makes the ''allocation'' independent of
that destination at that QoS. They share one controller and one rate.
RTT (see [[#6. RTT behaviour|Section 6]]).
A new flow joins at whatever rate the aggregate is already running; when
a flow leaves, its share goes to the others. The pacer (SFQ) splits
that rate fairly across the member flows, so per-flow fairness is the
scheduler's doing, not the rate law's.


The unit of control is the '''aggregate''', kept in the IPCP per
<code>(destination address, QoS cube)</code>. Every flow toward that
destination at that QoS shares one controller and one rate: a new
flow joins the aggregate at its current rate, and a departing flow
leaves the rate to the others. The pacer divides the aggregate rate
fairly across the member flows (SFQ), so per-flow fairness within an
aggregate is a scheduler property, distinct from the rate law
described here.


== 2. Forwarder marking ==
== Forwarder marking ==


A forwarder marks a packet as a function of the packets already queued
A forwarder marks a packet based on how many packets are already queued
on the outgoing link:
on the outgoing link:


Line 118: Line 108:
   ecn  = MAX(ecn, mark)                    (saturating, 8-bit field)
   ecn  = MAX(ecn, mark)                    (saturating, 8-bit field)


The quantum <code>Q</code> is a layer-wide constant: the marks it
<code>Q</code> is fixed for the whole layer, not tuned per hop: the marks
quantizes are combined by MAX across hops, so all forwarders in a
get MAX-combined across hops, so every forwarder MUST use the same
layer MUST use the same quantum for the signal to be comparable.
quantum or the numbers don't compare.


Two properties:
Two things worth noting:


* '''Multi-bit magnitude.''' The mark is an integer proportional to
* '''It's a magnitude, not a bit.''' The mark is an integer that tracks queue depth, in steps of <code>Q</code> packets: a coarse read of the standing queue.
  queue depth, quantized in steps of <code>Q</code> packets: a coarse
* '''MAX along the path.''' Cross several forwarders and the packet carries the largest mark, i.e. the deepest queue it passed. Taking the MAX (not a sum) keeps the number bounded and monotone across hops.
  readout of the standing queue.
* '''MAX along the path.''' A packet crossing several forwarders carries
  the maximum mark, i.e. the deepest queue on its path. MAX keeps the
  signal range-stable and monotone across hops.


There is a '''dead zone''': for <code>queued < Q</code> the mark is 0.
There's a '''dead zone''': below <code>Q</code> queued packets the mark is 0.


=== Capacity stamping ===
=== Capacity stamping ===


Each forwarder also measures the capacity of its outgoing link and
Each forwarder also measures its outgoing link's capacity and stamps it
stamps it into the packet. The estimate is a busy-period drain rate:
into the packet. The measurement is a busy-period drain rate: look at
the queue is observed at most once per millisecond, and a measurement
the queue at most once a millisecond, and keep a window open until 16
window stays open until 16 packets have drained, so its length
packets have drained. So the window '''sizes itself to the link''':
'''self-scales with the link rate''' &mdash; the millisecond cadence
the once-a-millisecond cadence caps it on fast links, the 16-packet
bounds it on fast links, the 16-packet drain time on slow ones
drain time on slow ones (~19 ms at 10 Mbit/s). If more than 1/8 of the
(~19 ms at 10 Mbit/s). A window in which more than 1/8 of
arrivals hit an empty queue, throw the window out: the link
the arrivals found the queue empty is discarded as unsaturated;
wasn't saturated. The odd empty sample (a token-bucket shaper grazing
occasional empty observations (token-bucket shapers grazing zero) are
zero) is fine. A saturated queue drains at the link rate, so a max
tolerated. The drain rate of a saturated queue is the link rate; a max
filter that decays slowly (1/16 per window) creeps up to it from below.
filter with slow decay (1/16 per window) converges on it from below. A
A window that started or ended on an empty queue might have drained
window that opened or closed on an empty queue may have drained into
into buffers downstream faster than the wire, so it may pull the
buffers below at above-wire rate, so it may lower the estimate but
estimate down but never up. On the wire the code is quarter-log2
never raise it. The code on the wire is quarter-log2
(<code>cap8 = 4 log2 C</code>, ~19% a step, 0 = unknown), and each hop
(<code>cap8 = 4 log2 C</code>, ~19% per step, 0 = unknown) and each hop
MIN-combines its own into the byte, so the packet arrives carrying the
MIN-combines its own code into the byte, so a packet arrives carrying
slowest hop's rate. A hop that has never backed up stamps nothing,
the bottleneck's rate. A hop that has never been backlogged stamps
which is fine, since only backed-up hops matter.
nothing; since only backlogged hops matter, the signal exists exactly
when it is needed.




== 3. Receiver estimate ==
== Receiver estimate ==


The receiver converts marks into a smoothed estimate <code>ece</code> over a
The receiver turns the marks into a smoothed estimate <code>ece</code>,
time window <code>W</code> that '''adapts to the incoming byte rate'''. It is
averaged over a window <code>W</code> that '''tracks the incoming byte rate'''.
a '''boxcar (rectangular) time-integral mean'''. Per packet, with
It's a plain time-average over that window (a box-car, not an EWMA).
<code>&Delta;t</code> the gap since the previous packet:
Per packet, with <code>&Delta;t</code> the gap since the last one:


   A += ecn * MIN(&Delta;t, W)              dwell-weighted; one packet
   A += ecn * MIN(&Delta;t, W)              dwell-weighted; one packet
Line 175: Line 159:
       A  = 0 ; B = 0                    hard reset
       A  = 0 ; B = 0                    hard reset


At each close the next window is nudged by a quarter-weight moving
When a window closes, the next one is nudged (a quarter-weight
average toward the size that would hold about 16 packets at the
moving average) toward the size that would hold about 16 packets
measured byte rate, from an initial ~67 ms. The target is byte-based (<code>B_target</code> = 16,000 bytes), so
at the current byte rate, starting from ~67 ms. The target is in bytes
"16 packets" is exact only at ~1000-byte packets, and the window
(<code>B_target</code> = 16,000), so "16 packets" is only exact at
converges to it over several closes. The mean always divides by the
~1000-byte packets, and it takes a few closes to settle. The mean
'''actual''' elapsed window, so a rate step leaves the current
always divides by the window that '''actually''' elapsed, so a change
estimate exact and only re-sizes the ''next'' window. The effect is a roughly constant sample
in rate leaves the current estimate correct and only resizes the
count &mdash; ~16 packets &mdash; from about 30 kbit/s to 122 Mbit/s;
''next'' one. Net effect: a roughly constant ~16 samples a window from
above that the window floors at ~1 ms (a cadence still carrying
about 30 kbit/s to 122 Mbit/s. Above that the window bottoms out at
thousands of samples at 10&ndash;100 GbE), and below ~30 kbit/s it
~1 ms (still thousands of samples a window at 10&ndash;100 GbE); below
saturates at the ~4.3 s ceiling. The averaging clock stretches with
~30 kbit/s it hits the ~4.3 s ceiling. The averaging clock stretches
the flow the way TCP's ACK clock stretches with the RTT. A speed-up
with the flow, the way TCP's ACK clock stretches with the RTT. A
escapes a stretched window early: once twice the target bytes arrive
sudden speed-up doesn't wait out a stretched window: once twice the
the window closes anyway (the window floor keeps that cadence bounded
target bytes have arrived it closes anyway (the ~1 ms floor keeps that
at high rate). The fed-back <code>ece</code> is thus a piecewise-constant
from running away at high rate). So the <code>ece</code> the sender sees
staircase whose step period tracks the rate. Two edge cases:
is a staircase: one step per window, steps getting shorter as
the rate climbs. Two edge cases:


* '''Onset''' (first mark after an idle estimate) emits the
* '''Onset.''' The first mark after an idle stretch is passed through as-is, so a queue that's just starting to build shows up without waiting out a whole window.
  instantaneous mark undiluted, so a starting queue is reported without
* '''Gap.''' A silence longer than 4 windows resets and sends the raw sample. "Long" is '''measured in the flow's own windows''', so a slow flow's normal spacing never reads as idle; and the dwell clamp above limits what a real pause can add to the mean before the reset fires.
  a full window of delay.
* '''Gap''' longer than 4 windows resets the window and emits the raw
  sample. The threshold is '''rate-relative''': idle means a few
  ''current'' windows of silence, so a slow flow's normal inter-packet
  gap never reads as idle, while the in-window dwell clamp above bounds
  what a genuine pause can add to the mean before the restart fires.


Alongside the mark integral, the receiver keeps the MIN of the nonzero
Alongside the marks, the receiver keeps the smallest non-zero capacity
capacity codes seen in the current window; each emitted <code>ece</code>
code it saw this window and ships it with each <code>ece</code> (on an
carries that minimum (the raw packet's code on onset/gap restarts) and
onset/gap restart, just the current packet's code). Then it resets, so
the fold then resets, so a reroute to a faster path can raise the
if the path reroutes onto something faster the fed-back capacity can
fed-back capacity within one window.
climb within one window.


The estimate is fed back to the sender on the reverse flow.
The estimate rides back to the sender on the reverse flow.




== 4. Sender pacer ==
== Sender pacer ==


The sender paces with an SFQ virtual clock. The context keeps a
The sender paces off an SFQ virtual clock. The aggregate keeps a
virtual time <code>V</code>; each flow keeps a finish tag <code>F</code>:
virtual time <code>V</code>; each flow keeps a finish tag <code>F</code>:


Line 220: Line 199:
   wait  = (s - V) / r                time until this packet may go
   wait  = (s - V) / r                time until this packet may go


The <code>wait</code> becomes the flow's scheduling deadline: a packet
That <code>wait</code> is the flow's scheduling deadline. A packet
already behind the virtual clock is sent immediately
that's behind the virtual clock goes out now (<code>wait = 0</code>);
(<code>wait = 0</code>); one ahead waits, deferring only its own flow.
one that's ahead waits, and only that flow waits, nobody else.
A clock advance across an idle gap longer than 50 ms credits at most
Across an idle gap longer than 50 ms the clock credits at most the
the calling flow's owed lead plus one burst (50 ms of service, at
flow's own owed lead plus one burst (50 ms of service, at least one
least one packet): a paced flow slower than one packet per 50 ms
packet). So a flow paced slower than a packet per 50 ms still gets its
receives its '''true''' elapsed service &mdash; a time-capped credit
'''real''' elapsed service (capping the credit by time would
would decay such a flow without bound &mdash; while an idle flow still
starve it) while an idle flow still can't bank an unbounded
cannot bank an unbounded burst.
burst.


The virtual time <code>V</code> is shared by every flow in the context;
<code>V</code> is shared by every flow in the aggregate; the finish
the finish tags are per flow. Because every flow computes its start
tags are not. Since every flow measures its start tag <code>s</code>
tag <code>s</code> against the same <code>V</code>, a flow that has just sent
against the same <code>V</code>, a flow that just sent is now ahead of
carries a finish tag ahead of <code>V</code> and waits, while an idle flow
<code>V</code> and has to wait, while an idle one (tag at or behind
(finish tag at or behind <code>V</code>) starts immediately &mdash; this is
<code>V</code>) goes right away. That is what splits the aggregate rate
what fairly divides the aggregate rate <code>r</code> across them.
<code>r</code> fairly among them.




== 5. Rate control (AIMD + PD) ==
== Rate control (AIMD + PD) ==


The rate law runs on packet sends, at most once per millisecond, and
The rate law runs on packet sends, at most once a millisecond, and
also on feedback arrivals so that a sender with no traffic of its own
also when feedback arrives, so a sender with no traffic of its
still reacts. Each increase term and the proportional decrease are
own still reacts. The increase terms and the proportional decrease are
scaled by elapsed wall-clock time, each by its own clock: the
each scaled by elapsed time, but not by the same clock: the increases
increase terms use time banked and '''capped''' at 50 ms (an idle gap
use banked time '''capped''' at 50 ms (an idle gap shouldn't buy an
cannot bank an unbounded ramp), while the proportional decrease uses
unbounded ramp), while the decrease uses the '''uncapped''' elapsed
the '''uncapped''' elapsed time (a starved sender still cuts by the
time (a starved sender still cuts by the right amount). The one-sided
right amount). The one-sided derivative (Section 5.3) is a per-delta
derivative (see the
term.
[[#Multiplicative decrease (proportional + derivative)|decrease]]) is
not scaled by time at all; it is per-step.


=== 5.1. Slow start ===
=== Slow start ===


Before the first congestion signal the rate ramps exponentially with a
Before the first congestion signal, the rate ramps up exponentially
20 ms time constant:
with a 20 ms time constant:


   r += r * &Delta;t / T_ss                  (T_ss = 20 ms)
   r += r * &Delta;t / T_ss                  (T_ss = 20 ms)


Slow start ends the first time a congestion signal arrives &mdash;
Slow start ends the first time a congestion signal shows up, whether
fed back from the receiver or observed at the sender's own first hop
fed back from the receiver or seen at the sender's own first hop, and
&mdash; and never resumes.
it never comes back.


Slow start belongs to the aggregate, so it runs '''once per
Slow start belongs to the aggregate, so it happens
(destination, QoS cube)''', on the first flow. A flow arriving at an
'''once per (destination, QoS cube)''', on the first flow. A flow that
established aggregate rides the existing estimates: it enters at the
shows up later, at an aggregate that's already running, rides the
aggregate's current rate and receives its fair share of it through
existing
the pacer (Section 4), with no probing of its own.
estimate: it starts at the current rate and takes its share through
the [[#Sender pacer|pacer]], without probing of its own.


=== 5.2. Increase: additive plus proportional probe (always on) ===
=== Increase: additive plus proportional probe (always on) ===


Past slow start, two increase terms are applied every step, both
Past slow start, two increase terms run every step, congested or not:
regardless of congestion:


   r += a * &Delta;t                        additive; a defaults to 2^16 B/s^2
   r += a * &Delta;t                        additive; a defaults to 2^16 B/s^2
   r += r * &Delta;t / T_probe              proportional; T_probe = 8 s
   r += r * &Delta;t / T_probe              proportional; T_probe = 8 s


The additive term is a fixed per-context slope (bytes/s per second),
The additive term is a fixed slope (bytes/s per second), sized to the
scaled to the path capacity (Section 5.5). The proportional probe
path capacity (see
grows the rate by a fixed fraction per unit time (e-folds over
[[#Capacity-derived floor and slope|the capacity floor]]). The
<code>T_probe</code>): a flow recovers in the same number of steps at any
proportional probe adds a fixed fraction of the rate per unit time (it
link rate; the fixed additive term's effect vanishes relative to a
e-folds over <code>T_probe</code>): that is what lets a flow recover in
fast link. Both are the "probing"
the same number of steps whatever the link rate; a fixed additive step
pressure the multiplicative decrease balances at equilibrium; the
is negligible next to a fast link. Together they are the
probe's cost is a rate-independent standing-queue floor (Section 7).
upward pressure the multiplicative decrease balances at equilibrium.
The probe is not free: it costs a standing queue that does not shrink
with rate (see [[#Fairness|fairness]]).


=== 5.3. Multiplicative decrease (proportional + derivative) ===
=== Multiplicative decrease (proportional + derivative) ===


With congestion level <code>m</code> (the fed-back <code>ece</code>, or a local
With congestion level <code>m</code> (the fed-back <code>ece</code>, or
first-hop mark as fallback), the decrease is the sum of a proportional
the local first-hop mark if there's no feedback yet), the cut is a
term and a one-sided derivative term &mdash; a PD controller on the
proportional term plus a one-sided derivative term, a PD controller on
congestion signal:
the congestion signal:


   mark = MIN(m, M)                                  M = 512
   mark = MIN(m, M)                                  M = 512
Line 301: Line 283:
   m_prev = m
   m_prev = m


* The '''proportional''' term is scaled by honest elapsed time in
* The '''proportional''' term uses the real elapsed milliseconds, so a starved sender that hasn't run the controller in a while still cuts by the right amount.
  milliseconds, so a starved sender that has not run the controller
* The '''derivative''' term (<code>rise</code>) only fires when the congestion level ''goes up'' between samples: one-sided, per-step. It sharpens the response right as congestion starts.
  for a while still cuts by the right amount.
* The cut is capped at <code>r/2</code>, so one step can at most halve the rate.
* The '''derivative''' term (<code>rise</code>) reacts to an ''increase'' in
  the congestion level between samples; it is one-sided and per-delta.
  It sharpens the reaction at the onset of congestion.
* The cut is clamped to <code>r/2</code> per step, bounding the decrease to
  a halving.


<code>M = 512 = 32 &times; 16</code> defines "full congestion" as a mean
<code>M = 512 = 32 &times; 16</code> pins "full congestion" at a mean
<code>ecn</code> of 16, i.e. a mean queue of <code>16 &times; Q = 64</code> packets.
<code>ecn</code> of 16, i.e. a mean queue of <code>16 &times; Q = 64</code>
packets.


=== 5.4. Staleness ===
=== Staleness ===


Feedback arrives once per receiver window, and the window stretches
Feedback comes once per receiver window, and that window stretches with
with the flow's byte rate (Section 3) &mdash; so the staleness horizon
the flow's byte rate (see
must stretch with it, or a slow flow's mark would age out ''between''
[[#Receiver estimate|the receiver estimate]]), so the staleness horizon
feedbacks and the rate would ramp against a congested path. The
has to stretch with it too, or a slow flow's mark would expire
sender mirrors the receiver: the horizon is four target windows of
''between'' feedbacks and the rate would ramp straight into a congested
bytes at the current rate, floored at ~268 ms so fast flows keep a
path. The sender mirrors the receiver: the horizon is four target
fixed horizon; at the default rate floor it is ~8 s. A signal older
windows' worth of bytes at the current rate, floored at ~268 ms so fast
than the horizon is aged out (both the fed-back estimate and the
flows keep a fixed one; at the default rate floor it's ~8 s. Once a
local mark), freeing the rate to ramp once congestion clears. The
signal is older than that it's dropped (both the fed-back estimate and
rate is clamped to
the local mark), which frees the rate to climb again after congestion
<code>[r_min, r_max]</code> on every step.
clears. Every step clamps the rate to <code>[r_min, r_max]</code>.


=== 5.5. Capacity-derived floor and slope ===
=== Capacity-derived floor and slope ===


The fed-back capacity code sets the two rate-scale constants of the
The fed-back capacity code sets the controller's two rate-scale
controller, per context:
constants, per aggregate:


   target = decode(cap8) / 32              clamped to [2^13, 2^32] B/s
   target = decode(cap8) / 32              clamped to [2^13, 2^32] B/s
Line 336: Line 314:
   a      = r_min
   a      = r_min


The defaults are fixed constants (<code>r_min = 2^13 B/s</code>,
On a path that never reports a capacity, both fall back to fixed
<code>a = 2^16 B/s^2</code>), used on paths that never report a capacity;
defaults (<code>r_min = 2^13 B/s</code>, <code>a = 2^16 B/s^2</code>).
with the estimator live the fairness floor and the post-cut recovery
When the estimator is live, the fairness floor and the post-cut
slope scale with the bottleneck, and capacity engages wherever
recovery slope scale with the bottleneck instead, and capacity kicks in
<code>C/32</code> clears the default floor, i.e. above ~2 Mbit/s. A
wherever <code>C/32</code> beats the default floor, roughly above
capacity older than 16 staleness horizons (~4.3 s for fast flows)
2 Mbit/s. A capacity older than 16 staleness horizons (~4.3 s for fast
reverts both to the defaults. That horizon deliberately outlives the
flows) reverts both to the defaults. That horizon is deliberately
feedback horizon: feedback stops the moment the marks clear, which is
longer than the feedback horizon: feedback stops the instant the marks
exactly when the recovery slope is needed; the onset-fresh capacity
clear, which is exactly when you need the recovery slope, and the
(Section 3) re-seeds it on the first mark of the next episode either
onset-fresh capacity (see
way.
[[#Receiver estimate|the receiver estimate]]) re-seeds it on the first
mark of the next episode anyway.


=== 5.6. Ramp and recovery ===
=== Ramp and recovery ===


Three mechanisms set how fast an aggregate reaches a fast link's
Three mechanisms set how fast an aggregate reaches a fast link's
rate, each owning one regime:
rate, each owning one regime:


* '''From scratch: slow start.''' The exponential ramp reaches any
* '''From scratch: slow start.''' The exponential ramp hits any real link rate in well under a second (~200 ms from the seed to 10 Gbit/s; every further doubling of link speed adds one ~14 ms doubling). Nothing marks on an uncongested path, so slow start just runs until the aggregate reaches its bottleneck.
  physical link rate in well under a second (~200 ms from the seed to
* '''After a cut: the probe.''' A cut is at most a halving, and the mark expires on the rate-relative horizon once congestion clears (see [[#Staleness|staleness]]); the probe then climbs back from a halving in <code>T_probe &middot; ln 2 &asymp; 5.5 s</code>, whatever the link rate.
  10 Gbit/s; each further doubling of link speed costs one more
* '''At the bottom: the capacity floor.''' The rate never sits more than 32&times; below a measured bottleneck, and the additive slope refills <code>C/32</code> a second, so the worst hole is bounded: floor to full capacity in tens of seconds, a halving in seconds.
  ~14 ms doubling time). On an uncongested path nothing marks, so
  slow start runs until the aggregate arrives at its bottleneck.
* '''After a cut: the probe.''' A decrease is at most a halving per
  step, and the mark ages out on the rate-relative horizon once
  congestion clears (Section 5.4); the proportional probe then heals
  a halving in <code>T_probe &middot; ln 2 &asymp; 5.5 s</code> at any
  link rate.
* '''At the bottom: the capacity floor.''' The rate sits at most a
  factor 32 below a measured bottleneck, and the additive slope
  refills <code>C/32</code> per second, so the deepest hole is bounded:
  floor to full capacity in tens of seconds, a halving in seconds.


These compose because a deep cut requires sustained congestion,
These fit together because a deep cut needs sustained congestion,
sustained congestion backlogs the bottleneck queue, and a backlogged
sustained congestion backs up the bottleneck queue, and a backed-up
queue is exactly when its capacity is measured and stamped
queue is exactly when its capacity gets measured and stamped (see
(Section 2): whenever the controller is cut deep, the scaled floor is
[[#Forwarder marking|forwarder marking]]). So whenever the controller
live. The default floor only governs paths that never congested the
has been cut deep, the scaled floor is already live. The default floor
sender &mdash; where there is nothing to recover from. This is why
only ever applies to paths that never congested the sender in the first
slow start can end for good: the floor and the probe cover every
place, where there's nothing to recover from. That's why slow start can
later recovery.
quit for good: the floor and the probe handle every recovery after it.


=== 5.7. Regimes ===
=== Regimes ===


<pre>
<pre>
Line 384: Line 352:
   | r *= e^(t/T)|                      |  avoidance    |
   | r *= e^(t/T)|                      |  avoidance    |
   +-------------+                      |                |
   +-------------+                      |                |
                                         | m == 0 -> AI  |
                                         | m == 0 ---> additive increase
                                         | m  > 0 -> AIPD |
                                         | m  > 0 ---> AI + PD decrease
                                         +----------------+
                                         +----------------+
     m = fed-back ece, or the local first-hop mark when ece == 0
     m = fed-back ece, or the local first-hop mark when ece == 0
</pre>
</pre>


== 6. RTT behaviour ==


Because every increase and decrease is scaled by wall-clock time,
== RTT behaviour ==
two flows of different RTT that share a bottleneck obey the ''same''
 
rate law and converge to the ''same'' rate. The steady-state
Because every increase and decrease is scaled by wall-clock time, two
flows with different RTTs sharing a bottleneck follow the ''same'' rate
law and settle at the ''same'' rate. So the steady-state
'''allocation is RTT-independent'''.
'''allocation is RTT-independent'''.


This is a property of the equilibrium, not of the dynamics. The
That is a property of the equilibrium, not the dynamics. The feedback
feedback delay in the loop ''is'' the path RTT <code>&tau;</code>, and the
delay in the loop ''is'' the path RTT <code>&tau;</code>, and the
controller does not measure or compensate for it. At low rate the
controller neither measures nor compensates for it. At low rate the
receiver window (up to its ~4.3 s ceiling) is the dominant lag &mdash;
receiver window (up to its ~4.3 s ceiling) is the biggest lag: the loop
the loop is slow there in proportion to how slow the flow is, exactly
is slow there in proportion to how slow the flow is, just like a
as a long-RTT TCP is &mdash; while as the rate rises the window shrinks
long-RTT TCP. As the rate climbs the window shrinks toward its ~1 ms
toward its ~1 ms floor, so the window's contribution to the loop delay
floor, so its share of the loop delay ''falls'' with capacity and the
''falls'' with capacity and the path RTT <code>&tau;</code> becomes the
path RTT <code>&tau;</code> becomes the irreducible part. Two things keep
irreducible term. Two mechanisms keep the loop damped: the adaptive
the loop damped: the adaptive window takes out the delay-dominated
window removes the delay-dominated corner at low <code>&tau;</code>, and a
corner at low <code>&tau;</code>, and a bucketed
bucketed '''rate-velocity damping''' (a washout: each ~31 ms bucket
'''rate-velocity damping''' (the washout: each ~31 ms bucket pulls the
pulls the rate a quarter of the way back toward its value at the
rate a quarter of the way back toward its value at the previous bucket)
previous bucket) supplies the damping the otherwise
supplies the damping this otherwise near-double-integrator loop lacks,
near-double-integrator lacks, which is what holds the loop together at
which is what holds it together at high capacity and moderate
high capacity and moderate <code>&tau;</code>. What remains is the loop gain
<code>&tau;</code>. What is left is the loop gain
<code>&radic;G &prop; &radic;C</code>: at very high capacity ''and'' large
<code>&radic;G &prop; &radic;C</code>: at very high capacity ''and''
<code>&tau;</code> the <code>&radic;G &middot; &tau;</code> phase budget is the
large <code>&tau;</code> the <code>&radic;G &middot; &tau;</code> phase
binding constraint, and no window or damping trick removes it (see
budget is the binding constraint, and no amount of windowing or damping
[[#8. Limitations|Section 8]]). A precise statement is therefore:
removes it (see [[#Limitations|Limitations]]). So, precisely:
'''RTT-fair in equilibrium, and dynamically stable up to a
'''RTT-fair in equilibrium, and dynamically stable up to a'''
capacity-dependent RTT limit set by <code>&radic;G &middot; &tau;</code>.'''
'''capacity-dependent RTT limit set by <code>&radic;G &middot; &tau;</code>.'''




== 7. Fairness ==
== Fairness ==


At equilibrium the two increase terms balance the proportional
At equilibrium the two increase terms balance the proportional
decrease, <code>a + r/T_probe = r &middot; m*/M</code>, which pins the
decrease, <code>a + r/T_probe = r &middot; m*/M</code>, which pins the
congestion level to <code>m* = M &middot; (a/r + 1/T_probe)</code> and, since
congestion level at <code>m* = M &middot; (a/r + 1/T_probe)</code>. Since
<code>ece = 8q</code> (one mark unit per <code>Q = 4</code> packets, 32 fixed-point
<code>ece = 8q</code> (one mark unit per <code>Q = 4</code> packets, 32
steps per unit), a standing queue of
fixed-point steps per unit), that is a standing queue of


   q* = 64 * a / r  +  64 / T_probe        (packets, r in bytes/s)
   q* = 64 * a / r  +  64 / T_probe        (packets, r in bytes/s)


The additive part (<code>64 &middot; a / r</code>) vanishes with rate under
The additive part (<code>64 &middot; a / r</code>) fades to nothing with
the default slope; with the capacity-derived slope (<code>a = C/32</code>,
rate under the default slope; with the capacity-derived slope
Section 5.5) it becomes <code>2 C / r</code> &mdash; about two packets per
(<code>a = C/32</code>; see
competing flow, rate-independent. The proportional-probe part
[[#Capacity-derived floor and slope|the capacity floor]]) it settles at
(<code>64 / T_probe</code>) is a rate-independent floor (~8 packets at
<code>2 C / r</code>, about two packets per competing flow, whatever the
8 s). That floor is the cost of rate-independent convergence, and it
rate. The probe part (<code>64 / T_probe</code>) is a floor that does
keeps <code>q*</code> above the marking threshold at high rate.
not move with rate (~8 packets at 8 s). That floor is what
rate-independent convergence costs you, and it is what keeps
<code>q*</code> above the marking threshold at high rate.


Two flows at the same bottleneck see the same <code>m*</code>, so each
Two flows at the same bottleneck see the same <code>m*</code>, so each
solves <code>r_i = a / (m*/M - 1/T_probe)</code> to the same rate. Under
solves <code>r_i = a / (m*/M - 1/T_probe)</code> to the same rate. In
the network-utility-maximization view this corresponds to
network-utility-maximization terms that is '''proportional fairness'''
'''proportional fairness''' (equal weights), which is max-min fair at
with equal weights, which at a single bottleneck is just max-min fair.
a single bottleneck.
 
Getting a good measurement at low rate is the estimator's job, not the
floor's: the averaging window stretches with the flow (see
[[#Receiver estimate|the receiver estimate]]), so a CA-limited flow
still keeps ~16 packets a window down to ~30 kbit/s, and the
rate-relative gap threshold keeps its normal spacing from reading as
idle. The default floor (<code>2^13 B/s</code>) only bounds the extremes
(window ceiling, staleness horizon, pacer arithmetic). The floor itself
scales with the fed-back path capacity (<code>r_min = C/32</code>; see
[[#Capacity-derived floor and slope|the capacity floor]]), so about 32
CA-limited flows fit above it on any class of link. Flows whose
bottlenecks are in different classes get different slopes, so fairness
between them is capacity-weighted, not equal; see
[[#Limitations|Limitations]].
 
 
== Limitations ==


Measurement fidelity at low rate comes from the estimator itself:
Things the algorithm '''doesn't''' do. Listed here so nobody credits it
the averaging window stretches with the flow
with more than it manages.
(Section 3), so a CA-limited flow keeps ~16 packets per window at any
rate down to ~30 kbit/s, and the rate-relative gap threshold keeps its
inter-packet spacing from reading as idle. The default floor
(<code>2^13</code> B/s) only bounds the extremes (window ceiling,
staleness horizon, pacer arithmetic). The floor scales with the
fed-back path capacity (<code>r_min = C/32</code>, Section 5.5), so
roughly 32 CA-limited flows fit above it at any link class. Flows
whose paths differ in bottleneck class derive different slopes, so
fairness across them is capacity-weighted; see
[[#8. Limitations|Section 8]].


* '''Queue-only signal, no rate term.''' <code>M</code> is a queue level, so mb-ecn is a standing-queue controller: it has to build a queue (<code>q* = 64 &middot; a / r + 64 / T_probe</code>) to get any signal at all, and can't sit at full utilisation on an empty queue the way a controller with a rate estimate can. The probe floor (~8 packets) keeps <code>q*</code> above the marking threshold at every rate, but you pay for it with a permanent standing queue.
* '''Aggregation is per destination, per QoS cube.''' The aggregate is a stand-in for the path: flows to the same destination are assumed to share a bottleneck, and flows to different destinations run separate loops even when they pile up at the same hop. Each loop still converges to its own fair share (see [[#Fairness|fairness]]), so the cost is duplicated state and probing at that shared queue, not unfairness. One day we could tag the queue in the packet: if a forwarder stamped a queue id next to its mark, senders could key their aggregates on the real bottleneck instead of the destination, and pool the estimate and the probing across everything sharing it.
* '''Capacity is learned, and there are gaps.''' The floor and slope ride on an in-band path-capacity estimate (see [[#Forwarder marking|forwarder marking]] and [[#Capacity-derived floor and slope|the capacity floor]]), and that estimate has its own limits. A hop only learns its rate while its queue is backed up, so a rerouted or brand-new bottleneck reads ''unknown'' at first and the path MIN can briefly come from faster hops (the sender's smoothing and the next feedback bound the overshoot). The measurement is order-of-magnitude on purpose: quarter-log2 code (~19% a step), burst noise held down by a 16-packet minimum window and a max filter, and nothing learned below ~120 packets a second of busy drain (16 packets inside the ~134 ms staleness cap; at MTU that's ~1.4 Mbit/s, below which capacity/32 sits under the default floor anyway). The derived floor clamps to <code>[2^13, 2^32]</code> B/s, and flows with bottlenecks in different classes get capacity-weighted, not equal, shares at a common queue (see [[#Fairness|fairness]]).
* '''The low end trades speed, not correctness.''' Below ~30 kbit/s the averaging window hits its ~4.3 s ceiling and holds fewer samples; below ~64 kbit/s of offered load an ''app-limited'' flow spaces its packets past the gap threshold and rides the raw onset branch, as it always did. At the floor, feedback and the staleness horizon are both measured in seconds, so a slow flow converges slowly, TCP-style. Nothing blows up; nothing's quick either.
* '''No loss response.''' The controller only reacts to ECN marks. Loss is FRCP's problem, handled by retransmission, out of the CA's sight. At a bottleneck that drops instead of marking (say a plain drop-tail queue at the Ethernet or UDP shim), mb-ecn sees nothing and additive increase just keeps ramping. It's built for Ouroboros-native bottlenecks where the forwarder marks; there's no classic-bottleneck fallback.
* '''Dynamic stability has a capacity-dependent RTT limit.''' As covered under [[#RTT behaviour|RTT behaviour]], the allocation is RTT-independent but the dynamics aren't: the binding constraint is <code>&radic;G &middot; &tau;</code> with <code>&radic;G &prop; &radic;C</code>. The adaptive window and the washout push that limit out a long way (a single flow at 10 GbE stays stable to ~50 ms RTT in the fluid model), but the far corner (100 GbE at &ge; 50 ms) is out of reach for a window/damping controller and would need explicit-rate signalling.


== 9. Architectural fit ==
== Architectural fit ==


Each property the preceding sections build toward attaches to a
Each of these properties falls out of something structural in the
structural feature of the recursive architecture. This section makes
recursive architecture. Here's the mapping.
the mapping explicit.


* '''Congestion avoidance is fully orthogonal to ARQ and to flow
* '''CA is fully separate from ARQ and flow control.''' Three concerns, three mechanisms, two scopes: FRCP does retransmission (ARQ) and flow control (the peer pacing the sender), end-to-end and per flow; CA runs in the IPCP, per aggregate. Each signal says one thing. A loss just means retransmit: a lossy link reads as lossy, and the congestion call is left to the marks (the [[#Limitations|limitations]] section's "no loss response" is this same separation from the other side). The peer's flow-control window just paces the endpoint: back-pressure from a slow receiver stays separate from congestion in the network. A retransmitted packet is ordinary traffic as far as the pacer cares. TCP jams all three into one window machine, where the receive window caps the congestion window and loss is both the reliability trigger and the congestion signal; here you can reason about, test, and change each one on its own.
  control.''' Three concerns, three mechanisms, two scopes: FRCP
* '''CA sits below what the application picks.''' Every flow in the layer runs under the same rate law, whatever its QoS: a greedy raw sender shares a bottleneck fairly with a reliable stream because the control belongs to the layer, on the aggregate, not to the endpoint transport's good manners.
  provides retransmission (ARQ) and flow control (the peer pacing the
* '''A layer owns its PCI, so the signal can be rich.''' Everyone in a layer enrolled into it, so the layer is one administrative domain by construction, at whatever scope it covers, and the header is the layer's own. That's why forwarders can write a multi-bit queue magnitude and a capacity byte straight into the packet. The datacenter schemes in [[#Heritage and positioning|heritage and positioning]] need exactly that kind of domain and only get it inside one operator's fabric; a recursive layer has it everywhere.
  sender) end-to-end, per flow; congestion avoidance runs in the
* '''Layer-internal addressing makes the aggregate well-defined.''' Flows in a layer run between layer addresses, so the (destination, QoS cube) aggregate, which is RFC 3124's macroflow, drops straight out of the naming. Controller state scales with the number of destinations, and each QoS cube keeps its own loop, so service classes don't share a fate.
  IPCP, per aggregate. Each signal means one thing. A loss triggers
* '''Flows are allocated, so feedback already has a channel.''' Every flow has state at both ends and a reverse direction, so the receiver's <code>ece</code> and the path capacity just ride the flow allocator's existing exchange.
  a retransmission and nothing else &mdash; a lossy link reads as
* '''There's no transport ack clock, and nothing leans on one.''' The layer has to handle raw flows, which carry no acknowledgements at all, so the controller paces on wall-clock time, and that constraint is exactly what buys the RTT-independent allocation under [[#RTT behaviour|RTT behaviour]]. A restriction turned into the feature.
  lossy, with the congestion verdict left to the marks (Section 8's
* '''Recursion scales it.''' Each layer handles congestion over its own scope and timescale; many N-flows ride one N-1 flow, so aggregation compounds down the stack. The capacity a layer sees at its egress queue is whatever the layer below is actually giving it (a paced, shared lower flow, not some nominal wire speed), so the signal means something at every level and back-pressure cascades down layer by layer.
  "no loss response" is this orthogonality stated from the other
  side). The peer's flow-control window paces the endpoint and
  nothing else &mdash; back-pressure from a slow receiver stays
  distinct from congestion in the network. A retransmitted packet is
  ordinary traffic to the pacer. TCP folds all three into one window
  machine, where the receive window bounds the congestion window and
  loss serves as both reliability trigger and congestion signal;
  here each mechanism can be reasoned about, tested and evolved
  alone.
* '''Congestion avoidance sits below application choice.''' Every
  flow in the layer is paced by the same rate law, whatever its QoS:
  a greedy raw sender shares a bottleneck fairly with a reliable
  stream because the control is a property of the layer, on the
  aggregate, rather than a courtesy of the endpoint transport.
* '''A layer defines its own PCI, so the signal can be rich.'''
  Every member of a layer is enrolled into it: the layer is a single
  administrative domain by construction, at whatever scope it spans,
  and its header is layer-internal. Forwarders therefore write a
  multi-bit queue magnitude and a capacity byte directly into the
  packet. The datacenter schemes of
  [[#10. Heritage and positioning|Section 10]] require exactly such a
  domain and find it only inside one operator's fabric; a recursive
  layer supplies it everywhere.
* '''Layer-internal addressing makes the aggregate well-defined.'''
  Flows in a layer run between layer addresses, so the
  (destination, QoS cube) aggregate &mdash; RFC 3124's macroflow
  &mdash; falls out of the naming structure. Controller state scales
  with destinations, and each QoS cube keeps its own loop, so
  service classes keep separate fates.
* '''Flows are allocated, so feedback has a channel.''' Every flow
  has state at both ends and a reverse direction; the receiver's
  <code>ece</code> and the path capacity ride the flow allocator's
  existing exchange.
* '''There is no transport ack clock, and none is relied on.''' The
  layer must control raw flows, which carry no acknowledgements at
  all, so the controller paces on wall-clock time &mdash; and that
  constraint yields the RTT-independent allocation of Section 6. An
  architectural restriction becomes the controller's distinguishing
  property.
* '''Recursion scales the scheme.''' Each layer controls congestion
  over its own scope and timescale; many N-flows ride one N-1 flow,
  so aggregation compounds down the stack. The capacity a layer
  measures at its egress queue is the rate the layer below actually
  provides &mdash; a paced, shared lower flow rather than a nominal
  wire speed &mdash; so the signal stays meaningful at every level,
  and pushback cascades layer by layer.


The fit has a boundary: at a shim over legacy media the layer below
There's a boundary: at a shim over legacy media the layer below neither
neither enrolls nor marks, which is the deployment edge behind
enrolls nor marks, which is the deployment edge behind the
Section 8's no-loss-response limitation.
[[#Limitations|limitations]] section's no-loss-response point.




== 10. Heritage and positioning ==
== Heritage and positioning ==


mb-ecn is best understood as a rate-based ECN controller in the DECbit /
mb-ecn is basically a rate-based ECN controller in the DECbit / QCN
QCN line, adapted to an acknowledgement-less recursive layer.
line, adapted to a recursive layer with no acknowledgements.


The core of the scheme dates to the first Ouroboros
The core goes back to the first Ouroboros CA policy (2020): a multi-bit
congestion-avoidance policy (2020): a multi-bit queue-depth mark
queue-depth mark relayed by forwarders to the receiver, a smoothed
relayed by forwarders to the receiver, a smoothed multi-bit
multi-bit <code>ece</code> fed back on the reverse flow at the same 1/32
<code>ece</code> fed back on the reverse flow at the same 1/32 fixed
fixed point used today, and a sender that paced packets from a time
point used today, and a sender that paced packets from a time budget
budget and ran slow start, additive increase and multiplicative
and ran slow start, additive increase and multiplicative decrease on
decrease on wall-clock time slots. Rate-based and acknowledgement-free
wall-clock time slots &mdash; rate-based and acknowledgement-free from
from the start, with RTT-independent allocation already the goal. The
the start, with RTT-independent allocation already an objective. The
present algorithm keeps that skeleton and reworks every estimator
present algorithm keeps that skeleton and reworks every estimator
around it: per-path aggregation with SFQ pacing, the rate-adaptive
around it: per-path aggregation with SFQ pacing, the rate-adaptive
boxcar receiver, elapsed-time-scaled AIMD with a derivative term and
box-car receiver, elapsed-time-scaled AIMD with a derivative term and
rate-velocity damping, and the in-band capacity signal.
rate-velocity damping, and the in-band capacity signal.


* '''DECbit''' (Ramakrishnan &amp; Jain, 1988; ToCS 1990) established
* '''DECbit''' (Ramakrishnan &amp; Jain, 1988; ToCS 1990) established binary congestion feedback set by the switch on queue occupancy. mb-ecn keeps the switch-sets-on-queue idea and carries a ''magnitude''.
  binary congestion feedback set by the switch on queue occupancy.
* '''The Congestion Manager''' (Balakrishnan et al., SIGCOMM 1999; RFC 3124, 2001) aggregates congestion state at the end host: one controller per macroflow, a scheduler apportioning its rate across the member streams, and new streams joining at the ensemble's current state instead of probing from scratch. mb-ecn's (destination, QoS cube) aggregate is the same idea placed inside the IPCP: every layer manages its own macroflows, recursively, the SFQ pacer plays the CM scheduler's role (see [[#Sender pacer|the pacer]]), and a joining flow rides the aggregate's estimates (see [[#Slow start|slow start]]). The CM gathers its signal from transport feedback at the edge; mb-ecn reads it off the wire from the forwarders.
  mb-ecn keeps the switch-sets-on-queue idea and carries a
* '''DCTCP''' (Alizadeh et al., SIGCOMM 2010) recovers a queue magnitude at the endpoint by averaging a single threshold bit. mb-ecn instead reads the magnitude off the wire, then time-averages it at the receiver. The additive-increase / proportional-decrease structure is shared.
  ''magnitude''.
* '''QCN''' (IEEE 802.1Qau, 2010) and '''DCQCN''' (Zhu et al., SIGCOMM 2015) contribute the rate-based (not window-based) ECN control skeleton. mb-ecn's proportional decrease plus a probing increase mirrors that skeleton; the increase here is wall-clock scaled rather than byte-counter/timer based.
* '''The Congestion Manager''' (Balakrishnan et al., SIGCOMM 1999;
* '''HPCC''' (Li et al., SIGCOMM 2019) and '''PowerTCP''' (Addanki et al., NSDI 2022) combine a queue term ''and'' a rate term (via in-band telemetry or delay). mb-ecn now carries a coarse in-band rate (the quarter-log2 path MIN; see [[#Forwarder marking|forwarder marking]]) but uses it only to scale the controller's gains; the congestion ''signal'' remains queue-only, which is still the main functional difference (see [[#Limitations|Limitations]]).
  RFC 3124, 2001) aggregates congestion state at the end host: one
* '''L4S / TCP Prague''' (RFC 9330&ndash;9332, 2023) make RTT-independence a requirement and use a high-frequency single-bit signal. mb-ecn shares the RTT-independent ''allocation'' goal but reaches it by wall-clock-scaled rate control, with a low-frequency multi-bit signal.
  controller per macroflow, a scheduler apportioning its rate across
  the member streams, and new streams joining at the ensemble's
  current state instead of probing from scratch. mb-ecn's
  (destination, QoS cube) aggregate is the same idea placed inside
  the IPCP: every layer manages its own macroflows, recursively, the
  SFQ pacer plays the CM scheduler's role (Section 4), and a joining
  flow rides the aggregate's estimates (Section 5.1). The CM gathers
  its signal from transport feedback at the edge; mb-ecn reads it
  off the wire from the forwarders.
* '''DCTCP''' (Alizadeh et al., SIGCOMM 2010) recovers a queue
  magnitude at the endpoint by averaging a single threshold bit. mb-ecn
  instead reads the magnitude off the wire, then time-averages it at
  the receiver. The additive-increase / proportional-decrease structure
  is shared.
* '''QCN''' (IEEE 802.1Qau, 2010) and '''DCQCN''' (Zhu et al., SIGCOMM
  2015) contribute the rate-based (not window-based) ECN control
  skeleton. mb-ecn's proportional decrease plus a probing increase
  mirrors that skeleton; the increase here is wall-clock scaled rather
  than byte-counter/timer based.
* '''HPCC''' (Li et al., SIGCOMM 2019) and '''PowerTCP''' (Addanki et
  al., NSDI 2022) combine a queue term ''and'' a rate term (via in-band
  telemetry or delay). mb-ecn now carries a coarse in-band rate (the
  quarter-log2 path MIN, Section 2) but uses it only to scale the
  controller's gains; the congestion ''signal'' remains queue-only,
  which is still the main functional difference (Section 8).
* '''L4S / TCP Prague''' (RFC 9330&ndash;9332, 2023) make
  RTT-independence a requirement and use a high-frequency single-bit
  signal. mb-ecn shares the RTT-independent ''allocation'' goal but
  reaches it by wall-clock-scaled rate control, with a low-frequency
  multi-bit signal.


The novel combination is the multi-bit magnitude on the wire fused with
What's new is the pairing: a multi-bit magnitude on the wire with a
a wall-clock-scaled rate pacer in a setting with no per-flow ack clock.
wall-clock-scaled rate pacer, in a setting with no per-flow ack clock.
The trade-off, made explicit above, is a queue-only signal and a
The trade-off, spelled out above, is a queue-only signal and a
reactivity capped by the averaging window.
reactivity capped by the averaging window.




== 11. Parameters ==
== Parameters ==


These are the values every figure in this document instantiates. They
These are the values behind every figure in this document. They're
are '''coupled''': the mechanisms compose only while certain relations
'''coupled''': the mechanisms only fit together while a few relations
hold &mdash; the gap-restart horizon at the rate floor must exceed a
hold. The gap-restart horizon at the rate floor has to exceed a
packet's service time (or a floor-rate flow restart-loops at onset),
packet's service time (or a floor-rate flow loops at onset); a
a floor-rate flow's target window must fit under the window ceiling,
floor-rate flow's target window has to fit under the window ceiling;
the control interval must sit under the damping bucket under the idle
the control interval has to sit under the damping bucket, which sits
credit cap, the rate floor must not exceed the slow-start seed, and
under the idle credit cap; the rate floor mustn't exceed the slow-start
the derived-floor ceiling must stay under the rate ceiling. Retune
seed; and the derived-floor ceiling has to stay under the rate ceiling.
them as a set, against those invariants.
Retune them as a set, against those constraints.


{| class="wikitable"
{| class="wikitable"
Line 661: Line 566:




== 12. References ==
== References ==


=== 12.1. Source ===
=== Source ===


The implementation follows the algorithm above:
The implementation follows the algorithm above:


* <code>src/ipcpd/unicast/ca/mb-ecn.c</code>, <code>mb-ecn.h</code> &mdash;
* <code>src/ipcpd/unicast/ca/mb-ecn.c</code>, <code>mb-ecn.h</code>: receiver estimator, pacer and rate control.
  receiver estimator, pacer and rate control.
* <code>src/ipcpd/unicast/ca/ops.h</code>: CA policy interface.
* <code>src/ipcpd/unicast/ca/ops.h</code> &mdash; CA policy interface.
* <code>src/ipcpd/unicast/cap.c</code>, <code>cap.h</code>: link-capacity estimator and quarter-log2 codec.
* <code>src/ipcpd/unicast/cap.c</code>, <code>cap.h</code> &mdash; link-capacity
* <code>src/ipcpd/unicast/dt.c</code>, <code>fa.c</code>: per-hop marking and stamping, and the feedback carrying <code>ece</code> + <code>cap8</code>.
  estimator and quarter-log2 codec.
* <code>src/ipcpd/unicast/psched.c</code>: consumes the pacing deadline.
* <code>src/ipcpd/unicast/dt.c</code>, <code>fa.c</code> &mdash; per-hop marking
  and stamping, and the feedback carrying <code>ece</code> + <code>cap8</code>.
* <code>src/ipcpd/unicast/psched.c</code> &mdash; consumes the pacing deadline.


=== 12.2. Literature ===
=== Literature ===


* K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion
* K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990.
  Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990.
* H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion Management Architecture for Internet Hosts", SIGCOMM 1999; H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001.
* H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion
  Management Architecture for Internet Hosts", SIGCOMM 1999;
  H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001.
* M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010.
* M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010.
* Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments"
* Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" (DCQCN), SIGCOMM 2015.
  (DCQCN), SIGCOMM 2015.
* R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015.
* R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015.
* Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019.
* Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019.
* V. Addanki et al., "PowerTCP", NSDI 2022.
* V. Addanki et al., "PowerTCP", NSDI 2022.
* K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and
* K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and Scalable Throughput (L4S)", RFC 9330/9331/9332.
  Scalable Throughput (L4S)", RFC 9330/9331/9332.
* F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; R. Srikant, "The Mathematics of Internet Congestion Control", 2004.
* F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997;
  R. Srikant, "The Mathematics of Internet Congestion Control", 2004.

Latest revision as of 22:23, 11 July 2026


mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast IPCP. Reliability, ordering and flow control are FRCP's job, end-to-end and per flow; congestion avoidance is a separate concern, and it runs in the IPCP, once per layer. Within a layer the IPCP controls aggregates: one control loop per (destination address, QoS cube), shared by every flow to that destination.

This document is about the algorithm: what it does and why. The code follows it; the reference section maps the pieces to source files. It doesn't oversell: the limitations section says plainly what it can't do.

"MUST", "SHOULD", "MAY" and friends mean what BCP 14 (RFC 2119, RFC 8174) says, but only where they appear in capitals.

Note: Co-authered by Anthropic Claude Fable 5.

Notation

Symbol Units Meaning
r B/s Paced send rate
C B/s Bottleneck capacity
S bytes Packet size
q packets Standing queue occupancy at the bottleneck
Q packets Marking quantum: packets per mark unit (4)
ecn 8-bit code Per-packet congestion mark set by a forwarder
ece 1/32 ecn Receiver-side congestion estimate: 32 × time-mean ecn
cap8 8-bit code Path capacity, quarter-log2: C ≈ 2^(cap8 / 4) B/s, 0 = unknown
W ns Receiver averaging window
a B/s^2 Additive-increase slope
M ece units Full-congestion reference level (512)
Δt ns Elapsed wall-clock time between two events
τ n/a Path round-trip time (feedback delay); the controller does not measure it

Rate is the control variable throughout.

Every concrete figure in this document (window bounds, horizons, rate thresholds) is just an instance of the parameter values in the parameters table. What the algorithm fixes is the mechanisms and the relations between parameters; retune the parameters and the figures move together.

Overview

mb-ecn is a rate-based, ECN-driven controller with three moving parts, one at each point of the loop:

        sender                                  forwarder
      +--------------------------------+       +--------------------------+
      |  SFQ virtual-clock pacer       |       |  ecn = MAX(ecn, queue/Q) |
      |  rate r, paced per packet      | data  |  (deepest queue wins)    |
      |  AIMD + PD control on r        |------>|  cap8 = MIN(cap8, link)  |
      +--------------------------------+       +--------------------------+
                 ^                                        |
                 | ece + cap8 feedback                    | marked data
                 |                                        v
      +--------------------------------+       +--------------------------+
      |  reads downstream ece,         |<------|  receiver                |
      |  updates the rate on feedback, |  ece  |  time-mean of ecn over   |
      |  scales floor/slope to cap8    |  cap8 |  an adaptive window      |
      +--------------------------------+       +--------------------------+
  1. The forwarder marks each packet with a multi-bit magnitude proportional to its standing queue, and MIN-stamps its measured outgoing-link capacity into the same PCI.
  2. The receiver turns the stream of marks into a time-averaged congestion estimate ece and feeds it back to the sender, together with the window minimum of the path-capacity codes.
  3. The sender paces its rate with a start-time fair-queuing (SFQ) virtual clock, and adjusts that rate with additive-increase / multiplicative-decrease (AIMD) driven by ece; the rate floor and the additive slope derive from the fed-back path capacity.

There's no timer thread: the controller runs when a packet goes out or feedback comes back. Every step is scaled by elapsed wall-clock time, and that is what makes the allocation independent of RTT (see RTT behaviour).

The unit of control is the aggregate: one per (destination address, QoS cube), shared by every flow to that destination at that QoS. They share one controller and one rate. A new flow joins at whatever rate the aggregate is already running; when a flow leaves, its share goes to the others. The pacer (SFQ) splits that rate fairly across the member flows, so per-flow fairness is the scheduler's doing, not the rate law's.


Forwarder marking

A forwarder marks a packet based on how many packets are already queued on the outgoing link:

 mark = queued / Q                         (Q = 4 packets)
 ecn  = MAX(ecn, mark)                     (saturating, 8-bit field)

Q is fixed for the whole layer, not tuned per hop: the marks get MAX-combined across hops, so every forwarder MUST use the same quantum or the numbers don't compare.

Two things worth noting:

  • It's a magnitude, not a bit. The mark is an integer that tracks queue depth, in steps of Q packets: a coarse read of the standing queue.
  • MAX along the path. Cross several forwarders and the packet carries the largest mark, i.e. the deepest queue it passed. Taking the MAX (not a sum) keeps the number bounded and monotone across hops.

There's a dead zone: below Q queued packets the mark is 0.

Capacity stamping

Each forwarder also measures its outgoing link's capacity and stamps it into the packet. The measurement is a busy-period drain rate: look at the queue at most once a millisecond, and keep a window open until 16 packets have drained. So the window sizes itself to the link: the once-a-millisecond cadence caps it on fast links, the 16-packet drain time on slow ones (~19 ms at 10 Mbit/s). If more than 1/8 of the arrivals hit an empty queue, throw the window out: the link wasn't saturated. The odd empty sample (a token-bucket shaper grazing zero) is fine. A saturated queue drains at the link rate, so a max filter that decays slowly (1/16 per window) creeps up to it from below. A window that started or ended on an empty queue might have drained into buffers downstream faster than the wire, so it may pull the estimate down but never up. On the wire the code is quarter-log2 (cap8 = 4 log2 C, ~19% a step, 0 = unknown), and each hop MIN-combines its own into the byte, so the packet arrives carrying the slowest hop's rate. A hop that has never backed up stamps nothing, which is fine, since only backed-up hops matter.


Receiver estimate

The receiver turns the marks into a smoothed estimate ece, averaged over a window W that tracks the incoming byte rate. It's a plain time-average over that window (a box-car, not an EWMA). Per packet, with Δt the gap since the last one:

 A += ecn * MIN(Δt, W)              dwell-weighted; one packet
                                         weighs at most one window
 B += S                                  accumulate bytes this window
 ...
 at window close (elapsed >= W, or B >= 2 * B_target with at
                  least the window floor elapsed):
     ece = 32 * A / elapsed              mean over the actual window
     W  += (B_target * elapsed / B - W) / 4    nudge toward the size
                                               that holds the target
     W   = clamp(W, ~1.05 ms, ~4.3 s)
     A   = 0 ; B = 0                     hard reset

When a window closes, the next one is nudged (a quarter-weight moving average) toward the size that would hold about 16 packets at the current byte rate, starting from ~67 ms. The target is in bytes (B_target = 16,000), so "16 packets" is only exact at ~1000-byte packets, and it takes a few closes to settle. The mean always divides by the window that actually elapsed, so a change in rate leaves the current estimate correct and only resizes the next one. Net effect: a roughly constant ~16 samples a window from about 30 kbit/s to 122 Mbit/s. Above that the window bottoms out at ~1 ms (still thousands of samples a window at 10–100 GbE); below ~30 kbit/s it hits the ~4.3 s ceiling. The averaging clock stretches with the flow, the way TCP's ACK clock stretches with the RTT. A sudden speed-up doesn't wait out a stretched window: once twice the target bytes have arrived it closes anyway (the ~1 ms floor keeps that from running away at high rate). So the ece the sender sees is a staircase: one step per window, steps getting shorter as the rate climbs. Two edge cases:

  • Onset. The first mark after an idle stretch is passed through as-is, so a queue that's just starting to build shows up without waiting out a whole window.
  • Gap. A silence longer than 4 windows resets and sends the raw sample. "Long" is measured in the flow's own windows, so a slow flow's normal spacing never reads as idle; and the dwell clamp above limits what a real pause can add to the mean before the reset fires.

Alongside the marks, the receiver keeps the smallest non-zero capacity code it saw this window and ships it with each ece (on an onset/gap restart, just the current packet's code). Then it resets, so if the path reroutes onto something faster the fed-back capacity can climb within one window.

The estimate rides back to the sender on the reverse flow.


Sender pacer

The sender paces off an SFQ virtual clock. The aggregate keeps a virtual time V; each flow keeps a finish tag F:

 V    += r * Δt                     virtual time advances by bytes served
 s     = MAX(F, V)                  a packet behind the clock starts now
 F     = s + S                      the one after it queues behind
 wait  = (s - V) / r                time until this packet may go

That wait is the flow's scheduling deadline. A packet that's behind the virtual clock goes out now (wait = 0); one that's ahead waits, and only that flow waits, nobody else. Across an idle gap longer than 50 ms the clock credits at most the flow's own owed lead plus one burst (50 ms of service, at least one packet). So a flow paced slower than a packet per 50 ms still gets its real elapsed service (capping the credit by time would starve it) while an idle flow still can't bank an unbounded burst.

V is shared by every flow in the aggregate; the finish tags are not. Since every flow measures its start tag s against the same V, a flow that just sent is now ahead of V and has to wait, while an idle one (tag at or behind V) goes right away. That is what splits the aggregate rate r fairly among them.


Rate control (AIMD + PD)

The rate law runs on packet sends, at most once a millisecond, and also when feedback arrives, so a sender with no traffic of its own still reacts. The increase terms and the proportional decrease are each scaled by elapsed time, but not by the same clock: the increases use banked time capped at 50 ms (an idle gap shouldn't buy an unbounded ramp), while the decrease uses the uncapped elapsed time (a starved sender still cuts by the right amount). The one-sided derivative (see the decrease) is not scaled by time at all; it is per-step.

Slow start

Before the first congestion signal, the rate ramps up exponentially with a 20 ms time constant:

 r += r * Δt / T_ss                  (T_ss = 20 ms)

Slow start ends the first time a congestion signal shows up, whether fed back from the receiver or seen at the sender's own first hop, and it never comes back.

Slow start belongs to the aggregate, so it happens once per (destination, QoS cube), on the first flow. A flow that shows up later, at an aggregate that's already running, rides the existing estimate: it starts at the current rate and takes its share through the pacer, without probing of its own.

Increase: additive plus proportional probe (always on)

Past slow start, two increase terms run every step, congested or not:

 r += a * Δt                         additive; a defaults to 2^16 B/s^2
 r += r * Δt / T_probe               proportional; T_probe = 8 s

The additive term is a fixed slope (bytes/s per second), sized to the path capacity (see the capacity floor). The proportional probe adds a fixed fraction of the rate per unit time (it e-folds over T_probe): that is what lets a flow recover in the same number of steps whatever the link rate; a fixed additive step is negligible next to a fast link. Together they are the upward pressure the multiplicative decrease balances at equilibrium. The probe is not free: it costs a standing queue that does not shrink with rate (see fairness).

Multiplicative decrease (proportional + derivative)

With congestion level m (the fed-back ece, or the local first-hop mark if there's no feedback yet), the cut is a proportional term plus a one-sided derivative term, a PD controller on the congestion signal:

 mark = MIN(m, M)                                   M = 512
 rise = MAX(m - m_prev, 0), capped at M             one-sided derivative
 cut  = r * rise / (4 * M)                          derivative, gain 1/4
 cut += r * mark * Δt_ms / (1000 * M)              proportional
 cut  = MIN(cut, r / 2)                             at most a halving
 r     -= cut
 m_prev = m
  • The proportional term uses the real elapsed milliseconds, so a starved sender that hasn't run the controller in a while still cuts by the right amount.
  • The derivative term (rise) only fires when the congestion level goes up between samples: one-sided, per-step. It sharpens the response right as congestion starts.
  • The cut is capped at r/2, so one step can at most halve the rate.

M = 512 = 32 × 16 pins "full congestion" at a mean ecn of 16, i.e. a mean queue of 16 × Q = 64 packets.

Staleness

Feedback comes once per receiver window, and that window stretches with the flow's byte rate (see the receiver estimate), so the staleness horizon has to stretch with it too, or a slow flow's mark would expire between feedbacks and the rate would ramp straight into a congested path. The sender mirrors the receiver: the horizon is four target windows' worth of bytes at the current rate, floored at ~268 ms so fast flows keep a fixed one; at the default rate floor it's ~8 s. Once a signal is older than that it's dropped (both the fed-back estimate and the local mark), which frees the rate to climb again after congestion clears. Every step clamps the rate to [r_min, r_max].

Capacity-derived floor and slope

The fed-back capacity code sets the controller's two rate-scale constants, per aggregate:

 target = decode(cap8) / 32              clamped to [2^13, 2^32] B/s
 r_min += (target - r_min) / 2           per feedback (moving average)
 a      = r_min

On a path that never reports a capacity, both fall back to fixed defaults (r_min = 2^13 B/s, a = 2^16 B/s^2). When the estimator is live, the fairness floor and the post-cut recovery slope scale with the bottleneck instead, and capacity kicks in wherever C/32 beats the default floor, roughly above 2 Mbit/s. A capacity older than 16 staleness horizons (~4.3 s for fast flows) reverts both to the defaults. That horizon is deliberately longer than the feedback horizon: feedback stops the instant the marks clear, which is exactly when you need the recovery slope, and the onset-fresh capacity (see the receiver estimate) re-seeds it on the first mark of the next episode anyway.

Ramp and recovery

Three mechanisms set how fast an aggregate reaches a fast link's rate, each owning one regime:

  • From scratch: slow start. The exponential ramp hits any real link rate in well under a second (~200 ms from the seed to 10 Gbit/s; every further doubling of link speed adds one ~14 ms doubling). Nothing marks on an uncongested path, so slow start just runs until the aggregate reaches its bottleneck.
  • After a cut: the probe. A cut is at most a halving, and the mark expires on the rate-relative horizon once congestion clears (see staleness); the probe then climbs back from a halving in T_probe · ln 2 ≈ 5.5 s, whatever the link rate.
  • At the bottom: the capacity floor. The rate never sits more than 32× below a measured bottleneck, and the additive slope refills C/32 a second, so the worst hole is bounded: floor to full capacity in tens of seconds, a halving in seconds.

These fit together because a deep cut needs sustained congestion, sustained congestion backs up the bottleneck queue, and a backed-up queue is exactly when its capacity gets measured and stamped (see forwarder marking). So whenever the controller has been cut deep, the scaled floor is already live. The default floor only ever applies to paths that never congested the sender in the first place, where there's nothing to recover from. That's why slow start can quit for good: the floor and the probe handle every recovery after it.

Regimes

   +-------------+   first mark seen    +----------------+
   | slow start  |--------------------->|  congestion    |
   | r *= e^(t/T)|                      |  avoidance     |
   +-------------+                      |                |
                                        | m == 0 ---> additive increase
                                        | m  > 0 ---> AI + PD decrease
                                        +----------------+
    m = fed-back ece, or the local first-hop mark when ece == 0


RTT behaviour

Because every increase and decrease is scaled by wall-clock time, two flows with different RTTs sharing a bottleneck follow the same rate law and settle at the same rate. So the steady-state allocation is RTT-independent.

That is a property of the equilibrium, not the dynamics. The feedback delay in the loop is the path RTT τ, and the controller neither measures nor compensates for it. At low rate the receiver window (up to its ~4.3 s ceiling) is the biggest lag: the loop is slow there in proportion to how slow the flow is, just like a long-RTT TCP. As the rate climbs the window shrinks toward its ~1 ms floor, so its share of the loop delay falls with capacity and the path RTT τ becomes the irreducible part. Two things keep the loop damped: the adaptive window takes out the delay-dominated corner at low τ, and a bucketed rate-velocity damping (the washout: each ~31 ms bucket pulls the rate a quarter of the way back toward its value at the previous bucket) supplies the damping this otherwise near-double-integrator loop lacks, which is what holds it together at high capacity and moderate τ. What is left is the loop gain √G ∝ √C: at very high capacity and large τ the √G · τ phase budget is the binding constraint, and no amount of windowing or damping removes it (see Limitations). So, precisely: RTT-fair in equilibrium, and dynamically stable up to a capacity-dependent RTT limit set by √G · τ.


Fairness

At equilibrium the two increase terms balance the proportional decrease, a + r/T_probe = r · m*/M, which pins the congestion level at m* = M · (a/r + 1/T_probe). Since ece = 8q (one mark unit per Q = 4 packets, 32 fixed-point steps per unit), that is a standing queue of

 q* = 64 * a / r  +  64 / T_probe        (packets, r in bytes/s)

The additive part (64 · a / r) fades to nothing with rate under the default slope; with the capacity-derived slope (a = C/32; see the capacity floor) it settles at 2 C / r, about two packets per competing flow, whatever the rate. The probe part (64 / T_probe) is a floor that does not move with rate (~8 packets at 8 s). That floor is what rate-independent convergence costs you, and it is what keeps q* above the marking threshold at high rate.

Two flows at the same bottleneck see the same m*, so each solves r_i = a / (m*/M - 1/T_probe) to the same rate. In network-utility-maximization terms that is proportional fairness with equal weights, which at a single bottleneck is just max-min fair.

Getting a good measurement at low rate is the estimator's job, not the floor's: the averaging window stretches with the flow (see the receiver estimate), so a CA-limited flow still keeps ~16 packets a window down to ~30 kbit/s, and the rate-relative gap threshold keeps its normal spacing from reading as idle. The default floor (2^13 B/s) only bounds the extremes (window ceiling, staleness horizon, pacer arithmetic). The floor itself scales with the fed-back path capacity (r_min = C/32; see the capacity floor), so about 32 CA-limited flows fit above it on any class of link. Flows whose bottlenecks are in different classes get different slopes, so fairness between them is capacity-weighted, not equal; see Limitations.


Limitations

Things the algorithm doesn't do. Listed here so nobody credits it with more than it manages.

  • Queue-only signal, no rate term. M is a queue level, so mb-ecn is a standing-queue controller: it has to build a queue (q* = 64 · a / r + 64 / T_probe) to get any signal at all, and can't sit at full utilisation on an empty queue the way a controller with a rate estimate can. The probe floor (~8 packets) keeps q* above the marking threshold at every rate, but you pay for it with a permanent standing queue.
  • Aggregation is per destination, per QoS cube. The aggregate is a stand-in for the path: flows to the same destination are assumed to share a bottleneck, and flows to different destinations run separate loops even when they pile up at the same hop. Each loop still converges to its own fair share (see fairness), so the cost is duplicated state and probing at that shared queue, not unfairness. One day we could tag the queue in the packet: if a forwarder stamped a queue id next to its mark, senders could key their aggregates on the real bottleneck instead of the destination, and pool the estimate and the probing across everything sharing it.
  • Capacity is learned, and there are gaps. The floor and slope ride on an in-band path-capacity estimate (see forwarder marking and the capacity floor), and that estimate has its own limits. A hop only learns its rate while its queue is backed up, so a rerouted or brand-new bottleneck reads unknown at first and the path MIN can briefly come from faster hops (the sender's smoothing and the next feedback bound the overshoot). The measurement is order-of-magnitude on purpose: quarter-log2 code (~19% a step), burst noise held down by a 16-packet minimum window and a max filter, and nothing learned below ~120 packets a second of busy drain (16 packets inside the ~134 ms staleness cap; at MTU that's ~1.4 Mbit/s, below which capacity/32 sits under the default floor anyway). The derived floor clamps to [2^13, 2^32] B/s, and flows with bottlenecks in different classes get capacity-weighted, not equal, shares at a common queue (see fairness).
  • The low end trades speed, not correctness. Below ~30 kbit/s the averaging window hits its ~4.3 s ceiling and holds fewer samples; below ~64 kbit/s of offered load an app-limited flow spaces its packets past the gap threshold and rides the raw onset branch, as it always did. At the floor, feedback and the staleness horizon are both measured in seconds, so a slow flow converges slowly, TCP-style. Nothing blows up; nothing's quick either.
  • No loss response. The controller only reacts to ECN marks. Loss is FRCP's problem, handled by retransmission, out of the CA's sight. At a bottleneck that drops instead of marking (say a plain drop-tail queue at the Ethernet or UDP shim), mb-ecn sees nothing and additive increase just keeps ramping. It's built for Ouroboros-native bottlenecks where the forwarder marks; there's no classic-bottleneck fallback.
  • Dynamic stability has a capacity-dependent RTT limit. As covered under RTT behaviour, the allocation is RTT-independent but the dynamics aren't: the binding constraint is √G · τ with √G ∝ √C. The adaptive window and the washout push that limit out a long way (a single flow at 10 GbE stays stable to ~50 ms RTT in the fluid model), but the far corner (100 GbE at ≥ 50 ms) is out of reach for a window/damping controller and would need explicit-rate signalling.

Architectural fit

Each of these properties falls out of something structural in the recursive architecture. Here's the mapping.

  • CA is fully separate from ARQ and flow control. Three concerns, three mechanisms, two scopes: FRCP does retransmission (ARQ) and flow control (the peer pacing the sender), end-to-end and per flow; CA runs in the IPCP, per aggregate. Each signal says one thing. A loss just means retransmit: a lossy link reads as lossy, and the congestion call is left to the marks (the limitations section's "no loss response" is this same separation from the other side). The peer's flow-control window just paces the endpoint: back-pressure from a slow receiver stays separate from congestion in the network. A retransmitted packet is ordinary traffic as far as the pacer cares. TCP jams all three into one window machine, where the receive window caps the congestion window and loss is both the reliability trigger and the congestion signal; here you can reason about, test, and change each one on its own.
  • CA sits below what the application picks. Every flow in the layer runs under the same rate law, whatever its QoS: a greedy raw sender shares a bottleneck fairly with a reliable stream because the control belongs to the layer, on the aggregate, not to the endpoint transport's good manners.
  • A layer owns its PCI, so the signal can be rich. Everyone in a layer enrolled into it, so the layer is one administrative domain by construction, at whatever scope it covers, and the header is the layer's own. That's why forwarders can write a multi-bit queue magnitude and a capacity byte straight into the packet. The datacenter schemes in heritage and positioning need exactly that kind of domain and only get it inside one operator's fabric; a recursive layer has it everywhere.
  • Layer-internal addressing makes the aggregate well-defined. Flows in a layer run between layer addresses, so the (destination, QoS cube) aggregate, which is RFC 3124's macroflow, drops straight out of the naming. Controller state scales with the number of destinations, and each QoS cube keeps its own loop, so service classes don't share a fate.
  • Flows are allocated, so feedback already has a channel. Every flow has state at both ends and a reverse direction, so the receiver's ece and the path capacity just ride the flow allocator's existing exchange.
  • There's no transport ack clock, and nothing leans on one. The layer has to handle raw flows, which carry no acknowledgements at all, so the controller paces on wall-clock time, and that constraint is exactly what buys the RTT-independent allocation under RTT behaviour. A restriction turned into the feature.
  • Recursion scales it. Each layer handles congestion over its own scope and timescale; many N-flows ride one N-1 flow, so aggregation compounds down the stack. The capacity a layer sees at its egress queue is whatever the layer below is actually giving it (a paced, shared lower flow, not some nominal wire speed), so the signal means something at every level and back-pressure cascades down layer by layer.

There's a boundary: at a shim over legacy media the layer below neither enrolls nor marks, which is the deployment edge behind the limitations section's no-loss-response point.


Heritage and positioning

mb-ecn is basically a rate-based ECN controller in the DECbit / QCN line, adapted to a recursive layer with no acknowledgements.

The core goes back to the first Ouroboros CA policy (2020): a multi-bit queue-depth mark relayed by forwarders to the receiver, a smoothed multi-bit ece fed back on the reverse flow at the same 1/32 fixed point used today, and a sender that paced packets from a time budget and ran slow start, additive increase and multiplicative decrease on wall-clock time slots. Rate-based and acknowledgement-free from the start, with RTT-independent allocation already the goal. The present algorithm keeps that skeleton and reworks every estimator around it: per-path aggregation with SFQ pacing, the rate-adaptive box-car receiver, elapsed-time-scaled AIMD with a derivative term and rate-velocity damping, and the in-band capacity signal.

  • DECbit (Ramakrishnan & Jain, 1988; ToCS 1990) established binary congestion feedback set by the switch on queue occupancy. mb-ecn keeps the switch-sets-on-queue idea and carries a magnitude.
  • The Congestion Manager (Balakrishnan et al., SIGCOMM 1999; RFC 3124, 2001) aggregates congestion state at the end host: one controller per macroflow, a scheduler apportioning its rate across the member streams, and new streams joining at the ensemble's current state instead of probing from scratch. mb-ecn's (destination, QoS cube) aggregate is the same idea placed inside the IPCP: every layer manages its own macroflows, recursively, the SFQ pacer plays the CM scheduler's role (see the pacer), and a joining flow rides the aggregate's estimates (see slow start). The CM gathers its signal from transport feedback at the edge; mb-ecn reads it off the wire from the forwarders.
  • DCTCP (Alizadeh et al., SIGCOMM 2010) recovers a queue magnitude at the endpoint by averaging a single threshold bit. mb-ecn instead reads the magnitude off the wire, then time-averages it at the receiver. The additive-increase / proportional-decrease structure is shared.
  • QCN (IEEE 802.1Qau, 2010) and DCQCN (Zhu et al., SIGCOMM 2015) contribute the rate-based (not window-based) ECN control skeleton. mb-ecn's proportional decrease plus a probing increase mirrors that skeleton; the increase here is wall-clock scaled rather than byte-counter/timer based.
  • HPCC (Li et al., SIGCOMM 2019) and PowerTCP (Addanki et al., NSDI 2022) combine a queue term and a rate term (via in-band telemetry or delay). mb-ecn now carries a coarse in-band rate (the quarter-log2 path MIN; see forwarder marking) but uses it only to scale the controller's gains; the congestion signal remains queue-only, which is still the main functional difference (see Limitations).
  • L4S / TCP Prague (RFC 9330–9332, 2023) make RTT-independence a requirement and use a high-frequency single-bit signal. mb-ecn shares the RTT-independent allocation goal but reaches it by wall-clock-scaled rate control, with a low-frequency multi-bit signal.

What's new is the pairing: a multi-bit magnitude on the wire with a wall-clock-scaled rate pacer, in a setting with no per-flow ack clock. The trade-off, spelled out above, is a queue-only signal and a reactivity capped by the averaging window.


Parameters

These are the values behind every figure in this document. They're coupled: the mechanisms only fit together while a few relations hold. The gap-restart horizon at the rate floor has to exceed a packet's service time (or a floor-rate flow loops at onset); a floor-rate flow's target window has to fit under the window ceiling; the control interval has to sit under the damping bucket, which sits under the idle credit cap; the rate floor mustn't exceed the slow-start seed; and the derived-floor ceiling has to stay under the rate ceiling. Retune them as a set, against those constraints.

Parameter Value Meaning
Marking quantum Q 4 packets Mark onset and step; layer-wide
Feedback resolution 1/32 ecn unit Fixed-point LSB of ece
Full-congestion reference M 512 Mean queue of 64 packets
Receiver window target 16 packets (16,000 bytes) Samples per window
Initial receiver window ~67 ms Before rate is known
Receiver window bounds ~1.05 ms – ~4.3 s Clamp on W
Window resize weight 1/4 Moving average toward target
Early window close 2 × target bytes Speed-up escape
Gap restart 4 windows Idle horizon (rate-relative)
Additive slope a (default) 2^16 B/s^2 Lower clamp of derived slope
Probe time constant T_probe 8 s Rate-proportional recovery
Derivative gain 1/4 Onset sharpening
Rate floor r_min (default) 2^13 B/s Lower clamp of derived floor
Slow-start seed rate 2^16 B/s Initial rate
Slow-start time constant 20 ms Exponential ramp
Rate ceiling r_max 2^37 B/s Upper clamp
Capacity scale C / 32 Floor and slope from bottleneck
Capacity smoothing 1/2 Moving average per feedback
Capacity staleness 16 × feedback horizon Revert to defaults
Derived floor ceiling 2^32 B/s Upper clamp on derived floor
Control interval 1 ms Minimum spacing of rate updates
Idle credit cap 50 ms Burst allowance / increase banking
Feedback staleness floor ~268 ms Floor of rate-relative horizon
Damping bucket ~31 ms Rate-velocity washout cadence
Damping fraction 1/4 Pullback per bucket
Estimator fold spacing 1 ms Minimum measurement cadence
Estimator window closure 16 packets drained Self-scaling window
Estimator window staleness ~134 ms Discard across traffic gaps
Estimator idle tolerance 1/8 of arrivals Unsaturated-window rejection
Estimator decay 1/16 per window Max-filter convergence from below


References

Source

The implementation follows the algorithm above:

  • src/ipcpd/unicast/ca/mb-ecn.c, mb-ecn.h: receiver estimator, pacer and rate control.
  • src/ipcpd/unicast/ca/ops.h: CA policy interface.
  • src/ipcpd/unicast/cap.c, cap.h: link-capacity estimator and quarter-log2 codec.
  • src/ipcpd/unicast/dt.c, fa.c: per-hop marking and stamping, and the feedback carrying ece + cap8.
  • src/ipcpd/unicast/psched.c: consumes the pacing deadline.

Literature

  • K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990.
  • H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion Management Architecture for Internet Hosts", SIGCOMM 1999; H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001.
  • M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010.
  • Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" (DCQCN), SIGCOMM 2015.
  • R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015.
  • Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019.
  • V. Addanki et al., "PowerTCP", NSDI 2022.
  • K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and Scalable Throughput (L4S)", RFC 9330/9331/9332.
  • F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; R. Srikant, "The Mathematics of Internet Congestion Control", 2004.