Congestion avoidance: Difference between revisions

From Ouroboros
Jump to navigation Jump to search
No edit summary
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 6: Line 6:
in the IPCP, once per layer. Within a layer the IPCP controls
in the IPCP, once per layer. Within a layer the IPCP controls
'''aggregates''': one control loop per (destination address, QoS cube),
'''aggregates''': one control loop per (destination address, QoS cube),
shared by every flow to that destination. The other policy is
shared by every flow to that destination.
<code>nop</code>, which does nothing.


This document is about the algorithm: what it does and why. The
This document is about the algorithm: what it does and why. The
Line 17: Line 16:
RFC 8174) says, but only where they appear in capitals.
RFC 8174) says, but only where they appear in capitals.


Note: Co-authered by Anthropic Claude Fable 5.


== Notation ==
== Notation ==

Latest revision as of 22:23, 11 July 2026


mb-ecn is a congestion-avoidance (CA) policy for the Ouroboros unicast IPCP. Reliability, ordering and flow control are FRCP's job, end-to-end and per flow; congestion avoidance is a separate concern, and it runs in the IPCP, once per layer. Within a layer the IPCP controls aggregates: one control loop per (destination address, QoS cube), shared by every flow to that destination.

This document is about the algorithm: what it does and why. The code follows it; the reference section maps the pieces to source files. It doesn't oversell: the limitations section says plainly what it can't do.

"MUST", "SHOULD", "MAY" and friends mean what BCP 14 (RFC 2119, RFC 8174) says, but only where they appear in capitals.

Note: Co-authered by Anthropic Claude Fable 5.

Notation

Symbol Units Meaning
r B/s Paced send rate
C B/s Bottleneck capacity
S bytes Packet size
q packets Standing queue occupancy at the bottleneck
Q packets Marking quantum: packets per mark unit (4)
ecn 8-bit code Per-packet congestion mark set by a forwarder
ece 1/32 ecn Receiver-side congestion estimate: 32 × time-mean ecn
cap8 8-bit code Path capacity, quarter-log2: C ≈ 2^(cap8 / 4) B/s, 0 = unknown
W ns Receiver averaging window
a B/s^2 Additive-increase slope
M ece units Full-congestion reference level (512)
Δt ns Elapsed wall-clock time between two events
τ n/a Path round-trip time (feedback delay); the controller does not measure it

Rate is the control variable throughout.

Every concrete figure in this document (window bounds, horizons, rate thresholds) is just an instance of the parameter values in the parameters table. What the algorithm fixes is the mechanisms and the relations between parameters; retune the parameters and the figures move together.

Overview

mb-ecn is a rate-based, ECN-driven controller with three moving parts, one at each point of the loop:

        sender                                  forwarder
      +--------------------------------+       +--------------------------+
      |  SFQ virtual-clock pacer       |       |  ecn = MAX(ecn, queue/Q) |
      |  rate r, paced per packet      | data  |  (deepest queue wins)    |
      |  AIMD + PD control on r        |------>|  cap8 = MIN(cap8, link)  |
      +--------------------------------+       +--------------------------+
                 ^                                        |
                 | ece + cap8 feedback                    | marked data
                 |                                        v
      +--------------------------------+       +--------------------------+
      |  reads downstream ece,         |<------|  receiver                |
      |  updates the rate on feedback, |  ece  |  time-mean of ecn over   |
      |  scales floor/slope to cap8    |  cap8 |  an adaptive window      |
      +--------------------------------+       +--------------------------+
  1. The forwarder marks each packet with a multi-bit magnitude proportional to its standing queue, and MIN-stamps its measured outgoing-link capacity into the same PCI.
  2. The receiver turns the stream of marks into a time-averaged congestion estimate ece and feeds it back to the sender, together with the window minimum of the path-capacity codes.
  3. The sender paces its rate with a start-time fair-queuing (SFQ) virtual clock, and adjusts that rate with additive-increase / multiplicative-decrease (AIMD) driven by ece; the rate floor and the additive slope derive from the fed-back path capacity.

There's no timer thread: the controller runs when a packet goes out or feedback comes back. Every step is scaled by elapsed wall-clock time, and that is what makes the allocation independent of RTT (see RTT behaviour).

The unit of control is the aggregate: one per (destination address, QoS cube), shared by every flow to that destination at that QoS. They share one controller and one rate. A new flow joins at whatever rate the aggregate is already running; when a flow leaves, its share goes to the others. The pacer (SFQ) splits that rate fairly across the member flows, so per-flow fairness is the scheduler's doing, not the rate law's.


Forwarder marking

A forwarder marks a packet based on how many packets are already queued on the outgoing link:

 mark = queued / Q                         (Q = 4 packets)
 ecn  = MAX(ecn, mark)                     (saturating, 8-bit field)

Q is fixed for the whole layer, not tuned per hop: the marks get MAX-combined across hops, so every forwarder MUST use the same quantum or the numbers don't compare.

Two things worth noting:

  • It's a magnitude, not a bit. The mark is an integer that tracks queue depth, in steps of Q packets: a coarse read of the standing queue.
  • MAX along the path. Cross several forwarders and the packet carries the largest mark, i.e. the deepest queue it passed. Taking the MAX (not a sum) keeps the number bounded and monotone across hops.

There's a dead zone: below Q queued packets the mark is 0.

Capacity stamping

Each forwarder also measures its outgoing link's capacity and stamps it into the packet. The measurement is a busy-period drain rate: look at the queue at most once a millisecond, and keep a window open until 16 packets have drained. So the window sizes itself to the link: the once-a-millisecond cadence caps it on fast links, the 16-packet drain time on slow ones (~19 ms at 10 Mbit/s). If more than 1/8 of the arrivals hit an empty queue, throw the window out: the link wasn't saturated. The odd empty sample (a token-bucket shaper grazing zero) is fine. A saturated queue drains at the link rate, so a max filter that decays slowly (1/16 per window) creeps up to it from below. A window that started or ended on an empty queue might have drained into buffers downstream faster than the wire, so it may pull the estimate down but never up. On the wire the code is quarter-log2 (cap8 = 4 log2 C, ~19% a step, 0 = unknown), and each hop MIN-combines its own into the byte, so the packet arrives carrying the slowest hop's rate. A hop that has never backed up stamps nothing, which is fine, since only backed-up hops matter.


Receiver estimate

The receiver turns the marks into a smoothed estimate ece, averaged over a window W that tracks the incoming byte rate. It's a plain time-average over that window (a box-car, not an EWMA). Per packet, with Δt the gap since the last one:

 A += ecn * MIN(Δt, W)              dwell-weighted; one packet
                                         weighs at most one window
 B += S                                  accumulate bytes this window
 ...
 at window close (elapsed >= W, or B >= 2 * B_target with at
                  least the window floor elapsed):
     ece = 32 * A / elapsed              mean over the actual window
     W  += (B_target * elapsed / B - W) / 4    nudge toward the size
                                               that holds the target
     W   = clamp(W, ~1.05 ms, ~4.3 s)
     A   = 0 ; B = 0                     hard reset

When a window closes, the next one is nudged (a quarter-weight moving average) toward the size that would hold about 16 packets at the current byte rate, starting from ~67 ms. The target is in bytes (B_target = 16,000), so "16 packets" is only exact at ~1000-byte packets, and it takes a few closes to settle. The mean always divides by the window that actually elapsed, so a change in rate leaves the current estimate correct and only resizes the next one. Net effect: a roughly constant ~16 samples a window from about 30 kbit/s to 122 Mbit/s. Above that the window bottoms out at ~1 ms (still thousands of samples a window at 10–100 GbE); below ~30 kbit/s it hits the ~4.3 s ceiling. The averaging clock stretches with the flow, the way TCP's ACK clock stretches with the RTT. A sudden speed-up doesn't wait out a stretched window: once twice the target bytes have arrived it closes anyway (the ~1 ms floor keeps that from running away at high rate). So the ece the sender sees is a staircase: one step per window, steps getting shorter as the rate climbs. Two edge cases:

  • Onset. The first mark after an idle stretch is passed through as-is, so a queue that's just starting to build shows up without waiting out a whole window.
  • Gap. A silence longer than 4 windows resets and sends the raw sample. "Long" is measured in the flow's own windows, so a slow flow's normal spacing never reads as idle; and the dwell clamp above limits what a real pause can add to the mean before the reset fires.

Alongside the marks, the receiver keeps the smallest non-zero capacity code it saw this window and ships it with each ece (on an onset/gap restart, just the current packet's code). Then it resets, so if the path reroutes onto something faster the fed-back capacity can climb within one window.

The estimate rides back to the sender on the reverse flow.


Sender pacer

The sender paces off an SFQ virtual clock. The aggregate keeps a virtual time V; each flow keeps a finish tag F:

 V    += r * Δt                     virtual time advances by bytes served
 s     = MAX(F, V)                  a packet behind the clock starts now
 F     = s + S                      the one after it queues behind
 wait  = (s - V) / r                time until this packet may go

That wait is the flow's scheduling deadline. A packet that's behind the virtual clock goes out now (wait = 0); one that's ahead waits, and only that flow waits, nobody else. Across an idle gap longer than 50 ms the clock credits at most the flow's own owed lead plus one burst (50 ms of service, at least one packet). So a flow paced slower than a packet per 50 ms still gets its real elapsed service (capping the credit by time would starve it) while an idle flow still can't bank an unbounded burst.

V is shared by every flow in the aggregate; the finish tags are not. Since every flow measures its start tag s against the same V, a flow that just sent is now ahead of V and has to wait, while an idle one (tag at or behind V) goes right away. That is what splits the aggregate rate r fairly among them.


Rate control (AIMD + PD)

The rate law runs on packet sends, at most once a millisecond, and also when feedback arrives, so a sender with no traffic of its own still reacts. The increase terms and the proportional decrease are each scaled by elapsed time, but not by the same clock: the increases use banked time capped at 50 ms (an idle gap shouldn't buy an unbounded ramp), while the decrease uses the uncapped elapsed time (a starved sender still cuts by the right amount). The one-sided derivative (see the decrease) is not scaled by time at all; it is per-step.

Slow start

Before the first congestion signal, the rate ramps up exponentially with a 20 ms time constant:

 r += r * Δt / T_ss                  (T_ss = 20 ms)

Slow start ends the first time a congestion signal shows up, whether fed back from the receiver or seen at the sender's own first hop, and it never comes back.

Slow start belongs to the aggregate, so it happens once per (destination, QoS cube), on the first flow. A flow that shows up later, at an aggregate that's already running, rides the existing estimate: it starts at the current rate and takes its share through the pacer, without probing of its own.

Increase: additive plus proportional probe (always on)

Past slow start, two increase terms run every step, congested or not:

 r += a * Δt                         additive; a defaults to 2^16 B/s^2
 r += r * Δt / T_probe               proportional; T_probe = 8 s

The additive term is a fixed slope (bytes/s per second), sized to the path capacity (see the capacity floor). The proportional probe adds a fixed fraction of the rate per unit time (it e-folds over T_probe): that is what lets a flow recover in the same number of steps whatever the link rate; a fixed additive step is negligible next to a fast link. Together they are the upward pressure the multiplicative decrease balances at equilibrium. The probe is not free: it costs a standing queue that does not shrink with rate (see fairness).

Multiplicative decrease (proportional + derivative)

With congestion level m (the fed-back ece, or the local first-hop mark if there's no feedback yet), the cut is a proportional term plus a one-sided derivative term, a PD controller on the congestion signal:

 mark = MIN(m, M)                                   M = 512
 rise = MAX(m - m_prev, 0), capped at M             one-sided derivative
 cut  = r * rise / (4 * M)                          derivative, gain 1/4
 cut += r * mark * Δt_ms / (1000 * M)              proportional
 cut  = MIN(cut, r / 2)                             at most a halving
 r     -= cut
 m_prev = m
  • The proportional term uses the real elapsed milliseconds, so a starved sender that hasn't run the controller in a while still cuts by the right amount.
  • The derivative term (rise) only fires when the congestion level goes up between samples: one-sided, per-step. It sharpens the response right as congestion starts.
  • The cut is capped at r/2, so one step can at most halve the rate.

M = 512 = 32 × 16 pins "full congestion" at a mean ecn of 16, i.e. a mean queue of 16 × Q = 64 packets.

Staleness

Feedback comes once per receiver window, and that window stretches with the flow's byte rate (see the receiver estimate), so the staleness horizon has to stretch with it too, or a slow flow's mark would expire between feedbacks and the rate would ramp straight into a congested path. The sender mirrors the receiver: the horizon is four target windows' worth of bytes at the current rate, floored at ~268 ms so fast flows keep a fixed one; at the default rate floor it's ~8 s. Once a signal is older than that it's dropped (both the fed-back estimate and the local mark), which frees the rate to climb again after congestion clears. Every step clamps the rate to [r_min, r_max].

Capacity-derived floor and slope

The fed-back capacity code sets the controller's two rate-scale constants, per aggregate:

 target = decode(cap8) / 32              clamped to [2^13, 2^32] B/s
 r_min += (target - r_min) / 2           per feedback (moving average)
 a      = r_min

On a path that never reports a capacity, both fall back to fixed defaults (r_min = 2^13 B/s, a = 2^16 B/s^2). When the estimator is live, the fairness floor and the post-cut recovery slope scale with the bottleneck instead, and capacity kicks in wherever C/32 beats the default floor, roughly above 2 Mbit/s. A capacity older than 16 staleness horizons (~4.3 s for fast flows) reverts both to the defaults. That horizon is deliberately longer than the feedback horizon: feedback stops the instant the marks clear, which is exactly when you need the recovery slope, and the onset-fresh capacity (see the receiver estimate) re-seeds it on the first mark of the next episode anyway.

Ramp and recovery

Three mechanisms set how fast an aggregate reaches a fast link's rate, each owning one regime:

  • From scratch: slow start. The exponential ramp hits any real link rate in well under a second (~200 ms from the seed to 10 Gbit/s; every further doubling of link speed adds one ~14 ms doubling). Nothing marks on an uncongested path, so slow start just runs until the aggregate reaches its bottleneck.
  • After a cut: the probe. A cut is at most a halving, and the mark expires on the rate-relative horizon once congestion clears (see staleness); the probe then climbs back from a halving in T_probe · ln 2 ≈ 5.5 s, whatever the link rate.
  • At the bottom: the capacity floor. The rate never sits more than 32× below a measured bottleneck, and the additive slope refills C/32 a second, so the worst hole is bounded: floor to full capacity in tens of seconds, a halving in seconds.

These fit together because a deep cut needs sustained congestion, sustained congestion backs up the bottleneck queue, and a backed-up queue is exactly when its capacity gets measured and stamped (see forwarder marking). So whenever the controller has been cut deep, the scaled floor is already live. The default floor only ever applies to paths that never congested the sender in the first place, where there's nothing to recover from. That's why slow start can quit for good: the floor and the probe handle every recovery after it.

Regimes

   +-------------+   first mark seen    +----------------+
   | slow start  |--------------------->|  congestion    |
   | r *= e^(t/T)|                      |  avoidance     |
   +-------------+                      |                |
                                        | m == 0 ---> additive increase
                                        | m  > 0 ---> AI + PD decrease
                                        +----------------+
    m = fed-back ece, or the local first-hop mark when ece == 0


RTT behaviour

Because every increase and decrease is scaled by wall-clock time, two flows with different RTTs sharing a bottleneck follow the same rate law and settle at the same rate. So the steady-state allocation is RTT-independent.

That is a property of the equilibrium, not the dynamics. The feedback delay in the loop is the path RTT τ, and the controller neither measures nor compensates for it. At low rate the receiver window (up to its ~4.3 s ceiling) is the biggest lag: the loop is slow there in proportion to how slow the flow is, just like a long-RTT TCP. As the rate climbs the window shrinks toward its ~1 ms floor, so its share of the loop delay falls with capacity and the path RTT τ becomes the irreducible part. Two things keep the loop damped: the adaptive window takes out the delay-dominated corner at low τ, and a bucketed rate-velocity damping (the washout: each ~31 ms bucket pulls the rate a quarter of the way back toward its value at the previous bucket) supplies the damping this otherwise near-double-integrator loop lacks, which is what holds it together at high capacity and moderate τ. What is left is the loop gain √G ∝ √C: at very high capacity and large τ the √G · τ phase budget is the binding constraint, and no amount of windowing or damping removes it (see Limitations). So, precisely: RTT-fair in equilibrium, and dynamically stable up to a capacity-dependent RTT limit set by √G · τ.


Fairness

At equilibrium the two increase terms balance the proportional decrease, a + r/T_probe = r · m*/M, which pins the congestion level at m* = M · (a/r + 1/T_probe). Since ece = 8q (one mark unit per Q = 4 packets, 32 fixed-point steps per unit), that is a standing queue of

 q* = 64 * a / r  +  64 / T_probe        (packets, r in bytes/s)

The additive part (64 · a / r) fades to nothing with rate under the default slope; with the capacity-derived slope (a = C/32; see the capacity floor) it settles at 2 C / r, about two packets per competing flow, whatever the rate. The probe part (64 / T_probe) is a floor that does not move with rate (~8 packets at 8 s). That floor is what rate-independent convergence costs you, and it is what keeps q* above the marking threshold at high rate.

Two flows at the same bottleneck see the same m*, so each solves r_i = a / (m*/M - 1/T_probe) to the same rate. In network-utility-maximization terms that is proportional fairness with equal weights, which at a single bottleneck is just max-min fair.

Getting a good measurement at low rate is the estimator's job, not the floor's: the averaging window stretches with the flow (see the receiver estimate), so a CA-limited flow still keeps ~16 packets a window down to ~30 kbit/s, and the rate-relative gap threshold keeps its normal spacing from reading as idle. The default floor (2^13 B/s) only bounds the extremes (window ceiling, staleness horizon, pacer arithmetic). The floor itself scales with the fed-back path capacity (r_min = C/32; see the capacity floor), so about 32 CA-limited flows fit above it on any class of link. Flows whose bottlenecks are in different classes get different slopes, so fairness between them is capacity-weighted, not equal; see Limitations.


Limitations

Things the algorithm doesn't do. Listed here so nobody credits it with more than it manages.

  • Queue-only signal, no rate term. M is a queue level, so mb-ecn is a standing-queue controller: it has to build a queue (q* = 64 · a / r + 64 / T_probe) to get any signal at all, and can't sit at full utilisation on an empty queue the way a controller with a rate estimate can. The probe floor (~8 packets) keeps q* above the marking threshold at every rate, but you pay for it with a permanent standing queue.
  • Aggregation is per destination, per QoS cube. The aggregate is a stand-in for the path: flows to the same destination are assumed to share a bottleneck, and flows to different destinations run separate loops even when they pile up at the same hop. Each loop still converges to its own fair share (see fairness), so the cost is duplicated state and probing at that shared queue, not unfairness. One day we could tag the queue in the packet: if a forwarder stamped a queue id next to its mark, senders could key their aggregates on the real bottleneck instead of the destination, and pool the estimate and the probing across everything sharing it.
  • Capacity is learned, and there are gaps. The floor and slope ride on an in-band path-capacity estimate (see forwarder marking and the capacity floor), and that estimate has its own limits. A hop only learns its rate while its queue is backed up, so a rerouted or brand-new bottleneck reads unknown at first and the path MIN can briefly come from faster hops (the sender's smoothing and the next feedback bound the overshoot). The measurement is order-of-magnitude on purpose: quarter-log2 code (~19% a step), burst noise held down by a 16-packet minimum window and a max filter, and nothing learned below ~120 packets a second of busy drain (16 packets inside the ~134 ms staleness cap; at MTU that's ~1.4 Mbit/s, below which capacity/32 sits under the default floor anyway). The derived floor clamps to [2^13, 2^32] B/s, and flows with bottlenecks in different classes get capacity-weighted, not equal, shares at a common queue (see fairness).
  • The low end trades speed, not correctness. Below ~30 kbit/s the averaging window hits its ~4.3 s ceiling and holds fewer samples; below ~64 kbit/s of offered load an app-limited flow spaces its packets past the gap threshold and rides the raw onset branch, as it always did. At the floor, feedback and the staleness horizon are both measured in seconds, so a slow flow converges slowly, TCP-style. Nothing blows up; nothing's quick either.
  • No loss response. The controller only reacts to ECN marks. Loss is FRCP's problem, handled by retransmission, out of the CA's sight. At a bottleneck that drops instead of marking (say a plain drop-tail queue at the Ethernet or UDP shim), mb-ecn sees nothing and additive increase just keeps ramping. It's built for Ouroboros-native bottlenecks where the forwarder marks; there's no classic-bottleneck fallback.
  • Dynamic stability has a capacity-dependent RTT limit. As covered under RTT behaviour, the allocation is RTT-independent but the dynamics aren't: the binding constraint is √G · τ with √G ∝ √C. The adaptive window and the washout push that limit out a long way (a single flow at 10 GbE stays stable to ~50 ms RTT in the fluid model), but the far corner (100 GbE at ≥ 50 ms) is out of reach for a window/damping controller and would need explicit-rate signalling.

Architectural fit

Each of these properties falls out of something structural in the recursive architecture. Here's the mapping.

  • CA is fully separate from ARQ and flow control. Three concerns, three mechanisms, two scopes: FRCP does retransmission (ARQ) and flow control (the peer pacing the sender), end-to-end and per flow; CA runs in the IPCP, per aggregate. Each signal says one thing. A loss just means retransmit: a lossy link reads as lossy, and the congestion call is left to the marks (the limitations section's "no loss response" is this same separation from the other side). The peer's flow-control window just paces the endpoint: back-pressure from a slow receiver stays separate from congestion in the network. A retransmitted packet is ordinary traffic as far as the pacer cares. TCP jams all three into one window machine, where the receive window caps the congestion window and loss is both the reliability trigger and the congestion signal; here you can reason about, test, and change each one on its own.
  • CA sits below what the application picks. Every flow in the layer runs under the same rate law, whatever its QoS: a greedy raw sender shares a bottleneck fairly with a reliable stream because the control belongs to the layer, on the aggregate, not to the endpoint transport's good manners.
  • A layer owns its PCI, so the signal can be rich. Everyone in a layer enrolled into it, so the layer is one administrative domain by construction, at whatever scope it covers, and the header is the layer's own. That's why forwarders can write a multi-bit queue magnitude and a capacity byte straight into the packet. The datacenter schemes in heritage and positioning need exactly that kind of domain and only get it inside one operator's fabric; a recursive layer has it everywhere.
  • Layer-internal addressing makes the aggregate well-defined. Flows in a layer run between layer addresses, so the (destination, QoS cube) aggregate, which is RFC 3124's macroflow, drops straight out of the naming. Controller state scales with the number of destinations, and each QoS cube keeps its own loop, so service classes don't share a fate.
  • Flows are allocated, so feedback already has a channel. Every flow has state at both ends and a reverse direction, so the receiver's ece and the path capacity just ride the flow allocator's existing exchange.
  • There's no transport ack clock, and nothing leans on one. The layer has to handle raw flows, which carry no acknowledgements at all, so the controller paces on wall-clock time, and that constraint is exactly what buys the RTT-independent allocation under RTT behaviour. A restriction turned into the feature.
  • Recursion scales it. Each layer handles congestion over its own scope and timescale; many N-flows ride one N-1 flow, so aggregation compounds down the stack. The capacity a layer sees at its egress queue is whatever the layer below is actually giving it (a paced, shared lower flow, not some nominal wire speed), so the signal means something at every level and back-pressure cascades down layer by layer.

There's a boundary: at a shim over legacy media the layer below neither enrolls nor marks, which is the deployment edge behind the limitations section's no-loss-response point.


Heritage and positioning

mb-ecn is basically a rate-based ECN controller in the DECbit / QCN line, adapted to a recursive layer with no acknowledgements.

The core goes back to the first Ouroboros CA policy (2020): a multi-bit queue-depth mark relayed by forwarders to the receiver, a smoothed multi-bit ece fed back on the reverse flow at the same 1/32 fixed point used today, and a sender that paced packets from a time budget and ran slow start, additive increase and multiplicative decrease on wall-clock time slots. Rate-based and acknowledgement-free from the start, with RTT-independent allocation already the goal. The present algorithm keeps that skeleton and reworks every estimator around it: per-path aggregation with SFQ pacing, the rate-adaptive box-car receiver, elapsed-time-scaled AIMD with a derivative term and rate-velocity damping, and the in-band capacity signal.

  • DECbit (Ramakrishnan & Jain, 1988; ToCS 1990) established binary congestion feedback set by the switch on queue occupancy. mb-ecn keeps the switch-sets-on-queue idea and carries a magnitude.
  • The Congestion Manager (Balakrishnan et al., SIGCOMM 1999; RFC 3124, 2001) aggregates congestion state at the end host: one controller per macroflow, a scheduler apportioning its rate across the member streams, and new streams joining at the ensemble's current state instead of probing from scratch. mb-ecn's (destination, QoS cube) aggregate is the same idea placed inside the IPCP: every layer manages its own macroflows, recursively, the SFQ pacer plays the CM scheduler's role (see the pacer), and a joining flow rides the aggregate's estimates (see slow start). The CM gathers its signal from transport feedback at the edge; mb-ecn reads it off the wire from the forwarders.
  • DCTCP (Alizadeh et al., SIGCOMM 2010) recovers a queue magnitude at the endpoint by averaging a single threshold bit. mb-ecn instead reads the magnitude off the wire, then time-averages it at the receiver. The additive-increase / proportional-decrease structure is shared.
  • QCN (IEEE 802.1Qau, 2010) and DCQCN (Zhu et al., SIGCOMM 2015) contribute the rate-based (not window-based) ECN control skeleton. mb-ecn's proportional decrease plus a probing increase mirrors that skeleton; the increase here is wall-clock scaled rather than byte-counter/timer based.
  • HPCC (Li et al., SIGCOMM 2019) and PowerTCP (Addanki et al., NSDI 2022) combine a queue term and a rate term (via in-band telemetry or delay). mb-ecn now carries a coarse in-band rate (the quarter-log2 path MIN; see forwarder marking) but uses it only to scale the controller's gains; the congestion signal remains queue-only, which is still the main functional difference (see Limitations).
  • L4S / TCP Prague (RFC 9330–9332, 2023) make RTT-independence a requirement and use a high-frequency single-bit signal. mb-ecn shares the RTT-independent allocation goal but reaches it by wall-clock-scaled rate control, with a low-frequency multi-bit signal.

What's new is the pairing: a multi-bit magnitude on the wire with a wall-clock-scaled rate pacer, in a setting with no per-flow ack clock. The trade-off, spelled out above, is a queue-only signal and a reactivity capped by the averaging window.


Parameters

These are the values behind every figure in this document. They're coupled: the mechanisms only fit together while a few relations hold. The gap-restart horizon at the rate floor has to exceed a packet's service time (or a floor-rate flow loops at onset); a floor-rate flow's target window has to fit under the window ceiling; the control interval has to sit under the damping bucket, which sits under the idle credit cap; the rate floor mustn't exceed the slow-start seed; and the derived-floor ceiling has to stay under the rate ceiling. Retune them as a set, against those constraints.

Parameter Value Meaning
Marking quantum Q 4 packets Mark onset and step; layer-wide
Feedback resolution 1/32 ecn unit Fixed-point LSB of ece
Full-congestion reference M 512 Mean queue of 64 packets
Receiver window target 16 packets (16,000 bytes) Samples per window
Initial receiver window ~67 ms Before rate is known
Receiver window bounds ~1.05 ms – ~4.3 s Clamp on W
Window resize weight 1/4 Moving average toward target
Early window close 2 × target bytes Speed-up escape
Gap restart 4 windows Idle horizon (rate-relative)
Additive slope a (default) 2^16 B/s^2 Lower clamp of derived slope
Probe time constant T_probe 8 s Rate-proportional recovery
Derivative gain 1/4 Onset sharpening
Rate floor r_min (default) 2^13 B/s Lower clamp of derived floor
Slow-start seed rate 2^16 B/s Initial rate
Slow-start time constant 20 ms Exponential ramp
Rate ceiling r_max 2^37 B/s Upper clamp
Capacity scale C / 32 Floor and slope from bottleneck
Capacity smoothing 1/2 Moving average per feedback
Capacity staleness 16 × feedback horizon Revert to defaults
Derived floor ceiling 2^32 B/s Upper clamp on derived floor
Control interval 1 ms Minimum spacing of rate updates
Idle credit cap 50 ms Burst allowance / increase banking
Feedback staleness floor ~268 ms Floor of rate-relative horizon
Damping bucket ~31 ms Rate-velocity washout cadence
Damping fraction 1/4 Pullback per bucket
Estimator fold spacing 1 ms Minimum measurement cadence
Estimator window closure 16 packets drained Self-scaling window
Estimator window staleness ~134 ms Discard across traffic gaps
Estimator idle tolerance 1/8 of arrivals Unsaturated-window rejection
Estimator decay 1/16 per window Max-filter convergence from below


References

Source

The implementation follows the algorithm above:

  • src/ipcpd/unicast/ca/mb-ecn.c, mb-ecn.h: receiver estimator, pacer and rate control.
  • src/ipcpd/unicast/ca/ops.h: CA policy interface.
  • src/ipcpd/unicast/cap.c, cap.h: link-capacity estimator and quarter-log2 codec.
  • src/ipcpd/unicast/dt.c, fa.c: per-hop marking and stamping, and the feedback carrying ece + cap8.
  • src/ipcpd/unicast/psched.c: consumes the pacing deadline.

Literature

  • K. K. Ramakrishnan, R. Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks" (DECbit), ACM ToCS, 1990.
  • H. Balakrishnan, H. S. Rahul, S. Seshan, "An Integrated Congestion Management Architecture for Internet Hosts", SIGCOMM 1999; H. Balakrishnan, S. Seshan, "The Congestion Manager", RFC 3124, 2001.
  • M. Alizadeh et al., "Data Center TCP (DCTCP)", SIGCOMM 2010.
  • Y. Zhu et al., "Congestion Control for Large-Scale RDMA Deployments" (DCQCN), SIGCOMM 2015.
  • R. Mittal et al., "TIMELY: RTT-based Congestion Control", SIGCOMM 2015.
  • Y. Li et al., "HPCC: High Precision Congestion Control", SIGCOMM 2019.
  • V. Addanki et al., "PowerTCP", NSDI 2022.
  • K. De Schepper, B. Briscoe (eds.), "Low Latency, Low Loss, and Scalable Throughput (L4S)", RFC 9330/9331/9332.
  • F. Kelly, "Charging and Rate Control for Elastic Traffic", 1997; R. Srikant, "The Mathematics of Internet Congestion Control", 2004.