A collection of observability tools for exporting and visualising metrics collected from Ouroboros.

The exporter, oexport, samples the Ouroboros Resource Information Base (RIB) at a fixed interval and either pushes the readings to InfluxDB or writes them to a local file in InfluxDB line protocol. A Grafana dashboard for exploring the exported metrics is provided with the exporter.

Requirements

See also the step-by-step instructions for the Python dependencies, including using a virtual environment.

Ouroboros version >= 0.25, built with FUSE support. The exporter reads the RIB, which the IPCPs and the applications expose as FUSE filesystems under /tmp/ouroboros (the FUSE_PREFIX build parameter). The IPCPs publish their per-flow statistics when built with IPCP_FLOW_STATS. Applications only publish a RIB at all when built with PROC_FLOW_STATS. Both build parameters are enabled by default when FUSE is available.
Python 3 and pip. The exporter depends on the Python packages influxdb-client (with its ciso extra, which may need a C compiler) and pytz, which are installed automatically. The package takes its version from git, so install it from a git clone.
InfluxDB OSS 2.x, only when pushing metrics directly to InfluxDB. The file mode of the exporter does not need a running InfluxDB.
It is highly recommended to install Grafana for visualization.

Set up InfluxDB

Install and run InfluxDB and create a bucket in InfluxDB for exporting Ouroboros metrics, and an API token for (reading and) writing to that bucket. Consult the InfluxDB documentation on how to do this. The default bucket name for the exporter is ouroboros-metrics, and it is simplest to use that name.

Set up Grafana

To use Grafana, install and run Grafana open source.

Go to the Grafana UI (usually http://localhost:3000) and add InfluxDB as a data source:

Go to Connections -> Data sources -> Add data source and select InfluxDB.
Set the URL to that of your InfluxDB server, for instance http://localhost:8086.
Set “Flux” as the Query Language.
Under “InfluxDB Details”, set your Organization as in InfluxDB and paste an API token that can read the bucket into the Token field.

Then add the Ouroboros dashboard:

Select Dashboards -> New -> Import.
Upload the file dashboards/general.json from the metrics repository (cloned in the next section), or paste its contents into the “Import via dashboard JSON model” textbox and click “Load”.
Select the InfluxDB data source you added when Grafana asks for it, and click “Import”.

The dashboard assumes the bucket name ouroboros-metrics and needs editing if you use a different bucket name.

Install and run the exporter

On each of the machines running Ouroboros, clone the metrics repository

$ git clone https://codeberg.org/o7s/ouroboros-metrics
$ cd ouroboros-metrics

Create a virtual environment for Python and install the exporter and its dependencies into it:

$ python -m venv ./venv
$ source venv/bin/activate
$ pip install --upgrade pip
$ pip install .

This installs the oexport command in the virtual environment.

InfluxDB mode

Edit the config.ini.example file and fill out the InfluxDB information (url, token, org). This file will be identical on all machines.

[influx2]
url=http://<IP address of machine running InfluxDB>:8086
org=<your-org>
token=<your-token>

Save it as config.ini in the directory from which you run the exporter (or pass its location with --config).

Now you can run the exporter:

$ oexport

If you chose a different InfluxDB bucket name, run the exporter with the --bucket <yourbucketname> option. The InfluxDB url, organization and token can also be passed on the command line, in which case no config.ini is needed.

File mode

The exporter can write the metrics as InfluxDB line protocol to a local file instead of pushing them to InfluxDB. This needs no InfluxDB connection and no config.ini, and is convenient for experiments on machines that have no access to the InfluxDB server:

$ oexport --output-file metrics.lp

The resulting file can be bulk-imported into InfluxDB afterwards:

$ influx write --bucket ouroboros-metrics --file metrics.lp

The influx command needs the organization and a token with write access, from its active configuration (influx config) or from its --org and --token options.

Collection intervals

The exporter runs two collection loops on separate threads. The application flow (FRCT) metrics are sampled at the --interval (default 1000 ms). All other metrics (IPCPs, link state, DHT, flow allocator, data transfer, points of attachment) are sampled at the --slow-interval, which defaults to the larger of the FRCT interval and 1000 ms.

Command line options

Flag	Description
`-i, --interval`	FRCT collection interval in ms (default: 1000)
`--slow-interval`	Collection interval for the IPCP, link state database, DHT, flow allocator, data transfer and point of attachment metrics in ms (default: max(interval, 1000))
`-b, --bucket`	InfluxDB bucket (default: ouroboros-metrics)
`-o, --output-file`	Write InfluxDB line protocol to this file instead of InfluxDB
`-t, --tag`	Extra tag as key=value (repeatable)
`-c, --config`	Path to config.ini (default: ./config.ini)
`--url`	InfluxDB server URL
`--org`	InfluxDB organization
`--token`	InfluxDB authentication token

All settings except --slow-interval can also be set in config.ini. Command line flags take precedence over the config file, with one exception: a flag given with its default value, such as --interval 1000, does not override a value set in config.ini. For extra tags, a tag given on the command line wins over a config.ini tag with the same name.

Section	Key	Equivalent flag
`[influx2]`	`url`	`--url`
`[influx2]`	`org`	`--org`
`[influx2]`	`token`	`--token`
`[exporter]`	`interval`	`--interval`
`[exporter]`	`bucket`	`--bucket`
`[output]`	`file`	`--output-file`
`[output]`	`tags`	`--tag` (comma-separated key=value pairs)

Exported metrics

Each reading is written as an InfluxDB measurement with a single field, and a set of tags for selecting series. The field is named after the metric: ouroboros_flow_allocator_paced_rate has the field paced_rate, and ouroboros_lsdb_nodes_total has the field nodes. The table below lists where in the RIB each group of metrics comes from and which tags it carries. The system tag and any extra --tag tags are present on every measurement.

In the table, an N+1 flow is a flow an IPCP provides to the layer above it, an N-1 flow is a flow it uses from the layer below, and a QoS cube is one of the quality-of-service classes an IPCP forwards separately. The Glossary has more on these terms.

Measurement	RIB source	Tags	Contents
`ouroboros_<type>_ipcps_total`	IPCP directories	type	Number of IPCPs of each type (unicast, broadcast, local, unknown)
`ouroboros_flow_allocator_flows_total`	`<ipcp>/flow-allocator/`	ipcp, layer	Number of N+1 flows of an IPCP, written only while it has at least one
`ouroboros_flow_allocator_<metric>`	`<ipcp>/flow-allocator/<fd>`	ipcp, layer, flow_id	Per N+1 flow packet and byte counters, flow updates and the congestion avoidance state; flow_id is the local endpoint identifier of the flow
`ouroboros_flow_allocator_jain_index`	derived	ipcp, layer	Jain's fairness index over the paced rates of the N+1 flows of an IPCP
`ouroboros_lsdb_<metric>_total`	`<ipcp>/lsdb/`	ipcp, layer	Link state database: neighbors, nodes, links
`ouroboros_dht_<metric>_total`	`<ipcp>/dht/stats`	ipcp, layer	DHT directory: keys, local values, non-local values
`ouroboros_data_transfer_<metric>`	`<ipcp>/dt.<address>/<fd>`	ipcp, layer, flow_descriptor, qos_cube, endpoint	Per N-1 flow and per QoS cube counters of the forwarding element
`ouroboros_data_transfer_queued_packets_{rx,tx}`	`<ipcp>/dt.<address>/<fd>`	ipcp, layer, flow_descriptor	Packets queued on an N-1 flow
`ouroboros_poa_<metric>`	`<ipcp>/poa/<poa>`	ipcp, layer, poa	Point of attachment transport statistics (Ethernet, UDP)
`ouroboros_process_frct_<metric>`	`proc.<pid>/<fd>/frct`	process, flow_descriptor	FRCT state and counters of an application flow

The endpoint tag of the data transfer metrics is the remote address for N-1 flows, and the name of the internal component (dht, flow-allocator) for the flows the forwarding element provides to the IPCP's own components.

Dashboard

The Grafana dashboard allows you to explore various aspects of Ouroboros running on your local or remote systems. The screenshots on this page were taken from runs of the Ouroboros integration test suite, with one exporter per node and the system tag set to the node name (node-a, node-b, node-c). Every test uses a unicast layer n1 over Ethernet points of attachment, with the sender on node a.

The IPCP rows come from the staggered convergence test: a line of three nodes, where the link from node b to node c is shaped to 10 Mbit/s with 1 ms delay. Three greedy ocbr flows with a reliable (msg) QoS from node a to node c enter and leave 30 seconds apart, so the bottleneck carries one, two, three, two and then one flow again for 30 seconds each.
The FRCT rows show the stream transfer test: oftp sends 1 GiB over a stream QoS flow between node a and node b, on a link shaped to 100 Mbit/s with 10 ms delay and 1% packet loss. The transfer takes about 95 seconds, about 90% of the link rate; the Mathis formula gives a loss-based congestion controller such as TCP Reno about 7% of this link.
The fragmentation row shows the large ping test: oping with a 5000-byte payload, one ping per millisecond for 60 seconds, over a link with 2% loss.

Each row says which run its screenshot shows.

Variables

At the top, you can set a number of variables to restrict what is seen on the dashboard:

System selects the systems (hosts) in the network. The list contains every system that wrote metrics to InfluxDB in the last 5 days; Grafana cannot restrict it to the selected time range.

Type allows you to select metrics for a certain IPCP type. All Ouroboros IPCP types that have recent metrics are shown, including an unknown type. This may briefly pop up when the IPCP type is not available yet or misread by the exporter, e.g. during startup or shutdown of an IPCP.

Layer restricts the metrics to a Layer, selected by its Layer name.

IPCP restricts the metrics to an IPCP, selected by its IPCP name.

Interval selects the window in which metrics are aggregated. Metrics are aggregated from the exported values (e.g. mean or last value) that fall in this window, so it should be larger than the exporter interval to ensure that each window has enough raw data. The rate panels (bandwidth, packets per second) do not use this variable: they take the derivative of the exported counters over Grafana's automatic window for the selected time range.

Rows

The dashboard is organised in collapsible rows. The first eight rows show the IPCPs (the layers): the IPCP processes, their link state and directory state, the flows they provide to the layer above (N+1 flows) and their management traffic, congestion avoidance, the traffic of their internal components, and the points of attachment (PoAs). The Delta-t, RTT and FRCT rows show the state of the Flow and Retransmission Control Task (FRCT, the implementation of the Flow and Retransmission Control Protocol) for the flows between the applications, which is independent of the IPCPs that carry them.

In the panels below, upstream is the direction from the local IPCP or process to the remote one (sent), and downstream is the direction from the remote to the local one (received). Panels whose title ends in /s, and the bandwidth and packet rate panels, show the rate of change of an exported counter. All other panels show the last exported value in each interval. Most series are labelled by IPCP name and, for per-flow metrics, the flow endpoint identifier; the System IPCPs panel is labelled by IPCP type, and the FRCT panels by system, process and flow descriptor.

System

System IPCPs is the number of IPCPs of each type on each system. The example has one unicast IPCP per node.
System flows is the number of N+1 flows each IPCP is currently serving. The ocbr flows have their endpoints in the IPCPs on node a and node c, so both count 1, 2, 3, 2 and 1 flows as the flows enter and leave. The IPCP on node b only forwards, and the panel shows no value for an IPCP without flows.

Link State Database

The link state database holds the knowledge a unicast IPCP has about the topology of the routing area(s) it is in.

Neighbors is the number of IPCPs the IPCP has a link to.
Nodes is the number of IPCPs the IPCP knows about, including itself.
Links (unidirectional) is the number of links in the database; each direction of a link between two IPCPs is counted separately.

In the example, the IPCP on node b in the middle of the line has 2 neighbors and the two at the ends have 1. All three know the 3 nodes and the 4 unidirectional links.

DHT

The directory of a unicast layer is a distributed hash table (DHT) that maps the hashes of registered names to the addresses of the IPCPs where they are registered.

Keys is the number of hashed names the IPCP stores.
Local Values is the number of values the IPCP stores for names that are registered with itself.
Non-local Values is the number of values the IPCP stores on behalf of other IPCPs.

In the example, the sink name is registered with the IPCP on node c. All three IPCPs store that one key: node c as a local value, nodes a and b as a non-local value, so the three lines in the Keys panel overlap.

IPCP N+1 flows

These graphs show the traffic on each flow the IPCP serves to the layer above, as seen by the flow allocator. The series are labelled with the IPCP name and the local endpoint identifier of the flow.

Upstream bandwidth and Upstream packet rate are the rates at which the IPCP sends on the flow, in bits per second and packets per second.
Total sent bytes and Total sent packets are the counters these rates are derived from.
Downstream bandwidth, Downstream packet rate, Total received bytes and Total received packets are the same for the receive direction.

The example shows the three greedy flows on node a sharing the 10 Mbit/s bottleneck: the first flow has it alone for 30 seconds, then halves its rate when the second enters, then drops to a third when the third enters, and the rates climb back as the flows leave. The flows at the sink on node c receive the same, together the full link rate. The traffic in the other direction is only the acknowledgments of FRCT, about 20 kbit/s per flow. The bandwidth axes are logarithmic, so the two directions fit on one plot. The IPCP on node b has no N+1 flows, so it does not appear.

N+1 flow management

These 4 panels show the flow updates between the flow allocators at both ends of a flow: the congestion feedback of congestion avoidance. The receiving IPCP monitors the congestion marks on the packets of the flow and sends its estimate of the congestion level back to the sending IPCP. The flow allocators also exchange heartbeats for congestion avoidance (see the next row), which these panels do not count.

Upstream flow update rate and Flow updates sent are the rate and the total of the flow updates an IPCP sends, i.e. the feedback it gives as a receiver.
Downstream flow update rate and Flow updates received are the rate and the total of the flow updates an IPCP receives as a sender.

In the example, node c, the receiver, sends between 5 and 15 flow updates per second on each flow to node a as long as the bottleneck is congested: most while a flow has the link alone, fewer as the flows share it.

IPCP Congestion avoidance

This row shows the internals of the Multi-Bit Explicit Congestion Notification (MB-ECN) congestion avoidance algorithm. MB-ECN is rate-based: the sender paces its packets at a rate that it adjusts according to the congestion level in the flow updates from the receiver. The forwarders mark each packet with a multi-bit value that grows with the logarithm of their queue depth, and stamp the capacity of their outgoing link into it. All flows of an IPCP to the same destination and QoS cube form one aggregate that shares one controller and one rate, which the pacer divides fairly between them. The panels show the values of the aggregate once for each flow in it, labelled by the endpoint identifier of the flow.

Congestion regime is the state of the rate controller as a code: 0 is slow start (no congestion seen yet, the rate ramps up exponentially), 1 is additive increase, 2 is proportional decrease (the congestion level is non-zero and the rate is being cut), 3 is source limited (the application offers less than the allowed rate) and 4 is increase held. In the increase held regime the controller stops increasing the rate. It enters this regime when the congestion level reaches the controller's reference level on a path with a measured capacity, when the feedback goes stale while the flow is backlogged, or after a signal-loss cut, and it leaves it when fresh feedback below the reference arrives.
Upstream congestion level is the congestion level the sender receives in the flow updates from the receiver.
Paced rate (Bytes/s) is the sending rate the controller has set. It reads 0 in slow start, before the first congestion feedback.
Downstream congestion level is the congestion level the receiver side measures from the marks on the incoming packets and sends back to its sender. For a flow between two monitored systems, the upstream level at one end tracks the downstream level at the other.
Pacer lead (Bytes) is how far the pacer has scheduled ahead of its virtual clock: the delay, expressed in bytes at the paced rate, before the next packet may leave. One or two packets' worth means packets are queueing at the pacer.
Target Bandwidth is the paced rate in bits per second, for comparison with the bandwidth panels.
Flow fairness (Jain index) is Jain's fairness index over the paced rates of the N+1 flows of an IPCP that are using their allocation (regimes 1, 2 and 4). It is 1 when all these flows have the same rate, and is only exported when there are at least two such flows. Flows in one aggregate always report the same rate, so the index is only informative across aggregates, i.e. for flows to different destinations or QoS cubes.
Signal-loss cuts/s is the rate at which the controller halved the rate because neither flow updates nor heartbeat replies arrived for several times the expected feedback interval while the flow was backlogged.
Path capacity (Bytes/s) is the capacity of the bottleneck link as measured by the forwarders along the path and reported by the receiver.
Capacity rate floor (Bytes/s) is a thirty-second of the path capacity. Despite its name it does not hold the rate up: it sets the slope of the additive increase, so that recovery after a cut scales with the bottleneck.
Ramp time constant (ss_tc) is the time constant of the exponential ramp in slow start, which the controller derives from the heartbeat round-trip time.
Heartbeat RTT samples/s is the rate of round-trip time samples from heartbeats. The sender sends a heartbeat whenever nothing came back from the receiver within the heartbeat interval, half the ramp time constant but at least 40 ms. The reply yields an RTT sample and counts as a sign of life for the signal-loss detection.

In the example, the three flows on node a share one aggregate, so all report the same paced rate: about 1.2 MB/s, or 10 Mbit/s in the Target Bandwidth panel, which the pacer splits evenly between the flows that are active. The regime alternates between additive increase (1) and proportional decrease (2) as the queue at the bottleneck builds up and drains, and the path capacity is measured at about 10 Mbit/s. The Jain index is 1, as it must be within one aggregate. The flows on node c only carry acknowledgments back, so their paced rate stays at a few tens of kB/s, and their downstream congestion level shows the marks on the data packets arriving over the bottleneck. The IPCP on node b has no flows and does not appear. The heartbeat samples arrive at about 15 to 20 per second.

IPCP Data Transfer - Local Components

This row shows the traffic of the IPCP's internal components as measured by the forwarding element (data transfer). Besides the N+1 flows, the forwarding element also carries the traffic of the components of the IPCP itself, each of which has its own endpoint: the DHT (directory) and the flow allocator. The panels show, per component, the upstream bandwidth and packet rate (traffic the component sends), the downstream bandwidth and packet rate (traffic delivered to the component), and the matching totals, all for QoS cube 0. The bandwidth panels in this row are in bytes per second.

The DHT didn't do much during the example run: a handful of packets per IPCP, too few to compute a rate from. It is only needed for name-to-address resolution and it only sends and receives packets when a name is resolved or when it needs to refresh its state, which happens once every 15 minutes or so. The flow allocator traffic on nodes a and c is the flow updates of the N+1 flow management row plus the congestion avoidance heartbeats and their replies, together about 50 packets per second in each direction. The IPCP on node b only forwards and has no flow allocator traffic of its own.

PoA transport

A point of attachment (PoA) connects a unicast or broadcast IPCP to an underlying transport such as Ethernet or UDP. This row shows the statistics of each PoA, labelled with the IPCP and PoA name, for example eth.e0.A000 for an Ethernet PoA or udp4.10.1.0.1.3435 for a UDP one. Most statistics are common to all PoA types. The kernel packet and queueing discipline (qdisc) counters are only available on Ethernet, and their panel titles end in (eth); on a UDP point of attachment those panels stay empty.

PoA queue depth (bytes) is the data in the kernel socket send queue of the PoA. PoA queue depth (packets) estimates the number of packets in that queue from the mean packet cost.
PoA socket buffers are the send and receive buffer sizes of the socket.
PoA mean packet cost is the mean size of a packet on the PoA in bytes, including the transport overhead.
PoA upstream bandwidth and PoA downstream bandwidth are the rates at which the PoA sends and receives.
PoA packet failures are the totals of packets with a bad endpoint identifier, buffer allocation failures, packet read failures, packet send failures and packets that could not be delivered to the N+1 flow.
PoA kernel packets dropped (eth) is the number of packets the kernel dropped on the socket.
PoA name queries and PoA name replies are the totals of the name queries and replies sent and received for resolving names over the PoA (the equivalent of ARP: finding the transport address of the IPCP a name is registered with).
PoA qdisc backlog (bytes) (eth) and PoA qdisc backlog (packets) (eth) are the data waiting in the queueing discipline of the network interface under the PoA. On a link shaped with a rate limit, this is where the queue builds.

MB-ECN marks packets according to the queue under the PoA: the qdisc backlog on an Ethernet PoA, and the socket send queue on a UDP PoA. In the example, the rate limit sits in the qdisc of the second interface of node b (eth.e1.A000), the one facing node c, so that is the only PoA where the socket queue and the qdisc backlog fill: about 30 packets when the first flow starts, and a few packets whenever the flows push the link to its limit after that. The upstream bandwidth is at 10 Mbit/s on the first interface of node a and the second of node b, and the downstream bandwidth on the receiving interfaces of nodes b and c. The mean packet cost is about 1.5 KiB on the PoAs that carry data and much lower on those that only carry acknowledgments. The name queries are the three each of nodes a and c sent to find their neighbor, node b, which answered them on both its interfaces.

Delta-t constants

The first row to do with FRCT shows the Delta-t timers of each flow. Delta-t is the protocol on which FRCT is based. The series are labelled with the system, the process (proc.<pid>) and the flow descriptor in that process. The screenshots of the FRCT rows show the stream transfer, except for the fragmentation row.

Maximum Packet Lifetime (MPL) is the maximum time a packet can live in the network. It is set by the layer that carries the flow; the default is 100 ms for unicast and broadcast layers and 50 ms for a local layer.
Retransmission timer (R-timer) is the maximum time during which a retransmission for a packet may be sent by the sender. The default is 32 seconds. The first retransmission happens after the retransmission timeout (RTO, see RTT-estimation), then after 2 * RTO, 4 * RTO and so on with an exponential back-off, but no packets are sent after R has expired. If this happens, the flow is considered failed / down.
Acknowledgment Timer (A-timer) is the maximum time within which a packet may be acknowledged by the receiver. The default is 1 second.

Delta-t Window

This row shows the flow control windows of FRCT. FRCT tracks the packets in flight: the packets that were sent by the sender, but have not been acknowledged yet by the receiver. Each packet is numbered sequentially starting from a random value.

Sender Window Size is the number of packets the sender may have in flight: the difference between the right and left edges of its window.
Sender Left Window Edge (sLWE) is the oldest unacknowledged sequence number; Sender Right Window Edge (sRWE) is the first sequence number the sender may not send yet; Sender Next Sequence Number is the sequence number of the next packet to be sent.
Receiver Window Size, Receiver Left Window Edge (rLWE) and Receiver Right Window Edge (rRWE) are the same quantities at the receiver: the left edge is the next sequence number it expects, the right edge the first it will not accept.
Receiver Last Acknowledged is the cumulative acknowledgment the receiver last sent, i.e. the next sequence number it expected at that time.
Duplicate packets received is the total number of data packets that arrived with a sequence number the receiver already had, delivered or waiting in the reordering queue. Duplicates that are retransmissions are counted separately, under the duplicate retransmissions of the RXM rates row.
Packets received outside of window is the total number of packets dropped because their sequence number fell outside the receive window.
Packets received outside of reorder queue is the total number of packets dropped because they did not fit the reordering queue, where out-of-order packets wait until the gap before them is filled.
Rendez-vous packets sent is the total number of rendez-vous packets. A sender whose window is closed sends these to find out when the receiver opens its window again.

The window can never be larger than the reordering queue, which holds 128 packets by default. The integration test build enlarges the queue to 32768 packets, which is why the window sizes in the screenshot sit at about 32768. The transfer moves about 780,000 packets, so the window edges and sequence numbers advance by that much, which is too little to see next to their random starting value of about 1.14 billion.

RTT-estimation

This row shows how FRCT estimates the round-trip time of a flow, which drives the retransmission timeout.

Smoothed Round-Trip Time (sRTT) is the exponentially smoothed round-trip time.
Retransmission timeout (RTO) is the time after which an unacknowledged packet is retransmitted, computed from the smoothed round-trip time and its deviation. It is never less than twice the smoothed round-trip time, nor less than the resolution of the retransmission timer (about 1 ms).
RTT deviation off mean is the smoothed deviation of the round-trip time samples from the mean.
Minimum RTT (RACK base) is the lowest round-trip time seen on the flow in the last 5 minutes. It is the base for the reordering window of the RACK (Recent ACKnowledgment) loss detection, described in the next row.
RTT samples/s is the rate at which the estimator receives new samples. A cumulative acknowledgment yields at most one sample, from the oldest packet it acknowledges, provided that packet was not retransmitted; a selective acknowledgment yields one sample from the newest packet it confirms. FRCT also sends its own RTT probes: at the start of a flow, and whenever no sample arrived for two smoothed round-trip times while data is in flight.

In the example, the link adds 10 ms in each direction and the queue at the shaped link a further 10 ms or so, so both ends see a smoothed round-trip time of about 30 ms, with a minimum of about 25 ms. The retransmission timeout starts at its initial value of a few hundred milliseconds and settles at about 60 ms, twice the smoothed round-trip time. The sender gets about 2,700 samples per second, one from every selective acknowledgment it receives; the receiver, which sends no data, only gets about a dozen per second, from its own RTT probes.

FRCT — RXM rates

This row shows the rates at which FRCT retransmits (RXM) packets, broken down by what triggered the retransmission.

RXMs sent/s (all drivers) is the sum of the RTO-driven, RACK-driven, zero-reorder-window and NACK-driven rates. The SACK-driven rate is not added because those repairs are already counted as RACK-driven or zero-reorder-window; tail loss probes are not counted as retransmissions.
RXMs rcvd/s is the rate at which the receiver receives retransmitted packets, including tail loss probes, so it can exceed the sent total.
RTO-driven RXMs/s are retransmissions because the retransmission timeout expired without an acknowledgment.
RACK-driven RXMs/s are retransmissions by the time-based RACK loss detection: a packet that was sent before an already acknowledged packet is declared lost once a reordering window has passed.
SACK-driven RXMs/s are retransmissions of holes reported in a selective acknowledgment (SACK). Each of these is also counted as RACK-driven or zero-reorder-window, depending on how old the hole was.
Zero-reorder-window RXMs/s are RACK repairs that were only allowed because the reordering tolerance had been collapsed to zero: a packet is repaired as soon as a later packet is acknowledged, without waiting out a reordering window. FRCT collapses the window during a recovery episode, or once three packets above a hole have been selectively acknowledged, as long as the flow has not seen reordering before (the reordering window multiplier is still 1).
NACK-driven RXMs/s are retransmissions requested by a negative acknowledgment (NACK) from the receiver.
Duplicate RXMs rcvd/s is the part of the received retransmissions the receiver already had.

In the example, the link drops 1% of the packets, so the sender on node a retransmits about 80 packets per second, nearly all of them SACK-driven repairs with the reordering window collapsed to zero: the first selective acknowledgment that reports a hole triggers the repair. Only a handful are RACK-driven with a reordering window, and 4 in total are RTO-driven. The receiver on node b gets them at the same rate; the few it already had are counted as duplicate retransmissions.

FRCT — ACK rates

This row shows the acknowledgment traffic of a flow. Acknowledgments are piggybacked on data packets when there is data flowing in the other direction, so a receiver that has nothing to send emits separate acknowledgment packets, delayed by a short timer so that one acknowledgment can cover several packets.

ACK packets sent/s is the rate of acknowledgment packets FRCT sends on its own: when the delayed-acknowledgment timer expires, or right after a packet arrives that carries no new data, such as a duplicate. Selective acknowledgments sent because out-of-order data arrived are only counted under SACKs sent.
ACK packets rcvd/s is the rate of received control packets that carry an acknowledgment, including all selective acknowledgments, so it does not match the peer's sent rate exactly.
Delayed-ACK fires/s counts every time FRCT considered sending an acknowledgment: each expiry of the delayed-acknowledgment timer and each received packet without new data. Most of these are suppressed: when there is nothing new to acknowledge, when the receiver has been idle for longer than the acknowledgment timeout, or when the sender is about to piggyback the acknowledgment on a data packet.
ACK wire dups/s is the rate of acknowledgments received that were exact duplicates of a previous one and were dropped.
SACKs sent/s and SACKs rcvd/s are the rates of selective acknowledgments (SACKs), which report the ranges the receiver has beyond the first gap.
D-SACKs sent/s and D-SACKs received/s are the rates of duplicate SACK (D-SACK) blocks. A D-SACK tells the sender a retransmission was unnecessary because the receiver already had the packet, which lets the sender correct its reordering window.
D-SACKs out-of-range/s is the rate of D-SACK blocks the sender dropped because they referred to packets too far in the past.

In the example, the receiver on node b almost always has a hole in its reordering queue, so most of what it sends are selective acknowledgments, about 2,700 per second, and only about 50 plain acknowledgments per second. The sender on node a receives all of them as acknowledgments. Each of the 10 duplicate retransmissions the receiver gets is answered with a D-SACK block, which the sender receives. The sender sends no acknowledgments of its own, yet shows about 2,700 delayed-ACK checks per second: one for every acknowledgment it receives, all suppressed.

FRCT — anomalies

This row shows events that indicate loss, reordering or an idle peer.

Duplicates rcvd/s is the rate of data packets received that the receiver already had, either delivered or waiting in the reordering queue. Duplicates that are retransmissions are counted as duplicate RXMs instead.
OOO arrivals/s is the rate of packets that arrived out of order and were stored in the reordering queue. It is only counted on flows that use selective acknowledgments.
Out-of-window/s and Out-of-rqueue/s are the rates of packets dropped because they fell outside the receive window or did not fit the reordering queue.
Keepalives sent/s is the rate of keepalive packets. Keepalives are sent on a flow with a QoS timeout when the sender has been idle for a quarter of that timeout, so the peer can tell an idle flow from a dead one.

In the example, nearly every packet the receiver on node b gets arrives out of order, about 7,500 per second: with 1% loss, there is almost always a hole waiting for its repair, and everything that arrives behind it is queued. Duplicates stay at zero because the link does not duplicate packets; the only duplicates are a few retransmissions, counted in the RXM rates row. The receiver, which has no data to send, sends a keepalive every 30 seconds.

FRCT — rbuff queues

RX rbuff queued is the number of packets in the receive ring buffer of the flow in shared memory, waiting for the application to read them.
TX rbuff queued is the number of packets in the transmit ring buffer, waiting for the IPCP to send them.

In the example, the transmit ring on node a holds about 80 packets most of the time: the application writes faster than the shaped link drains, so the sender runs against its ring. The receive ring on node b is read almost as fast as it fills, and only backs up briefly, by a few dozen packets, when a repair lets a run of queued packets through at once.

FRCT — Fragmentation

When an application writes a message (a service data unit, SDU) that is larger than the maximum packet size of the flow, FRCT fragments it and reassembles it at the receiver.

Fragments sent/s and Fragments rcvd/s are the rates of fragments sent and received.
Fragments dropped/s is the rate of fragments dropped because the sequence of fragments was malformed or the reassembled message did not fit the flow's maximum message size or the reader's buffer.
SDU writes fragged/s is the rate of application writes that had to be fragmented.

In the example, each 5000-byte ping spans 4 packets on the Ethernet link, and so does its reply, so both ends write about 1,000 fragmented messages per second and send and receive about 4,000 fragments per second; the two lines overlap. The 2% loss on the link costs a few thousand fragments over the run, but the flow is reliable, so FRCT retransmits them and every message is reassembled: no fragments are dropped. Fragments are only dropped on a flow with a loss-tolerant QoS, where an incomplete message is discarded when its run of fragments turns out to be malformed.

FRCT — Stream

For flows with a stream (byte-oriented) QoS, FRCT delivers a byte stream instead of messages.

Stream tx rate and Stream rx rate are the rates at which stream bytes are sent and received.
Stream delivered is the rate at which bytes are delivered to the reading application.
Stream pkts dropped/s is the rate of stream packets the receiver had to drop.

In the example, the sender on node a pushes about 10.9 MiB/s (about 91 Mbit/s) into the stream, close to the 100 Mbit/s of the shaped link, and the receiver on node b takes the same out of it and delivers it to the application. No stream packets are dropped.

FRCT — Signaling

This row shows the less frequent control events of FRCT.

Pre-DRF NACKs sent/s and Pre-DRF NACKs received/s are negative acknowledgments (NACKs) the receiver sends when data packets arrive before the first packet carrying the Data Run Flag (DRF) has established the window, for instance when the first packet of a flow was lost.
TLP probes sent/s is the rate of tail loss probes (TLPs). When the last packets of a burst are not acknowledged in time, the sender probes with a retransmission of the oldest unacknowledged packet instead of waiting for the full retransmission timeout.
Inactivity drops/s are data packets that arrived after the receiver's window state had aged out and were discarded without a NACK, because NACKs are rate-limited to one per round-trip time.
DRF window rebases/s is the rate at which a packet carrying the DRF re-established the receiver window after it had aged out.
rq slots cleared/s is the rate at which packets were discarded from the reordering queue because the window was rebased or the flow was torn down.

All of these stay at zero during the transfer.

FRCT — Tx ring drops

Tx ring drops/s is the rate of packets FRCT failed to send. For control packets the main cause is a full transmit ring buffer, which FRCT waits on for at most 250 microseconds; prioritised retransmissions do not wait at all. Failures to checksum, encrypt or hand a packet to the point of attachment are counted here too.
RXM tx into dead flow/s is the rate of retransmissions that were attempted on a flow that was already being torn down.
Tx drops/s (ACK), (SACK), (KA), (RTTP), (NACK), (RDV) and (other) break the drops down by packet type: acknowledgment, selective acknowledgment, keepalive, RTT probe, negative acknowledgment, rendez-vous, and everything else, which includes all data packets and retransmissions.

On an unimpaired flow these are all zero. In the example, the start of the transfer overruns the receiver's ring on node b: about 350 selective acknowledgments and 6 plain acknowledgments are dropped in two short bursts, plus a few RTT probes on both ends over the whole run. The sender drops no data packets at all.

Ouroboros Metrics

Contents

Requirements

Set up InfluxDB

Set up Grafana

Install and run the exporter

InfluxDB mode

File mode

Tags

Collection intervals

Command line options

Exported metrics

Dashboard

Variables

Rows

System

Link State Database

DHT

IPCP N+1 flows

N+1 flow management

IPCP Congestion avoidance

IPCP Data Transfer - Local Components

PoA transport

Delta-t constants

Delta-t Window

RTT-estimation

FRCT — RXM rates

FRCT — ACK rates

FRCT — anomalies

FRCT — rbuff queues

FRCT — Fragmentation

FRCT — Stream

FRCT — Signaling

FRCT — Tx ring drops

Navigation menu

Ouroboros Metrics

Requirements

Set up InfluxDB

Set up Grafana

Install and run the exporter

InfluxDB mode

File mode

Tags

Collection intervals

Command line options

Exported metrics

Dashboard

Variables

Rows

System

Link State Database

DHT

IPCP N+1 flows

N+1 flow management

IPCP Congestion avoidance

IPCP Data Transfer - Local Components

PoA transport

Delta-t constants

Delta-t Window

RTT-estimation

FRCT — RXM rates

FRCT — ACK rates

FRCT — anomalies

FRCT — rbuff queues

FRCT — Fragmentation

FRCT — Stream

FRCT — Signaling

FRCT — Tx ring drops

Navigation menu

Search