Ouroboros and TCP/IP

 This page is under construction  

OSI model - original sketch

The five-layer Internet model is the result of retro-actively fitting the TCP/IP protocol stack to the seven-layer OSI model. An original sketch of the seven-layer model is shown here for reference, adapted from a blog post by John Day. How the Ouroboros layers map onto the five of the Internet model is on the functional layering page. This page takes the other direction: it lists problems in the Internet architecture and the TCP/IP protocol stack, and says what Ouroboros does about each.

The list is in no particular order, and the Ouroboros answers are not claimed to be unique to Ouroboros. Most of them run in the prototype today, and the tutorials show them; where an answer is still research, the section says so. The implementation overview has the status per component.

Network ossification

The problem

Ossification of the Internet is the name for its lack of evolvability: the difficulty of changing and deploying core protocols. The prime example is the slow adoption of IPv6. At the root is the lack of separation between mechanism and policy at layer 3: there is no service interface. Deploying a new network-layer protocol is always a large undertaking, requiring changes to operating systems and network hardware. But this essentially layer-3 change bubbles up to the application (AF_INET6, sockaddr_in6, AAAA records, a new pseudo-header in the transport checksum) and trickles down into the link layer (ethertype 0x86DD), and every box in between keys on the version. The question does this application support IPv6? should never have been a thing.

Ouroboros

Every layer hides its internals behind the same interface. An application asks for a flow to a name with a QoS, and the layer below an IPCP only sees flows; neither knows the address format, the header or the routing of the layer in between. The same unicast IPCP runs over Ethernet, over UDP or over another Ouroboros layer, and nothing above it can tell which. Where a change leaks in the two models:

Change	TCP/IP	Ouroboros
A new network protocol	The socket API, DNS, the transport checksum, the link-layer type field, every middlebox	Confined to the layer: the application API carries names and QoS, the layer below carries flows
A new transport	A new protocol number, which middleboxes drop; QUIC hides in UDP	Reliability is a QoS requested by the application and provided by FRCT in the application's own process; the network has no transport field to check
A new link technology	A new address family at the bottom, and ARP, DHCP and the like to tie it to IP	A point of attachment: the IPCP attaches to it and the layer is unchanged

The second row has a cost in code as well as in deployment. Homa, the receiver-driven transport for the datacentre, is about nineteen thousand lines in the Linux kernel, of which some two thousand are the idea itself: the grants, shortest message first and the priority cutoffs. The rest is there because the sockets world gives a new transport nothing to stand on, so it must bring its own socket family, RPC state, buffer pools, loss repair, offload path and metrics, and then win a protocol number so the network will pass it. In Ouroboros the same idea would be one congestion-avoidance policy of about fifteen hundred lines, seams included, because everything else is already provided and shared with every other policy: flows and the IRMd for naming and allocation, FRCT for fragmentation and repair, shared-memory rings for buffers, and the Layer MTU for message boundaries. A peer of TCP must reimplement the host; a policy only has to say what it does differently, because the host is already a Layer.

Protocol ossification

The problem

If a protocol is designed with a flexible structure, but that flexibility is never used in practice, some implementation is going to assume it is constant. The TCP/IP protocols are riddled with fields that have only a few values that are ever used, and those values name the protocol or the service above. The Protocol field in IP is one: a lot of hardware drops anything that is non-standard, and adding a value to the standard requires updating a lot of hardware. Well-known port numbers are another: they are a public list of what is running where, and the basis of every port scan.

Ouroboros

The Ouroboros protocols have no field that names what is above them, and no field with a well-defined value. Where the TCP/IP headers identify a protocol or a service, the Ouroboros headers carry a label chosen for one flow or one attachment:

Field	TCP/IP	Ouroboros
Protocol field	Names the transport; new values are dropped	None. The data transfer header carries a destination address, QoS cube, TTL, ECN, a capacity code and an endpoint id
Port number	Names the service; well known, listened on, scanned	A 64-bit endpoint id, drawn at random for each flow, valid for the life of the flow
Source address	In every packet; forgeable, and the basis of amplification	None. A reply goes to the peer stored at flow allocation, not to whatever the packet says

Topology ossification

The problem

Addressing can follow topology or topology can follow addressing. Choose one. -- Rekhter's law

Public IP addresses are administratively distributed, cementing a large-scale global topology for the Internet that does not reflect its physical connectivity. The mismatch between logical and physical topology is mitigated by adding specific routing entries, resulting in super-linear routing table sizes. The fact that IP addresses are assigned to interfaces doesn't help, but is not the core issue here. The problem is not restricted to public networks. Private networks too are defined by their addressing, and since all networks grow and change over time, small topological changes often conflict with the existing subnetting and addressing, requiring a periodic network redesign to keep things efficient. Optimal network layout is also hindered by administrative habit: network addresses, especially IPv4, are so engraved in the consciousness of network engineers that they are used as the primary labels for servers and virtual machines. IPv6's large address space should in theory improve the topology ossification issue. Some argue that its 128-bit addresses are a waste of bits; in our opinion, it is the right choice to allow for topology flexibility in a network with evolving topologies, especially so in the case of the Internet, where the topology follows the manually assigned addressing. There is a waste of bits, it is the source address in the IPv6 packet header. The impact of IPv6's less intuitive address syntax on its adoption should not be underestimated.

Ouroboros

Ouroboros provides primitives to define and modify the network topology (enrolment and adjacency management). Addresses are internal to the network forwarding layer and never visible to applications, so nothing outside the Layer can depend on them, and the Layer is free to change them.

The forwarding layer scales by giving an address several levels: one name per level, from the most significant down. Each level has its own routing area, a set of IPCPs that share one dissemination scope, realised as a broadcast Layer, and an IPCP joins the areas of the levels it takes part in. Areas form a tree; in the parent's routing a child area is a single vertex, reached through the gateways that are members of both, so an IPCP holds a routing table per level it joined, not one for the whole Layer. Forwarding descends the address: the first level at which the destination differs from the local address is the level the packet is forwarded at, and an IPCP that has no scope at that level hands the packet to a member of its own area that does.

Within an area, a name is bound to exactly one thing, a member or a child area, but a member or a child may bear two names at once. That is what makes the topology changeable while it runs: renumbering an area, splitting a tree at its root, joining trees under a new root, merging two sibling areas and moving a node into another area are each a sequence of make-before-break steps, adopt the new name, retire the old one, with every pair of nodes reachable before and after each step. This model has been worked out in detail and checked against a model of the forwarding and routing, for delivery between every pair of nodes, for path stretch against flat shortest paths, and for connectivity across every step of every operation, and the changes to the unicast IPCP have been mapped out: the forwarding lookup, the routing component and the address authority change, management flows become joins into broadcast Layers, and the flat address of the prototype today is the case of a single level. It is not in the tree yet. In the prototype, a Layer has one address level, the address authority hands out a random address, and routing is link state over the whole Layer. What is still open is automation: which areas exist, which IPCPs join them and which names they get are given by the operator, as subnets are in IP, and whether a Layer can configure its areas from its own topology is a question for later.

Names are synonyms for addresses

The problem

The Internet has no name for a service. DNS maps a host name to an address, and the application connects to that address and a port. From there on the name is gone: the kernel holds a pair of addresses and ports and has never heard of the name, so when the service moves, or the client's own address changes, nothing looks the name up again and follows it. The connection breaks, and it is the application's job to notice and start another. Everything that needs a service rather than a host is added on the side: SRV records, which still answer with an address and a port, load balancers, anycast addresses, service meshes. Which address family to use, where the resolver is, and what port the service listens on are all the application's concern, and every application solves them again. DNS is also more than one job: besides resolving names it is the register of who owns a name, and the trust chain that vouches for it, DNSSEC, is a third system bolted on.

Ouroboros

A service has a name, and that is all a client needs to know. The name goes into the flow allocation itself and the Layer resolves it; the process never learns an address, so there is nothing for it to hold on to and nothing to go stale. The name is bound to a process or a program, not to a location, and the directory of the Layer keeps track of where the service is. The directory does only that: given a name, it finds an address inside the Layer. It is closer to ARP than to DNS, as it holds no notion of who is entitled to a name; ownership of names is a separate job, and one the prototype has not started on. flow_alloc takes the name and a QoS; the IRMd and the IPCP do the rest.

Function	TCP/IP	Ouroboros
Finding the service	DNS, configured per host, mapping a host name to an address; the port is the application's problem	The directory of the Layer, a DHT, keyed by the hash of the service name
Several instances	A load balancer, an anycast address or DNS round robin, each a separate system	A name bound to several processes; a per-name policy, round robin or spillover, picks one
Starting the server	Always running, or a superserver per protocol	A program bound with auto is started when a flow arrives for its name
Knowing it is the right one	TLS, matching the certificate to the host name	Part of the flow allocation: the server's certificate must carry the service name
The service moves	The connection breaks; the application resolves again and reconnects	The next allocation finds it; a name can resolve to several places, and to different places in different Layers

Where the Locator/Identifier split puts this, and why Ouroboros does it with a single lookup, is on the functional layering page. Tutorial 02 and Tutorial 05 show anycast and load balancing.

Where do addresses go?

The problem

IP addresses are assigned to interfaces, causing issues with multihoming, especially over multiple networks. Every so often the argument pops up whether addresses should be assigned to interfaces or to nodes, a false dichotomy.

Ouroboros

In every system, there is one address per Layer the system is a member of, and that address belongs to the IPCP's forwarding components, not to any hardware. No application ever sees it, and two Layers can use overlapping addresses with no conflict to resolve. The links of the IPCP are flows from the layer below, or from the points of attachment it is attached to, up to sixteen of them. An IPCP attached to two Ethernet interfaces, or to Ethernet and UDP, has one address in its Layer, and when one of the links goes down, its flows move to the other. The exception is the transmission layer at the bottom, where Ethernet and IP addresses are what they are.

IP fragmentation

The problem

An application-layer message may span multiple TCP segments, which in turn may span several IP fragments. When a fragment is lost, TCP retransmits the whole segment, and with it every fragment that did arrive. IP fragmentation is mostly deprecated, and path MTU discovery was added to avoid it.

Ouroboros

The forwarding layer never fragments. Fragmentation is end to end, in FRCT, in the process that sends the message, and three things follow that IP fragmentation does not have. Each fragment is an ordinary FRCT packet with its own sequence number, so losing one fragment of a hundred retransmits one fragment, where losing one IP fragment of a hundred loses the datagram. Flow control is message-atomic: the sender waits for enough window to carry the whole message before sending the first fragment, so a message is never half sent into a window that then closes. And the size limit is known, not discovered: the flow allocator reports the MTU of the flow when it is allocated, so there is no probing, no black holes, and no reliance on ICMP messages that firewalls eat. Deriving the layer MTU from the path below, rather than from a per-layer setting, is on the to-do list.

TCP is byte-stream only

The problem

TCP sequence numbers are per byte instead of per packet. This is efficient for large bulk transfers, but application protocols that are message-based have to manage message boundaries themselves. QUIC uses two sequence number spaces, to solve the retransmission ambiguity, with the stream offset still per byte. SCTP is a packet-based reliable protocol, but rarely used. The application chooses the protocol (SOCK_STREAM or SOCK_SEQPACKET, IPPROTO_TCP or IPPROTO_SCTP) and with it the semantics.

Ouroboros

FRCT numbers packets, and uses fragmentation bits to preserve message boundaries. A message flow delivers what was written, one message per read; a stream flow chunks a byte stream on top of the same protocol. The application asks for one or the other as part of its QoS, and does not know of protocols. Every TCP application invents its own framing, a length prefix or a delimiter, and writes the same reassembly loop again; a message flow has the boundary already. Tutorial 08 runs a reliable flow over a lossy link.

Connections

The problem

TCP opens with SYN, SYN-ACK and ACK, and closes into TIME_WAIT for twice the maximum segment lifetime. Both exist to make sure the two ends agree on sequence numbers and that no packet from an earlier incarnation is read as a current one. The handshake costs a round trip before any data, on top of the round trips TLS adds, and TIME_WAIT costs a port for minutes after the connection is gone, which is where port exhaustion comes from.

The kernel also acknowledges on arrival, into a socket buffer the application has not looked at, and the advertised window reports the space left in that buffer, megabytes of it with autotuning. A sender can be told there is plenty of room while the receiving application is blocked, descheduled or wedged, and data can be acknowledged that the application will never see.

Ouroboros

FRCT has no handshake. A flow starts sending with a data run flag and a random sequence number, and correctness comes from bounded packet lifetime rather than from agreed initial state, Watson's delta-t: after three maximum packet lifetimes plus the timers, nothing from an earlier run can still exist. The round trip that TCP spends on its handshake is spent on flow allocation, which is where the security handshake rides too, so an authenticated and encrypted flow costs one round trip in total. There is no TIME_WAIT and there are no ports to exhaust.

FRCT runs in the application's process, in the library, and that changes what an acknowledgement means. The acknowledgement edge advances on arrival, as the question of retransmission is whether the packet got there; the window edge advances only when the application reads. An application that stops reading stops opening the window, and the peer finds out. There is no kernel keeping up appearances on its behalf: flow control describes the consumer, which is the only thing it was ever supposed to describe. The FRCT header carries no connection identifier, no ports and no options: demultiplexing happened in the Layer below, and the algorithm choices were made at allocation.

No notion of who

The problem

The Internet model has no idea who is talking to whom. Names are an afterthought, the network delivers any packet from anyone to any open port, and a source address is whatever the sender put there. Security is bolted on at every layer separately, Wi-Fi encryption, IPsec, TLS, application logins, and the gaps in between are where the attacks live: forged sources, amplification, port scanning, SYN floods. TLS, the one that applications use, sits above the transport, so a connection exists before anyone is authenticated, and every application carries its own copy of the problem.

Ouroboros

Authentication and key exchange are part of flow allocation, in the IRMd, below the application. A flow is authenticated, and if requested encrypted, before the application sees any data, and the same handshake serves application flows, IPCPs joining a Layer, and the flows an IPCP allocates over its points of attachment.

	TCP/IP with TLS 1.3	Ouroboros
Where it runs	In the application, above the transport	In the IRMd, inside flow allocation, below the application
Round trips before application data	Two, one for TCP and one for TLS 1.3; three with TLS 1.2; one with QUIC	One: the key exchange, the certificates and the application's own request ride in the flow request and reply, the exchange the network had to do anyway
Who is authenticated	The server; client certificates are rare	The server by default; the client when the server requires it
What the network learns	The server name travels in the clear unless ECH is deployed	The server's certificate, data and signature are sealed under a handshake key; under mutual authentication the client's certificate is signed but not sealed
What an unsolicited packet reaches	Any listening port; a SYN costs the server state	Nothing: there are no listening ports, endpoint ids are 64-bit random and per flow, and replies go to the stored peer
Renewing the keys	At the application's initiative	By the IRMd, every two minutes and after a packet count, with an anti-replay window on the data
Replay	Handled per protocol; 0-RTT data is replayable	A timestamp window and a replay cache in the handshake
Post-quantum	Hybrid key exchange in TLS deployments	The same hybrids, and ML-DSA and SLH-DSA signatures
Who else needs it	Each layer has its own: 802.1X, WPA, IPsec, TLS	One handshake for applications, Layers and points of attachment

Tutorial 06 sets up the certificates, and Tutorial 07 shows the handshake and the re-keying on the wire.

One transport, four jobs

The problem

TCP does four jobs at once, and conspicuously does not do a fifth. Three of the four are the business of the two endpoints. Congestion is not: it is a property of a path the endpoints do not own, cannot see, and have no authority over. TCP does it anyway, because in the split between IP and TCP there is nowhere else to put it. IP forwards packets and holds no per-flow state, TCP is the only layer with any state at all, and there is nothing in between.

Job	Whose concern	TCP/IP	Ouroboros
Retransmission	the two endpoints	TCP	FRCT, end to end, per flow, in the application's process
Ordering	the two endpoints	TCP	FRCT
Flow control	the receiver's buffer	TCP	FRCT
Fragmentation	whoever meets a smaller MTU	IP, and everyone avoids it	the library, above FRCT, per message
Congestion control	the network	TCP	the IPCP, inside the Layer, per path

Ouroboros

Reliability and congestion control are independent, and both combinations that TCP/IP lacks exist. A raw flow has no FRCT at all, no sequence numbers and no acknowledgments, and it is still congestion controlled, because the control is in the IPCP that forwards it; the TCP/IP equivalent is UDP, which is not and cannot be without a transport under it or a middlebox outside the architecture. The inverse, a reliable flow without congestion control, is what a Layer with the none policy gives, for a dedicated link where there is nothing to be fair to.

Loss is not congestion

The problem

TCP uses a lost packet for two things at once: it retransmits the packet, and it takes the loss as a sign of congestion and cuts its rate. On a link that drops packets for other reasons, a radio link or a lossy cable, it cuts its rate for nothing. Thirty years of work has gone into disentangling the two, delay-based controllers, ECN and BBR among them, each working around one signal carrying two meanings. The deeper causes are where the loop stands and what it controls: the sender infers a queue it cannot see, it controls a connection when the shared resource is a path, its signal is one bit because IP is one global layer that everyone has to agree on, its round-trip time is in the law, it is voluntary, and one loop has to be right for a datacentre fabric and a transatlantic backbone at once.

Ouroboros

The window, the rate of the path and the repair of losses are three limits in three places, each with its own signal. FRCT, in the processes at the ends of a flow, keeps the window and repairs the holes that the receiver reports; it never reads the ECN byte. The congestion controller, in the IPCP, paces the flows of a path at a rate it steers from the depth of the queues along the path; it never counts a lost packet.

	TCP/IP	Ouroboros
Unit of control	the connection: more connections, more bandwidth	the path: one controller per destination and QoS cube, shared by every flow to it; more flows get nothing
The signal	one ECN bit, agreed by the whole Internet	an ECN byte that belongs to the Layer: a graded measure of queue depth, the maximum along the path
Who reads the queue	the sender, by inference from loss and delay	the IPCP that owns the queue, at every hop
Round-trip time in the law	yes: a nearby flow beats a distant one	none: every path through a bottleneck reads the same queue depth and responds the same way
Who enforces	the sender, on itself; UDP does not take part	the packet scheduler of the IPCP, on every flow, raw ones included
Scope	one loop, end to end, for the whole path	one loop per Layer, over its own scope, chosen at bootstrap; the loops compose, since the inner Layer's flows are traffic to the outer one

Tutorial 09 shows flows sharing a bottleneck without a dropped packet, and Tutorial 10 runs the same lossy, delayed link under FRCT and under CUBIC, Reno, DCTCP and BBR.

For the reader who knows the field, the table puts the two Ouroboros loops next to the TCP congestion controls and two transports that pace on a queue rather than on loss: the Ultra Ethernet Transport, whose sender-side congestion control follows SMaRTT, and Homa, which hands the control to the receiver. The row of mb-ecn describes the IPCP as it is today, a process that reads the queue of its own point of attachment; in hardware it would be a switch that marks its outgoing queues, which is future work.

Transport	Congestion signal	Who reads the queue	What is controlled	A lost packet is	Round-trip time in the law
TCP CUBIC, Reno	loss	the sender, by inference	a window	congestion	yes: one step per round trip
DCTCP	one ECN bit, the fraction marked over a round trip	the switch, at a threshold	a window	congestion	yes
TCP Prague, L4S	one ECN bit at a shallow queue, the fraction marked over a round trip	the switch, in a queue of its own	a window	congestion, by requirement	yes, floored at 25 ms
BBRv1	bandwidth and minimum round-trip time estimated from the acknowledgments	the sender, by inference, pausing to probe	a paced rate under a window cap	repair only	yes: twice the bandwidth-delay product in flight
UET NSCC, SMaRTT	one ECN bit gating the delay above 1.5 times the base round trip; a trimmed packet as overload	the switch marks at dequeue, the sender turns delay into queue	a window, started at the bandwidth-delay product	a trim: overload; an untrimmed loss: repair only	yes: base round trip and bandwidth-delay product are inputs, gains scaled by them for fairness
Homa	grants from the receiver, shortest message first	the receiver, with the priority queues of the switch	the grants: a receiver-driven window	repair only, on the receiver's request	one round trip of unscheduled data, then the grants
Ouroboros, mb-ecn and FRCT	a graded mark, the log of the queue depth, integrated over a pricing window	the IPCP that owns the queue, at every hop	a paced rate per path, shared by its flows	repair only, by FRCT at the ends	none: a heartbeat round trip sets the time constants only

What the table does not show is where the queue is allowed to sit. The window-based transports tolerate a queue measured in round trips: SMaRTT aims at half a base round trip of delay, BBR keeps a round trip's worth of data in flight, CUBIC takes what the buffer gives. mb-ecn aims at a queue measured in packets.

Duplicate functionality between layers

The problem

Some mechanisms are present in several layers of the TCP/IP model: retransmission and flow control in the LLC sublayer of the data link and in the transport layer, name-to-address mapping in ARP and in DNS, security in about every layer.

Ouroboros

A functional model without duplication, in which one structure, the Layer, repeats, defined by scope rather than by function. The application end-to-end layer maps onto LLC and the network end-to-end layer onto MAC, so the data link layer is gone, and what is left of the transport layer is split in two; the functional layering page has the mapping. That the link layer is not needed was shown on hardware in 2016: a proof of concept put Ouroboros straight onto the wire of a NetFPGA 10G card, with a preamble, a start-frame delimiter and then the packet. No MAC addresses, no ethertype, no frame check sequence, and nothing above the point of attachment changed.

Configuration per layer

The problem

Every layer of the stack is configured by hand, in its own format, by its own specialists: interface addresses and routes, the resolver, the firewall, and in every application its listening port, its address family, its certificates and its TLS settings. An application carries network configuration it has no business knowing.

Ouroboros

Applications carry no network configuration. One file per machine tells the IRMd which Layers to create or join, which points of attachment to use, which names to serve and which programs to bind to them, and the security settings live per service name next to it. Every application on the machine inherits that. Tutorial 03 sets up a machine from the file.

Technological ambiguity and contradictions

The problem

A lot of technologies do not fit the five- or seven-layer model, such as MPLS (layer 2.5?) and the many encapsulations: VLANs, VPNs, tunnels, overlays. If IP is layer 3, how is ICMP, running on top of IP, also layer 3? If UDP is layer 4, what is QUIC? The five-layer model has some use as marketing terminology, but a model with so many logical contradictions has no place in a science curriculum.

Ouroboros

The repeating structure of the Layer readily accommodates tunnels and overlays without internal inconsistencies, duplication or contradictions: a Layer over a Layer is the same thing as a Layer over Ethernet. The prototype runs that way, as an overlay over Ethernet or UDP, so there is no need to chuck out your NICs and switches.