Ouroboros Functional Layering
The Ouroboros model is the result of countless refinements made while implementing the prototype, and the implementing is not over. It is a layered model with a twist. Every layer has a well-defined service interface and appears as a black box to the layers around it, and the layers that make up the network repeat. The model treats broadcast as distinct from unicast.
The unicast model has five layers: the application layer, the application end-to-end layer, the network end-to-end layer, the network forwarding layer and the transmission layer. Broadcast replaces forwarding with a network flooding layer.
Applications request network services through the Unicast API and the Broadcast API. For unicast, the application end-to-end layer talks to the network end-to-end layer through the Network IPC API, which the application never sees: it is the demarcation line between "the application" and "the network". The Broadcast API talks straight to the network flooding layer. A network, unicast or broadcast, is itself a distributed application, so it inherits the application end-to-end layer in turn. That is what gives the model its repeating structure.
This page describes the model as it stands today and sets it next to the TCP/IP Internet model, the Locator/Identifier split and the Recursive InterNetwork Architecture (RINA).
Unicast model
Unicast API
The Unicast API lets an application create, manage and destroy unicast flows, and read from and write to them. It is network-agnostic, offers synchronous and asynchronous inter-process communication, and carries either messages or byte streams.
Application end-to-end layer
The application end-to-end layer makes the packets on a flow reliable and private. A unicast flow is initiated by a client process towards a server process, which is identified by a service name.
It can provide:
- encryption with the symmetric key that flow allocation hands it;
- reliability, implemented by FRCT: fragmentation, in-order delivery, duplicate suppression, Automatic Repeat reQuest (ARQ) and flow control;
- integrity, through hash-based checks such as CRC32;
- liveness monitoring.
Everything in that list is optional. A flow without FRCT is a raw flow.
Below it, the application end-to-end layer uses the Network IPC API to reach the network end-to-end layer.
Network end-to-end layer, or flow allocator
The network end-to-end layer creates a network flow between two unicast IPCPs, the source and the destination, in a suitable Unicast Layer. That network flow carries a client flow between two end-user processes, the client and the server, which live on the same systems as their IPCPs. The layer is usually called the flow allocator, after the IPCP component that implements it.
Its core functions are:
- name-to-address resolution: given a service name, find the address of an IPCP that can serve as the destination. The directory holds this mapping for the layer;
- flow allocation: create the shared state between the source and destination IPCP for a flow;
- multiplexing: generate a local endpoint identifier for the flow and map it to the peer's address;
- Congestion avoidance: watch the network flow for congestion and police its throughput as needed.
Flow establishment, authentication and symmetric key distribution are each a two-way handshake. When the MTU allows, authentication and key exchange piggyback on the flow establishment request and reply, so all three complete within a single round-trip time. The exchange also maps the QoS the application asked for onto a network traffic class, and it carries the application-level request and response inside its own, so that application-level and network-level flow allocation together fit in one round trip. See Flow Allocation for the details.
Authentication lives on the boundary between the two end-to-end layers rather than in either of them. The IRMd, which implements the Network IPC API, runs the handshake on behalf of both: it holds the credentials, verifies the peer against the certificate for the service name, in both directions if required, and hands the resulting symmetric key to the application end-to-end layer, while the flow allocator only carries the exchange. The same boundary is where an IPCP authenticates when it enrols in a Layer, through the flow it allocates over the Layer below, and where a process could one day authenticate with a Broadcast Layer when it joins one.
The network end-to-end layer is the interface for the application end-to-end layer above it. The two always go hand in hand.
Network forwarding layer
The network forwarding layer forwards Ouroboros Data Transfer Protocol packets from the source IPCP to the destination IPCP, based on their address and QoS class. The forwarding function takes the destination address and picks the flow or flows to forward on, usually from a forwarding table. Building that table needs distance information at every IPCP, which is the job of the routing dissemination function. The forwarding layer never fragments. Fragmentation is an end-to-end matter, done in FRCT, with the flow allocator reporting the path MTU.
Transmission layer
At the bottom sits the transmission layer, the abstraction of a point-to-point channel whose operation is completely independent of the rest of Ouroboros. It can be a wire protocol over a physical medium (copper, radio, machine memory) or a network technology such as Ethernet, IP, UDP or Bluetooth, which gives Ouroboros-over-Ethernet and Ouroboros-over-UDP.
The transmission layer is best seen as a special case of the forwarding or flooding layer, used to build a 0-Layer that stops the recursion. It comes with its own, tailored network end-to-end layer, because the application end-to-end layer above always needs one. For a dumb link a minimal flow allocator will do; when the transmission layer wraps a legacy network, it may pay to have every feature of a full network end-to-end layer.
In the prototype this 0-Layer is not a separate process. Ethernet and UDP are points of attachment (PoAs): the transmission layer and its minimal flow allocator live inside the IPCP that uses them, attached with irm ipcp poa attach. A PoA resolves peers, assigns endpoint identifiers and runs the flow allocation handshake, authentication included. It hands its flows to the IPCP's application end-to-end layer through the same Network IPC API as any Layer, so encryption and reliability on those flows come from that layer, and nothing above can tell a PoA from a Layer. An IPCP can attach to several PoAs at once. The one 0-Layer that is still an IPCP of its own is the local IPCP, which connects processes on the same machine through memory.
Broadcast model
The broadcast model adds two things to the unicast model: the Broadcast API and the network flooding layer.
Broadcast API
The Broadcast API lets an application join and leave broadcast flows, and read from and write to them. Like the Unicast API it is network-agnostic, with synchronous and asynchronous primitives and message or stream delivery. QoS on broadcast flows is inherently more limited than on unicast flows. A broadcast flow maps directly onto a Broadcast Layer. Joining one is not authenticated today; the Broadcast API is where that would be added.
Network flooding layer
The network flooding layer floods packets from the data transfer flow they arrive on to all other data transfer flows. In essence it is stateless. It sits on an application end-to-end layer that provides the data transfer flows forming the links of the Broadcast Layer. Those links typically form a tree, though any directed acyclic graph will do.
Layers and recursive networking
An application end-to-end layer always pairs with a network end-to-end layer, and the forwarding and flooding layers are part of a network application. The unicast model therefore has a sandwich structure, ham and cheese, that can be stacked as often as needed. If the Network IPC API between the application and network end-to-end layers is one universal standard, every Layer (capital L) in the sandwich can be moved around at will. That is a recursive network. The Ouroboros model hints strongly at recursion, but does not require it.
Unicast Layer
A Unicast Layer consists of three-layer applications. Each implements a network end-to-end layer and a network forwarding layer, and inherits the application end-to-end layer. Such an application is a Unicast IPCP. The IPCPs in a Layer are equals in mechanism, that is in which functionality they implement, but not identical, because each mechanism can be realised by a different policy. Routing can be link-state in one Layer and path-vector in another. Even within a Layer the IPCPs may differ, although some policies have to be the same Layer-wide.
Broadcast Layer
A Broadcast Layer consists of two-layer applications that implement a network flooding layer and inherit the application end-to-end layer. Such an application is a Broadcast IPCP.
Multicast is not a separate concept in the model. It is two steps: enrol a Broadcast IPCP in a Broadcast Layer whose other IPCPs sit on the systems hosting the would-be multicast group, then let the application use that Broadcast Layer.
0-Layers
A unicast 0-Layer consists of two-layer applications with a network end-to-end layer, possibly tailored and limited, that faces the applications above through the Unicast API. The prototype has two kinds: the local IPCP, a 0-Layer over machine memory, and the Ethernet and UDP points of attachment, which fold the 0-Layer into the IPCP that attaches to them.
A broadcast 0-Layer is a two-layer application with a tailored network flooding layer that faces broadcast applications through the Broadcast API. The figures leave it out. It is the trivial case of an Ouroboros application over a legacy broadcast technology, such as flooding over memory or a wrapper around Ethernet or IP broadcast. In the prototype, broadcast IPCPs attach to the same Ethernet and UDP points of attachment as unicast IPCPs, so they run over a unicast 0-Layer rather than a broadcast one.
Relation to the TCP/IP model
Main page: Ouroboros and TCP/IP
Set next to the five-layer Internet model of the TCP/IP stack, four differences stand out:
- the transport layer is replaced by two end-to-end layers, one on the application side and one on the network side;
- the network layer has two counterparts, the forwarding layer for unicast and the flooding layer for broadcast;
- the data link layer is gone;
- the physical layer has become the transmission layer.
Splitting the transport layer
The functions of the transport layer, TCP and UDP, move into two independent layers. The application end-to-end layer takes ARQ and flow control; the network end-to-end layer takes multiplexing, name-to-address resolution and congestion control. Authentication sits on the boundary between the two, as part of flow allocation, where TCP/IP leaves it to the application. The split resembles the sublayering of the data link layer into Logical Link Control and Media Access Control. Name-to-address resolution, roughly what a DNS SRV lookup does, lands inside the network end-to-end layer instead of being an optional chore for the application. The Network IPC API takes a service name and resolves it internally, whereas the TCP/IP transport layer wants an IP address and a port and leaves the lookup to the application.
Untangling the network layer
The network layer functionality is much the same in both models. The differences are that broadcast is split off, and that the forwarding layer holds no reference to anything above it: there is no "protocol" field.
No data link layer
The Internet model splits the data link layer into LLC and MAC. The application end-to-end layer maps directly onto LLC, and the network end-to-end layer onto MAC (name-to-address mapping, multiplexing), so there is no reason left for a distinct data link layer.
A transmission layer instead of a physical layer
The physical layer is generalised to include virtual and legacy networks, a more general "end of the line" than a physical medium. What sets it apart is that it does not reuse the Ouroboros API and so does not interface with the application end-to-end layer. Its functionality is tailored to the technology it wraps.
Relation to the Locator/Identifier split
Loc/Id split starts from the observation that an application is tied to a network address through the transport layer and a port. That hampers mobility: when the IP address changes, the TCP connection breaks. The proposal is to split the network address semantically into an identifier, location-independent, the who at the transport layer, and a locator, location-dependent, the where at the network layer. A 128-bit IPv6 address has room for both.
Architecturally this is a name-to-address resolution step, identifier to locator, in a sub-layer[1] between the transport layer, bound to the identifier, and the network layer, which uses the locator. Changing locator on the move no longer breaks the transport connection. The solution works, but its premise, that IP addresses are semantically overloaded[2], is in our view mistaken. The real problem is a missing service name, with DNS providing only synonyms for addresses. DNS SRV is a step towards naming services, but it still maps those names straight to an address and a port, and the application binds to those rather than to the name. Both pieces of the puzzle exist; they only need putting together.
Taken as a whole, a Loc/Id application does two lookups: domain name to identifier, then identifier to locator. That indirection keeps Loc/Id compatible with the TCP/IP stack, but from the Ouroboros perspective it is redundant. It is more efficient to bind applications to service names and resolve a service name to an address in a single lookup.
Ouroboros splits the transport layer and puts service-name resolution in the network end-to-end layer. The differences with the Loc/Id sub-layer are that the network end-to-end layer also takes congestion control and multiplexing from the transport layer, and that what remains, the application end-to-end layer, is considered part of the application.
Relation to RINA
Main page: Ouroboros and RINA
The original goal of the prototype was a portable user-space implementation of RINA for POSIX systems that could be tailored to embedded devices. It started as an implementation of the principles RINA lays out, so the Ouroboros model inherits many concepts and terms from RINA, and that is where the credit for them goes. As the prototype evolved, the architecture was changed in ways that simplified it further.
Splitting data transfer control from data transfer
RINA reads the IP fragmentation problem as proof that the TCP/IP split was wrong[3]. Its answer is to keep the functions of TCP and IP in one logical layer and one protocol, the Error and Flow Control Protocol (EFCP). EFCP has two component protocols, the Data Transfer Protocol (DTP) and the Data Transfer Control Protocol (DTCP), but they share a state vector and are not independent.
While implementing EFCP in Ouroboros, we found that its functionality, and so that of TCP plus IP, could and therefore should be split not in two but in three independent layers: application end-to-end, network end-to-end and network forwarding. The IP fragmentation problem is solved by putting fragmentation in FRCT rather than in the data transfer protocol, with the flow allocator reporting the path MTU of a flow as input to the fragmentation function.
Layers versus DIFs
In RINA, DTP and DTCP live in the same layer. Since end-user programs do not normally forward packets, there is a hard line between ordinary distributed applications and the distributed application responsible for reliable packet transfer. A process that implements EFCP is an Inter-Process Communication process (IPCP), a term that originated in Livermore's LINCS network, and the distributed application made of IPCPs is a Distributed IPC Facility (DIF). An ordinary distributed application consists of distributed application processes (DAPs) and is a Distributed Application Facility (DAF).
The three-way split raised the question where FRCT belongs: in the DIF, or in the application? Putting it in the application immediately resolved several questions that come with RINA:
- Multi-homing of end-user applications, flows over different networks, is still a problem in RINA: only IPCPs multi-home efficiently. In Ouroboros every process can.
- DTCP state in an IPCP, sequence numbers and windows, is hard to recover after that IPCP crashes, even when the application process survives.
- Reliable flow allocation over a shim DIF, which the IPCP needs, required a hack where the IPCP allocates a flow over itself.
- If a DIF is needed for reliable IPC between two programs, who provides reliable IPC between those programs and the DIF? Answering that the OS provides test-and-set is unsatisfying, as it rests on reasonable but non-trivial assumptions outside the model.
With the application end-to-end layer, the distinction between DIF and DAF disappears from the Ouroboros model. For want of better terminology, Ouroboros still calls the processes that make up a Layer IPCPs, fully aware of the confusion that may cause.
Notes on layering
The figures show Broadcast Layers over Unicast Layers, and Unicast Layers over Unicast Layers. What about the other combinations?
Unicast over broadcast
In the general case, where some data packets are flooded to several next-hop IPCPs, unicast IPCPs would have to tell whether an arriving packet is for them. That scales poorly, its utility within Ouroboros is unclear, and it needs more study, so it is not part of the general model. A Unicast Layer may still use dedicated Broadcast Layers for specific functions; routing dissemination in the forwarding layer is one example. Ethernet does implement unicast over a broadcast domain, and a separate page discusses how Ethernet maps onto Ouroboros.
Broadcast over broadcast
One Broadcast Layer using another is technically feasible. Its utility is also unclear, so it is not elaborated here, but allowing it would barely touch the model.
Mixed implementations
Why not merge broadcast and unicast into a single Layer? Anyone is free to implement an IPCP that does both. The result is a Layer with a broadcast network and a unicast network at the full scope of that Layer, which may suit some use cases. From the model perspective, however, unicast and broadcast are always two distinct Layers, and mixing the implementation does not change that. Likewise, one program can implement several IPCP modules internally instead of running a process per IPCP.