Ouroboros Tutorial 02
The description of this tutorial uses a debug build.
The objective for this tutorial is to create a 2-layer network. We will add a simple 2-node unicast Layer on top of a local Layer and will ping an oping server from an oping client over this network.
We recommend opening a couple of terminal windows for this tutorial.
First, we will create a local loopback network like in the previous tutorial:
$ irm ipcp bootstrap type local name local-ipcp layer local-layer
Now we will create the 2-node unicast network on top of this Local Layer 'network'.
To create this small unicast network, we bootstrap the first IPCP; and then enroll a second IPCP with that first IPCP.
$ irm ipcp bootstrap type unicast name uni-1 layer unicast-layer
Let's quickly have a look at the IRMd log, which shows the following output:
==07907== irmd(II): Created IPCP 8030. ==08030== ipcpd/ipcp(II): Bootstrapping... ==08030== unicast-ipcp(II): IPCP got address 3241766228. ==08030== ca(DB): Using multi-bit ECN. ==08030== link-state-routing(DB): Using link state routing policy. ==08030== pff(DB): Using simple PFF policy. ==08030== pff(DB): Using simple PFF policy. ==08030== pff(DB): Using simple PFF policy. ==08030== ipcpd/ipcp(II): Finished bootstrapping: 0. ==07907== irmd(II): Bootstrapped IPCP 8030 in layer unicast-layer. ==07907== irmd(DB): New instance (8030) of ipcpd-unicast added. ==07907== irmd(DB): This process accepts flows for:
The IRMd has created our first IPCP, in our case it has pid 8030 and got address 3241766228. It is using the default policies: explicit congestion notification (ECN), flat addressing, and link-state routing with a simple forwarding function. Other forwarding functions currently implemented are loop-free alternates and equal-cost multi-path. The directory is a Distributed Hash Table (DHT). It shows 3 PFF policies as we support 3 levels of QoS, each with its own independent packet scheduler.
After bootstrapping the IPCP, we see that it has no bound names yet. To be able to enroll, the second IPCP will need to allocate a flow to the first one (over the local Layer), and to be able to allocate the flow, the first IPCP will need to be bound to a name that is registered into the local Layer.
When creating layers, we will leverage Ouroboros' anycast: instead of binding the IPCP to a single name, we will bind it to two names: the first name will be unique for each IPCP and can be used to create flows to that specific IPCP, the second name will be common for all IPCPs in the layer. This is useful for enrollment, as the new IPCP can enroll with any IPCP in the network. Let's bind and register these names now:
$ irm bind ipcp uni-1 name uni-1 $ irm bind ipcp uni-1 name unicast-layer $ irm name register uni-1 layer local-layer $ irm name register unicast-layer layer local-layer
With the first IPCP reachable, let's create and enroll the second one. Just like the bootstrap command, the enroll command will create an IPCP if there is no IPCP with that name yet in the system:
$ irm ipcp enroll name uni-2 layer unicast-layer autobind
This command will do a number of things
- create a unicast IPCP with name uni-2, as there is no IPCP with that name yet.
- allocate a flow from uni-2 to the "unicast-layer". IPCP uni-1 is the only process that accepts these flows.
- over this flow, create a connection with the enrollment component
- enroll uni-2 with uni-1: Uni-1 will send its configuration (default policies), and uni-2 will configure with these policies.
- deallocate the flow used for enrollment
- bind the process to the unique name uni-2 and the layer name unicast-layer [autobind]
Let's break it down further using the IRMd log output:
==07907== irmd(II): Created IPCP 8139. ==08139== ipcpd/ipcp(II): Enrolling with unicast-layer... ==08139== unicast-ipcp(II): [773fbf64335df4fc] Requesting enrollment. ==07907== irmd(II): Allocating flow for 8139 to unicast-layer. ==07930== ipcpd/ipcp(II): Allocating flow 0 to 23958da5. ==07930== ipcpd-local(DB): Allocating flow to 23958da5 on fd 64. ==07907== irmd(DB): Flow req arrived from IPCP 7930 for 23958da5. ==07907== irmd(II): Flow request arrived for unicast-layer. ==07930== ipcpd-local(II): Pending local allocation request on fd 64. ==07930== ipcpd/ipcp(II): Finished allocating flow 0 to 23958da5: 0. ==07930== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 1. ==07907== irmd(II): Flow on flow_id 0 allocated. ==07930== ipcpd-local(II): Flow allocation completed, fds (64, 65). ==07930== ipcpd/ipcp(II): Finished responding to allocation request: 0 ==07907== irmd(II): Flow on flow_id 1 allocated.
The first part shows the creation of the IPCP and the flow allocation from uni-2 to uni-1. Enrollment requests receive an random 64-bit ID so they can be traced more simply in the logs (we will do the same with flow allocation requests in the future). The explanation for the flow allocation procedure can be found in tutorial 01.
With this flow established, the unicast IPCP will connect to the Enrollment component to get its policy configuration using the Ouroboros Enrollment Protocol (OEP). This connection establishment is taken care of by the IPCP's connection-manager:
==08139== connection-manager(DB): Sending cacep info for protocol OEP to fd 64. ==08030== connection-manager(II): Handling incoming flow 64 ==08030== connection-manager(II): Request to connect to Enrollment. ==08030== connection-manager(II): Finished handling incoming flow 64 for Enrollment: 0.
The first log line above is from uni-2: it sends the connection establishment information for protocol OEP. The next three log lines are from uni-1: it notices the request is for Enrollment (OEP) and hands the flow over to this component inside uni-1, which sends the configuration information blob to uni-2.
==08030== enrollment(II): Incoming enrollment connection on flow 64. ==08030== enrollment(II): [773fbf64335df4fc] Handling incoming enrollment. ==08030== enrollment(DB): [773fbf64335df4fc] Sending enrollment info (63 bytes). ==08139== enrollment(DB): [773fbf64335df4fc] Received configuration info (63 bytes).
The uni-2 IPCP can now start its internals using this configuration. This is identical to the bootstrap procedure, except it uses the information received from an existing node in the network:
==08139== unicast-ipcp(II): IPCP got address 1978304124. ==08139== ca(DB): Using multi-bit ECN. ==08139== link-state-routing(DB): Using link state routing policy. ==08139== pff(DB): Using simple PFF policy. ==08139== pff(DB): Using simple PFF policy. ==08139== pff(DB): Using simple PFF policy. ==08030== enrollment(II): [773fbf64335df4fc] Enrollment completed.
Now that uni-2 has configured, using (random) address 1978304124 and started its internals, it will deallocate the flow:
==07907== irmd(DB): Deallocating flow 0 for process 8139. ==07907== irmd(DB): Partial deallocation of flow_id 0 by process 8139. ==07907== irmd(DB): Deallocating flow 1 for process 8030. ==07907== irmd(DB): Partial deallocation of flow_id 1 by process 8030. ==07930== ipcpd/ipcp(II): Deallocating flow 0. ==07930== ipcpd/ipcp(II): Deallocating flow 1. ==07907== irmd(DB): Deallocating flow 0 for process 7930. ==07907== irmd(DB): Deallocating flow 1 for process 7930. ==07907== irmd(II): Completed deallocation of flow_id 0 by process 7930. ==07930== ipcpd-local(II): Flow with fd 64 deallocated. ==07930== ipcpd/ipcp(II): Finished deallocating flow 0: 0. ==08139== unicast-ipcp(II): [773fbf64335df4fc] Enrolled with unicast-layer. ==08139== ipcpd/ipcp(II): Finished enrolling with unicast-layer: 0. ==07907== irmd(II): Enrolled IPCP 8139 in layer unicast-layer. ==07907== irmd(II): Completed deallocation of flow_id 1 by process 7930. ==07930== ipcpd-local(II): Flow with fd 65 deallocated. ==07930== ipcpd/ipcp(II): Finished deallocating flow 1: 0. ==08030== enrollment(II): Enrollment flow 64 closed.
The autobind option will bind the process for uni-2 to its unique name (uni-2) and the layer name (unicast-layer). This is shorthand for the 2 bind operations that we performed explicitly when bootstrapping uni-1. We could have used this autobind there as well.
==07907== irmd(II): Bound process 8139 to name uni-2. ==07907== irmd(II): Bound process 8139 to name unicast-layer.
Note that the irmd logs will also show that the 2 IPCPs are now accepting flows:
==07907== irmd(DB): New instance (8139) of ipcpd-unicast added. ==07907== irmd(DB): This process accepts flows for: ==07907== irmd(DB): New instance (8030) of ipcpd-unicast added. ==07907== irmd(DB): This process accepts flows for: ==07907== irmd(DB): unicast-layer ==07907== irmd(DB): uni-1
The debug log for uni-1 will now also show the 2 names it's bound to. For uni-2 the binding happened after the process called accept(), so these are not logged in our case.
Now we have two IPCP that are enrolled with eachother. In order to transfer packets, they need to have a forwarding adjacency, which we call a data transfer flow for the Ouroboros Data Transfer Protocol. We will also need to disseminate the routing information between the routing components. We call these management flows (this will be deprecated, so we won't go into detail on the management flow). The link-state protocol in the IPCP uses these management flows to send link-state updates on any data transfer flows in the network.
$ irm ipcp connect name uni-2 dst uni-1 comp mgmt $ irm ipcp connect name uni-2 dst uni-1 comp dt
Let's go through the establishment of the data transfer flow (forwarding adjacency):
==08139== ipcpd/ipcp(II): Connecting Data Transfer to uni-1... ==07907== irmd(II): Allocating flow for 8139 to uni-1. ==07930== ipcpd/ipcp(II): Allocating flow 2 to 291186bb. ==07930== ipcpd-local(DB): Allocating flow to 291186bb on fd 66. ==07907== irmd(DB): Flow req arrived from IPCP 7930 for 291186bb. ==07907== irmd(II): Flow request arrived for uni-1. ==07930== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 3. ==07930== ipcpd-local(II): Pending local allocation request on fd 66. ==07930== ipcpd/ipcp(II): Finished allocating flow 2 to 291186bb: 0. ==07907== irmd(II): Flow on flow_id 2 allocated. ==07930== ipcpd-local(II): Flow allocation completed, fds (66, 67). ==07930== ipcpd/ipcp(II): Finished responding to allocation request: 0 ==07907== irmd(II): Flow on flow_id 3 allocated. ==08139== connection-manager(DB): Sending cacep info for protocol dtp to fd 65. ==08030== connection-manager(II): Handling incoming flow 65 ==08030== connection-manager(II): Request to connect to Data Transfer. ==08030== connection-manager(II): Finished handling incoming flow 65 for Data Transfer: 0. ==08139== link-state-routing(DB): Type dt neighbor 3241766228 added. ==07907== irmd(DB): New instance (8030) of ipcpd-unicast added. ==07907== irmd(DB): This process accepts flows for: ==07907== irmd(DB): unicast-layer ==07907== irmd(DB): uni-1 ==08030== link-state-routing(DB): Type dt neighbor 1978304124 added. ==08139== dt(DB): Added fd 65 to packet scheduler. ==08030== dt(DB): Added fd 65 to packet scheduler. ==08139== dht(DB): Enrollment of DHT completed. ==08139== ipcpd/ipcp(II): Finished connecting: 0. ==07907== irmd(II): Established Data Transfer connection between IPCP 8139 and uni-1.
The establishment of the management and data transfer flows is the exact same procedure as the establishment of the enrollment flow, and also handled by the connection manager. A Distributed Hash Table is used to store names registered in this 2-node unicast Layer.
Now that the network is up-and-running, we can start configuring the oping service. We will use the name oping-server, so let's register it in our unicast Layer. We use the CLI shorthand feature, which will match the commands with the characters provided. So, instead of typing irm name register, we can use irm name reg or even irm n r:
$ irm n r oping-server layer unicast-layer
This will register the name oping-server with both IPCPs in our network. In a real network scenario using multiple servers, this command would only register with the IPCPs on that server. The default hash is again an SHA3-256, just as the local IPCP.
==07907== irmd(II): Created new name: oping-server. ==08030== ipcpd/ipcp(II): Registering f6c93ff2... ==08030== ipcpd/ipcp(II): Finished registering f6c93ff2 : 0. ==07907== irmd(II): Registered oping-server with IPCP 8030 as f6c93ff2. ==08139== ipcpd/ipcp(II): Registering f6c93ff2... ==08139== ipcpd/ipcp(II): Finished registering f6c93ff2 : 0. ==07907== irmd(II): Registered oping-server with IPCP 8139 as f6c93ff2.
In the first tutorial, we first started the server and then bound that process to the oping-server name. Here, we will it a bit differently: will bind the program to that name.
$ irm b prog oping n oping-server auto -- --listen
The command irm b prog is CLI shorthand for irm bind program. It will look for the specified binary in some default locations.
==07907== irmd(II): Bound program /usr/bin/oping to name oping-server.
In our case, it found oping in /usr/bin/oping.
When we now start the oping server using the oping --listen command, we will see the following in the IRMd log:
==07907== irmd(DB): Process 8259 inherits name oping-server from program oping. ==07907== irmd(DB): New instance (8259) of oping added. ==07907== irmd(DB): This process accepts flows for: ==07907== irmd(DB): oping-server
Whenever we start this binary, the IRMd will automatically bind the service name for the program to that new process. In our case, 8259 was the server process.
Finally, when we start the oping client to send a few pings:
$ oping -n oping-server -c 4 Pinging oping-server with 64 bytes of data (4 packets):
64 bytes from oping-server: seq=0 time=1.058 ms 64 bytes from oping-server: seq=1 time=0.821 ms 64 bytes from oping-server: seq=2 time=0.891 ms 64 bytes from oping-server: seq=3 time=0.944 ms
--- oping-server ping statistics --- 4 packets transmitted, 4 received, 0 out-of-order, 0% packet loss, time: 4002.468 ms rtt min/avg/max/mdev = 0.821/0.928/1.058/0.100 ms
The IRMd logs the process between the two unicast IPCPs:
==07907== irmd(DB): Process 8290 inherits name oping-server from program oping. ==07907== irmd(II): Allocating flow for 8290 to oping-server. ==08030== ipcpd/ipcp(II): Allocating flow 4 to f6c93ff2 ==08030== ipcpd/ipcp(II): Finished allocating flow 4 to f6c93ff2: 0. ==07907== irmd(DB): Flow req arrived from IPCP 8139 for f6c93ff2. ==07907== irmd(II): Flow request arrived for oping-server. ==08139== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 5. ==08139== ipcpd/ipcp(II): Finished responding to allocation request: 0 ==07907== irmd(II): Flow on flow_id 5 allocated. ==07907== irmd(II): Flow on flow_id 4 allocated.
Our client process 8290 (which, because it is /usr/bin/oping will bind to the name oping-server. But don't worry, since it doesn't call flow_accept() this process is not network-reachable. The client allocates the flow via uni-1 (process 8030) to uni-2 (process 8139) and packets will travel over the data transfer connection between uni-1 and uni-2.
Ouroboros will always use the lowest layer in the network to establish flows. To demonstrate this, see what happens if we register the name oping-server in the local layer:
$ irm n r oping-server layer local-layer
and then run the oping client again. We see that the ipcpd-local is now used instead of the unicast Layer:
==07907== irmd(DB): Process 9822 inherits name oping-server from program oping. ==07907== irmd(II): Allocating flow for 9822 to oping-server. ==07930== ipcpd/ipcp(II): Allocating flow 4 to f6c93ff2. ==07930== ipcpd-local(DB): Allocating flow to f6c93ff2 on fd 68. ==07907== irmd(DB): Flow req arrived from IPCP 7930 for f6c93ff2. ==07907== irmd(II): Flow request arrived for oping-server. ==07930== ipcpd-local(II): Pending local allocation request on fd 68.
A small note on performance: don't worry too much about the latencies being hundreds of microseconds. That's because of CPU scaling and the system not doing much (and a little bit because we are using a debug build). Latencies drop to a few microseconds if the system is under load. To see this for yourself, you can run:
$ oping -n oping-server -c 10000 --interval 0 --quiet --- oping-server ping statistics --- 10000 packets transmitted, 10000 received, 0 out-of-order, 0% packet loss, time: 617.447 ms rtt min/avg/max/mdev = 0.024/0.048/0.487/0.018 ms
There are a lot of things we can do to get these latencies to the lowest possible number, but those numbers usually have little relevance to real-world performance.