Ouroboros Tutorial 05
In this tutorial, we build a unicast Layer over UDP, and show how the IRMd spreads flow allocation requests over several server processes. To make it easy to follow on a single machine, both IPCPs use the loopback address, 127.0.0.1. At the end, we show the configuration for two machines on an IP network, such as a LAN or the Internet. As in the previous tutorials, we use a debug build with default build options.
UDP points of attachment
In Tutorial 04, the unicast IPCPs sent their packets in Ethernet frames. A unicast IPCP can also attach to a UDP socket, and send its packets in UDP datagrams.[1] The UDP PoA is identified by an IP address and a UDP port, 3435 by default. Two IPCPs on the same machine need a different port.
Unlike an Ethernet PoA, a UDP PoA does not look for its peers with a query. We tell the IPCP the IP address, or the host name, and port of the peer to reach.
The configuration file
Save the text below as /etc/ouroboros/tut05.conf:
### Ouroboros Tutorial 05
[name.oping-server]
prog=["/usr/bin/oping"]
lb="round-robin"
[name.oping-server2]
prog=["/usr/bin/oping"]
[unicast.uni-1]
bootstrap="unicast-layer"
autobind=true
udp=["127.0.0.1"]
reg=["oping-server", "oping-server2"]
[unicast.uni-2]
enrol={dst="unicast-layer", udp="127.0.0.1"}
autobind=true
udp=["127.0.0.1:3436"]
conn=[{dst="uni-1", udp="127.0.0.1"}]
This time, we create two names for our oping server, oping-server and oping-server2, and bind the oping program to both of them, but without args. The IRMd will not start the servers itself: we start them ourselves in a moment. The lb option sets how the IRMd picks a process when more than one process accepts flows for a name. For oping-server, it takes turns (round-robin). oping-server2 keeps the default, spill, which gives the flow to the process that accepted a flow last, as long as it is available.
The stanzas for the IPCPs are those of Tutorial 04, with UDP instead of Ethernet. uni-1 attaches to 127.0.0.1 with the default port 3435, and uni-2 to 127.0.0.1 with port 3436. uni-2 enrolls with the Layer name and connects to uni-1, both at 127.0.0.1, port 3435.
Security configuration
As in the previous tutorials, the clients need a security configuration for the names they allocate flows to: the IPCPs to unicast-layer and uni-1, and the oping client to our two server names:
$ for n in unicast-layer uni-1 oping-server oping-server2; do
> sudo mkdir -p /etc/ouroboros/security/client/$n
> printf 'auth=optional\nencryption=none\n' | sudo tee /etc/ouroboros/security/client/$n/sec.conf > /dev/null
> doneStarting the IRMd
To see the traffic, start tcpdump in a separate terminal before starting the IRMd, and capture the datagrams on the two ports:
$ sudo tcpdump -i lo -nn -X udp port 3435 or udp port 3436Then start the IRMd with the configuration file:
$ sudo irmd --stdout --config /etc/ouroboros/tut05.confThe IRMd creates uni-1 and attaches it to UDP:
==82417== irmd/configuration(DB): Found IPCP uni-1 in configuration file.
==82417== irmd(II): Instantiated /usr/sbin/ipcpd-unicast as process 82423.
==82417== irmd(II): Process added: 82423 (ipcpd-unicast).
==82423== ipcpd/ipcp(II): IPCP unicast 82423 initialized.
==82417== irmd(II): Created IPCP 82423.
==82423== ipcpd/ipcp(II): Attached udp4 127.0.0.1:3435.
==82417== irmd(II): Attached IPCP 82423.The rest of the setup is the same as in Tutorial 04. uni-2 enrolls with unicast-layer, and connects to uni-1 for its management flow and its data transfer flow. These lines of the log show the three PoA flows:
==82417== irmd(II): Allocating PoA flow 2 for 82481 to unicast-layer.
==82417== irmd(II): PoA flow request arrived for unicast-layer.
==82417== irmd(II): PoA flow 2 allocated to unicast-layer.
==82417== irmd(II): Allocating PoA flow 1 for 82481 to uni-1.
==82417== irmd(II): PoA flow request arrived for uni-1.
==82417== irmd(II): PoA flow 1 allocated to uni-1.
==82417== irmd(II): Allocating PoA flow 5 for 82481 to uni-1.
==82417== irmd(II): PoA flow request arrived for uni-1.
==82417== irmd(II): PoA flow 5 allocated to uni-1.The irm tool lists the PoAs of each IPCP:
$ irm ipcp poa list name uni-1
udp4 127.0.0.1:3435
$ irm ipcp poa list name uni-2
udp4 127.0.0.1:3436
Starting the servers
Now open two terminals, and start an oping server in each:[2]
$ sudo oping -l
Ouroboros ping server started.The oping program is bound to both names, so each oping process accepts flows for both oping-server and oping-server2 as soon as it starts:
==82417== irmd(II): Process added: 82547 (oping).
==82417== reg/name(DB): Process 82547 accepting flows for oping-server2.
==82417== reg/name(DB): Process 82547 accepting flows for oping-server.
==82417== irmd(II): Process added: 82553 (oping).
==82417== reg/name(DB): Process 82553 accepting flows for oping-server2.
==82417== reg/name(DB): Process 82553 accepting flows for oping-server.The irm tool shows the names and their load-balancing policy:
$ irm name list
+----------------------------------------------------+----------------------+
| name | load-balance policy |
+----------------------------------------------------+----------------------+
| unicast-layer | spillover |
| uni-2 | spillover |
| uni-1 | spillover |
| oping-server2 | spillover |
| oping-server | round-robin |
+----------------------------------------------------+----------------------+Round robin
Now ping oping-server four times, with one ping each time. The -Q option only prints the statistics:
$ oping -n oping-server -c 1 -Q
--- oping-server ping statistics ---
1 packets transmitted, 1 received, 0 out-of-order, 0% packet loss, time: 1000.547 ms
rtt min/avg/max/mdev = 0.512/0.512/0.512/NaN msThe IRMd hands the four flows to the two servers in turn, 82547 and 82553:
==82417== irmd(II): Flow 7 accepted by 82547 for oping-server (uid 0).
==82417== irmd(II): Flow 8 accepted by 82553 for oping-server (uid 0).
==82417== irmd(II): Flow 10 accepted by 82547 for oping-server (uid 0).
==82417== irmd(II): Flow 11 accepted by 82553 for oping-server (uid 0).Each server terminal shows two new flows.
Spill
Next, ping oping-server2 four times in the same way. All four flows go to the same server:
==82417== irmd(II): Flow 13 accepted by 82553 for oping-server2 (uid 0).
==82417== irmd(II): Flow 14 accepted by 82553 for oping-server2 (uid 0).
==82417== irmd(II): Flow 15 accepted by 82553 for oping-server2 (uid 0).
==82417== irmd(II): Flow 9 accepted by 82553 for oping-server2 (uid 0).After a server accepts a flow, it waits for the next one, and it goes back to the front of the line: the IRMd keeps giving new flows to the same process for as long as it is waiting. If you mix the pings to oping-server and oping-server2, you will see that spill does not guarantee that a name always goes to the same process: it is the process that accepted a flow last, for either name. To send all flows for a name to one process, bind only that process to the name, instead of the program:
$ irm bind process <pid> name <name>
On the wire
Every datagram starts with the IPv4 header (20 bytes) and the UDP header (8 bytes), which tcpdump shows on the first line. After these headers, a UDP PoA only adds a 4-byte header:
uint32_t eid; endpoint ID of the PoA flow, 0 is the PoA management channel
There is no need for a length field or a checksum, as UDP has both. The rest is the same as over Ethernet.
This is the FLOW_REQ that uni-2 sends to enroll:
08:47:40.714696 IP 127.0.0.1.3436 > 127.0.0.1.3435: UDP, length 120
0x0000: 4500 0094 daad 4000 4011 61a9 7f00 0001 E.....@.@.a.....
0x0010: 7f00 0001 0d6c 0d6b 0080 fe93 0000 0000 .....l.k........
0x0020: 0100 0000 0000 0001 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 03e8 0000 0000 ................
0x0040: 0000 0000 0000 07d0 0001 d4c0 0001 0044 ...............D
0x0050: 2395 8da5 310c 98d0 398d 50ac fc3f ac98 #...1...9.P..?..
0x0060: 6f44 e7bb 8c73 e431 a615 7f25 2de4 dd7b oD...s.1...%-..{
0x0070: eda0 178e ec47 2a06 6990 66db d7b4 76cd .....G*.i.f...v.
0x0080: 18d9 213a 336f f7ba 0000 0000 02a0 0000 ..!:3o..........
0x0090: 0000 0000 ....After the headers (at 0x001c), the EID is 0: a management message. The code is 01, FLOW_REQ, and its data (data_len 0044 = 68 bytes) start with the SHA3-256 hash of unicast-layer (23 95 8d a5 ... dd 7b), followed by the OAP request, exactly as in Tutorial 04. This is a ping request from the client to oping-server:
08:47:53.776637 IP 127.0.0.1.3436 > 127.0.0.1.3435: UDP, length 84
0x0000: 4500 0070 db86 4000 4011 60f4 7f00 0001 E..p..@.@.`.....
0x0010: 7f00 0001 0d6c 0d6b 005c fe6f 0000 0002 .....l.k.\.o....
0x0020: ec19 354a 003c 0000 4200 0000 d261 91b9 ..5J.<..B....a..
0x0030: 0000 0000 0000 0000 0327 0000 0000 0000 .........'......
0x0040: f6c8 5d2a 0000 0000 0000 0000 0000 0000 ..]*............
0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................It goes over the data transfer flow, with EID 2, followed by the DTP header and the 64-byte oping message.
Two machines
To run this tutorial on two machines, split the configuration file. We call the machine with the oping servers the server, and the other one the client. Replace the addresses with the IP addresses of your machines. On the server:
### Ouroboros Tutorial 05, server machine
[name.oping-server]
prog=["/usr/bin/oping"]
lb="round-robin"
[name.oping-server2]
prog=["/usr/bin/oping"]
[unicast.uni-1]
bootstrap="unicast-layer"
autobind=true
udp=["192.168.0.2"]
reg=["oping-server", "oping-server2"]
On the client:
### Ouroboros Tutorial 05, client machine
[name.oping-server]
[name.oping-server2]
[unicast.uni-2]
enrol={dst="unicast-layer", udp="192.168.0.2"}
autobind=true
udp=["192.168.0.3"]
conn=[{dst="uni-1", udp="192.168.0.2"}]
Both IPCPs now use the default port 3435, as they are on different machines; make sure a firewall between them lets UDP port 3435 through. Instead of an IP address, the udp option of enrol and conn also accepts a host name, such as udp="server.example.com", which the IRMd resolves. As in Tutorial 04, the client configuration creates the names of the servers, without a program, and the client security configuration goes on the client. Start the IRMd on the server first, and then on the client, and start the oping servers on the server.
Notes
- ↑ Before version 0.25, UDP was a separate IPC process type (the ipcpd-udp) with its own Layer underneath the unicast Layer. It found the IP address for a name by looking up the hash of that name in DNS, or in /etc/hosts. The unicast IPCPs now attach to UDP directly, and find names in the directory of the unicast Layer.
- ↑ As in Tutorial 04, we run the servers as root and the oping client as a regular user, so the flows go over the Layer. If they all run as the same user, the IRMd creates direct flows, and nothing shows up on lo.