Ouroboros Tutorial 04

From Ouroboros
Jump to navigation Jump to search
Ouroboros Tutorial 04

In this tutorial, we build a unicast Layer over Ethernet, run oping over it, and look at the traffic on the wire. To make it easy to follow on a single machine, we attach both IPCPs to the loopback device, lo. At the end, we show the configuration for two machines on an Ethernet LAN. As in the previous tutorials, we use a debug build with default build options.

Ethernet points of attachment

In the previous tutorials, the unicast IPCPs reached each other with direct flows, because they ran on the same machine. IPCPs on different machines need a way to send packets to each other. In Ouroboros, that is a point of attachment (PoA): the unicast IPCP attaches directly to an Ethernet device and sends its packets in Ethernet frames, with its own Ethertype (0xA000 by default).[1] The PoA is identified by the device and the Ethertype, so different Layers can share the same wire by using different Ethertypes.

A PoA also provides a way to find peers on the wire: the IPCP can broadcast a query for a name, and the IPCP that answers to that name replies with its MAC address. We will use that to enroll and connect without configuring any MAC addresses.

The loopback device behaves like an Ethernet device, with two differences. It has no MAC address, so each PoA on lo makes up a random one. And every frame sent on lo comes back to every PoA on lo, including the one that sent it; each PoA drops its own frames.

The configuration file

We set up the Layer with a configuration file, as in Tutorial 03. Save the text below as /etc/ouroboros/tut04.conf: 

###  Ouroboros Tutorial 04
[name.oping-server]
prog=["/usr/bin/oping"]
args=["--listen"]

[unicast.uni-1]
bootstrap="unicast-layer"
autobind=true
eth=["lo"]
reg=["oping-server"]

[unicast.uni-2]
enrol={dst="unicast-layer", eth={}}
autobind=true
eth=["lo"]
conn=[{dst="uni-1", eth={}}]

The stanzas for the name and uni-1 are the configuration of Tutorial 03, with one addition: the eth option attaches uni-1 to lo with the default Ethertype. uni-1 bootstraps the Layer, binds to its own name and the Layer name (autobind), and registers oping-server, which the IRMd will start automatically when a flow request arrives.

uni-2 attaches to lo as well. It enrolls with an IPCP that answers to the Layer name unicast-layer, and then connects to uni-1. The eth={} tables say that the IRMd should find these peers with a query over the attached Ethernet PoAs. Without them, the IRMd would connect the two IPCPs with direct flows, as in Tutorial 03.

Security configuration

As explained in Tutorial 02, a client requires the server it allocates a flow to, to present a certificate, unless the client security configuration says otherwise. A flow over a PoA is no different: the IRMd looks for the client security configuration of the name the flow goes to. The IPCPs and the oping client allocate flows to unicast-layer, uni-1 and oping-server, so we create a client security configuration for these names, as in the previous tutorials: 

$ for n in unicast-layer uni-1 oping-server; do
>   sudo mkdir -p /etc/ouroboros/security/client/$n
>   printf 'auth=optional\nencryption=none\n' | sudo tee /etc/ouroboros/security/client/$n/sec.conf > /dev/null
> done

Starting the IRMd

To see the traffic, start tcpdump in a separate terminal before starting the IRMd, and capture all frames on lo with Ethertype 0xA000: 

$ sudo tcpdump -i lo ether proto 0xa000

Then start the IRMd with the configuration file: 

$ sudo irmd --stdout --config /etc/ouroboros/tut04.conf

The IRMd creates the name oping-server, as in Tutorial 03, and then creates uni-1 and attaches it to lo: 

==82131== irmd/configuration(DB): Found IPCP uni-1 in configuration file.
==82131== irmd(II): Instantiated /usr/sbin/ipcpd-unicast as process 82139.
==82131== irmd(II): Process added: 82139 (ipcpd-unicast).
==82139== ipcpd/ipcp(II): IPCP unicast 82139 initialized.
==82131== irmd(II): Created IPCP 82139.
==82139== ipcpd/ipcp(II): Attached eth lo 0xA000.
==82131== irmd(II): Attached IPCP 82139.

uni-1 then bootstraps the Layer, exactly as in Tutorial 02, binds its names and registers oping-server in the DHT. As uni-1 is the only member of the Layer, the DHT stores the entry locally and has no peers to publish it to: 

==82131== irmd(II): Bootstrapped IPCP 82139.
==82131== irmd(II): Created new name: uni-1.
==82131== reg/name(DB): Process 82139 accepting flows for uni-1.
==82131== irmd(II): Bound process 82139 to name uni-1.
==82131== irmd(II): Created new name: unicast-layer.
==82131== reg/name(DB): Process 82139 accepting flows for unicast-layer.
==82131== irmd(II): Bound process 82139 to name unicast-layer.
==82131== irmd/configuration(DB): Registering oping-server in 82139
==82131== reg(DB): Name oping-server already exists.
==82131== irmd(II): Name oping-server already exists.
==82139== ipcpd/ipcp(II): Registering f6c93ff2...
==82139== dht(DB): <f6c93ff23bd91094, 0000000081366d15> Adding entry (publish).
==82139== dht(DB): <f6c93ff23bd91094, 0000000081366d15> Adding lval.
==82139== dht(DB): <f6c93ff23bd91094, 0000000081366d15> Failed to get peer list.
==82139== ipcpd/ipcp(II): Finished registering f6c93ff2.
==82131== irmd(II): Registered oping-server with IPCP 82139 as f6c93ff2.

Next, the IRMd creates uni-2, attaches it to lo, and enrolls it. The IRMd finds an IPCP for the Layer name on the wire, and allocates a PoA flow to it. We left out the lines where uni-2 initializes its policies, which are the same as in Tutorial 02: 

==82131== irmd/configuration(DB): Found IPCP uni-2 in configuration file.
==82131== irmd(II): Instantiated /usr/sbin/ipcpd-unicast as process 82195.
==82131== irmd(II): Process added: 82195 (ipcpd-unicast).
==82195== ipcpd/ipcp(II): IPCP unicast 82195 initialized.
==82131== irmd(II): Created IPCP 82195.
==82195== ipcpd/ipcp(II): Attached eth lo 0xA000.
==82131== irmd(II): Attached IPCP 82195.
==82195== ipcpd/ipcp(II): Enrolling with unicast-layer...
==82195== unicast-ipcp(II): [ec5c041982c90ff5] Requesting enrollment.
==82131== irmd/oap(DB): [2194ec95575d7d3f] Preparing OAP request for unicast-layer.
==82131== irmd/oap(DB): [2194ec95575d7d3f] OAP request prepared for unicast-layer.
==82131== irmd(II): Allocating PoA flow 2 for 82195 to unicast-layer.
==82131== irmd(II): PoA flow request arrived for unicast-layer.
==82131== irmd(DB): IPCP 82139 accepting flow 1 for unicast-layer.
==82131== irmd/oap(DB): Processing OAP request for unicast-layer.
==82131== irmd/oap(DB): [2194ec95575d7d3f] No crt provided.
==82131== irmd/oap(II): [2194ec95575d7d3f] OAP request processed for unicast-layer.
==82139== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 1.
==82139== ipcpd/ipcp(II): Finished responding 0 on PoA flow 1.
==82131== irmd(II): Flow 1 accepted by 82139 for unicast-layer (uid 0).
==82139== connection-manager(II): Handling incoming flow 64.
==82131== reg/name(DB): Process 82139 accepting flows for unicast-layer.
==82131== reg/name(DB): Process 82139 accepting flows for uni-1.
==82131== irmd/oap(DB): [2194ec95575d7d3f] Completing OAP for unicast-layer.
==82131== irmd/oap(DB): [2194ec95575d7d3f] No crt provided.
==82131== irmd/oap(II): [2194ec95575d7d3f] OAP completed for unicast-layer.
==82131== irmd(II): PoA flow 2 allocated to unicast-layer.
==82195== connection-manager(DB): Sending OCEP info for protocol OEP to fd 64.
==82139== connection-manager(II): Request to connect to Enrollment.
==82139== connection-manager(II): Finished handling incoming flow 64 for Enrollment.
==82139== enrollment(II): Incoming enrollment connection on flow 64.
==82139== enrollment(II): [ec5c041982c90ff5] Handling incoming enrollment.
==82139== enrollment(DB): [ec5c041982c90ff5] Sending enrollment info (103 bytes).
==82195== enrollment(DB): [ec5c041982c90ff5] Received configuration info (103 bytes).
==82139== enrollment(II): [ec5c041982c90ff5] Enrollment completed.
==82131== irmd(II): Deallocating flow 1 for process 82139 (timeout: 34 s).
==82139== ipcpd/ipcp(II): Deallocating flow 1.
==82139== ipcpd/ipcp(II): Finished deallocating PoA flow 1.
==82131== irmd(II): Completed deallocation of flow_id 1 by process 82139.
==82139== enrollment(II): Enrollment flow 64 closed.
==82131== irmd(II): Deallocating flow 2 for process 82195 (timeout: 34 s).
==82195== ipcpd/ipcp(II): Deallocating flow 2.
==82195== ipcpd/ipcp(II): Finished deallocating PoA flow 2.
==82131== irmd(II): Completed deallocation of flow_id 2 by process 82195.

The OAP request is prepared for unicast-layer, the name uni-2 enrolls with, and the request arrives for that name at uni-1, the IPCP that answered the query and is bound to it. uni-1 hands the flow to its enrollment component, as in Tutorial 02. The IRMd serves both ends of the flow, so it has a flow ID for each end: 2 for uni-2, and 1 for uni-1. When the enrollment is done, each IPCP deallocates its end of the flow.

The enrollment then completes as in Tutorial 02, and the IRMd binds uni-2 to its names: 

==82195== unicast-ipcp(II): [ec5c041982c90ff5] Enrolled with unicast-layer.
==82195== ipcpd/ipcp(II): Finished enrolling with unicast-layer in layer unicast-layer.
==82195== ipcpd/ipcp(II):   type: unicast
==82195== ipcpd/ipcp(II):   hash: SHA3-256 [32 bytes]
==82131== irmd(II): Enrolled IPCP 82195 in layer unicast-layer.
==82131== irmd(II): Created new name: uni-2.
==82131== reg/name(DB): Process 82195 accepting flows for uni-2.
==82131== irmd(II): Bound process 82195 to name uni-2.
==82131== reg/name(DB): Process 82195 accepting flows for unicast-layer.
==82131== irmd(II): Bound process 82195 to name unicast-layer.

Finally, the IRMd connects uni-2 to uni-1. The management flow is again a PoA flow, found with a query for uni-1: 

==82131== irmd(DB): Connecting Management to uni-1.
==82195== ipcpd/ipcp(II): Connecting Management to uni-1...
==82131== irmd/oap(DB): [1f2798d39f38e746] Preparing OAP request for uni-1.
==82131== irmd/oap(DB): [1f2798d39f38e746] OAP request prepared for uni-1.
==82131== irmd(II): Allocating PoA flow 1 for 82195 to uni-1.
==82131== irmd(II): PoA flow request arrived for uni-1.
==82131== irmd(DB): IPCP 82139 accepting flow 3 for uni-1.
==82131== irmd/oap(DB): Processing OAP request for uni-1.
==82131== irmd/oap(DB): [1f2798d39f38e746] No crt provided.
==82131== irmd/oap(II): [1f2798d39f38e746] OAP request processed for uni-1.
==82139== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 3.
==82139== ipcpd/ipcp(II): Finished responding 0 on PoA flow 3.
==82131== irmd(II): Flow 3 accepted by 82139 for uni-1 (uid 0).
==82139== connection-manager(II): Handling incoming flow 64.
==82131== reg/name(DB): Process 82139 accepting flows for unicast-layer.
==82131== reg/name(DB): Process 82139 accepting flows for uni-1.
==82131== irmd/oap(DB): [1f2798d39f38e746] Completing OAP for uni-1.
==82131== irmd/oap(DB): [1f2798d39f38e746] No crt provided.
==82131== irmd/oap(II): [1f2798d39f38e746] OAP completed for uni-1.
==82131== irmd(II): PoA flow 1 allocated to uni-1.
==82195== connection-manager(DB): Sending OCEP info for protocol LSP to fd 64.
==82139== connection-manager(II): Request to connect to Management.
==82139== connection-manager(II): Finished handling incoming flow 64 for Management.
==82195== link-state-routing(DB): Type mgmt neighbor 15.6d.36.81 added.
==82139== link-state-routing(DB): Type mgmt neighbor 59.ce.de.45 added.
==82195== ipcpd/ipcp(II): Finished connecting.
==82131== irmd(II): Established Management connection between IPCP 82195 and uni-1.

The data transfer flow follows in the same way. You can list the PoAs of each IPCP with the irm tool. It shows the type, device, Ethertype and the MAC address the PoA made up: 

$ irm ipcp poa list name uni-1
eth    lo 0xA000 8e:a8:49:f8:b0:e6
$ irm ipcp poa list name uni-2
eth    lo 0xA000 76:f3:91:bf:5e:bb
The unicast Layer over lo. Each IPCP has an Ethernet PoA, and the management and data transfer flows are PoA flows.

Pinging the server

Now run the oping client as a regular user:[2] 

$ oping -n oping-server -c 3
Pinging oping-server with 64 bytes of data (3 packets):

64 bytes from oping-server: seq=0 time=0.731 ms
64 bytes from oping-server: seq=1 time=0.705 ms
64 bytes from oping-server: seq=2 time=0.871 ms

--- oping-server ping statistics ---
3 packets transmitted, 3 received, 0 out-of-order, 0% packet loss, time: 3001.036 ms
rtt min/avg/max/mdev = 0.705/0.769/0.871/0.089 ms

The IRMd starts the oping server and first tries a direct flow. The server runs as root and the client does not, so they do not share a packet pool, and the IRMd falls back to the unicast Layer, with a new OAP request. uni-2 looks up oping-server in the DHT, finds the address of uni-1, and sends the flow allocation request over the data transfer flow. uni-1 passes it to the IRMd, which hands it to the server: 

==82131== irmd(II): Allocating flow for 82275 to oping-server (uid 967).
==82131== irmd(II): Instantiated /usr/bin/oping as process 82278.
==82131== irmd(II): Starting /usr/bin/oping for oping-server.
==82131== irmd(II): Process added: 82278 (oping).
==82131== reg/name(DB): Process 82278 accepting flows for oping-server.
==82131== irmd/oap(DB): [c8700f46635b60be] Preparing OAP request for oping-server.
==82131== irmd/oap(DB): [c8700f46635b60be] OAP request prepared for oping-server.
==82131== irmd(DB): UID mismatch, falling back.
==82195== dht(DB): K<f6c93ff23bd91094> No local values.
==82139== dht(II): K<f6c93ff23bd91094> Responding with 1 contacts, 1 values.
==82195== dht(DB): K<f6c93ff23bd91094> 1 new values received.
==82131== irmd/oap(DB): [ce17f3172af1c8f3] Preparing OAP request for oping-server.
==82131== irmd/oap(DB): [ce17f3172af1c8f3] OAP request prepared for oping-server.
==82195== ipcpd/ipcp(II): Allocating flow 8 for 82275 to f6c93ff2.
==82195== dht(DB): K<f6c93ff23bd91094> No local values.
==82195== ipcpd/ipcp(II): Finished allocating flow 8 to f6c93ff2.
==82131== irmd(DB): Flow req arrived from IPCP 82139 for f6c93ff2.
==82131== irmd(II): Flow request arrived for oping-server.
==82131== irmd(DB): IPCP 82278 accepting flow 7 for oping-server.
==82131== irmd/oap(DB): Processing OAP request for oping-server.
==82131== irmd/oap(DB): [ce17f3172af1c8f3] No crt provided.
==82131== irmd/oap(II): [ce17f3172af1c8f3] OAP request processed for oping-server.
==82139== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 7.
==82139== ipcpd/ipcp(II): Finished responding 0 to allocation request.
==82195== flow-allocator(DB): IPCP received msg response 0 for flow on fd 66.
==82131== irmd(II): Flow 7 accepted by 82278 for oping-server (uid 0).
==82131== irmd(DB): Response for flow 8 to oping-server.
==82131== irmd/oap(DB): [ce17f3172af1c8f3] Completing OAP for oping-server.
==82131== irmd/oap(DB): [ce17f3172af1c8f3] No crt provided.
==82131== irmd/oap(II): [ce17f3172af1c8f3] OAP completed for oping-server.

On the wire

Every frame starts with the Ethernet header: the destination and source MAC addresses and the Ethertype, which tcpdump shows on the first line of each frame. In our capture, uni-1 has MAC address 8e:a8:49:f8:b0:e6 and uni-2 has MAC address 76:f3:91:bf:5e:bb. The hex dump starts right after the Ethernet header, with the 5-byte PoA header: 

uint16_t eid;    endpoint ID of the PoA flow, 0 is the PoA management channel
uint16_t len;    length of the payload
uint8_t  hcs;    CRC-8 checksum over eid and len

The length field is needed because Ethernet pads short frames (runt frames) with zeroes, and the receiver must know where the payload ends.

The frames in this tutorial, after the Ethernet header: a management message, a ping and an encrypted ping.

Finding the peer

The first two frames are the query for the Layer name and its answer: 

08:28:06.909375 76:f3:91:bf:5e:bb (oui Unknown) > Broadcast, ethertype Unknown (0xa000), length 99:
        0x0000:  0000 0050 5a04 0000 0000 0000 0000 0000  ...PZ...........
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 2023 958d a531 0c98 d039 8d50  .....#...1...9.P
        0x0040:  acfc 3fac 986f 44e7 bb8c 73e4 31a6 157f  ..?..oD...s.1...
        0x0050:  252d e4dd 7b                             %-..{
08:28:06.909432 8e:a8:49:f8:b0:e6 (oui Unknown) > 76:f3:91:bf:5e:bb (oui Unknown), ethertype Unknown (0xa000), length 99:
        0x0000:  0000 0050 5a05 0000 0000 0000 0000 0000  ...PZ...........
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 2023 958d a531 0c98 d039 8d50  .....#...1...9.P
        0x0040:  acfc 3fac 986f 44e7 bb8c 73e4 31a6 157f  ..?..oD...s.1...
        0x0050:  252d e4dd 7b                             %-..{

Both are PoA management messages (eid 0000, length 0050 = 80 bytes). A management message has a fixed format of 48 bytes, followed by the message data: 

uint8_t  code;           1 = FLOW_REQ, 2 = FLOW_REPLY, 3 = FLOW_UPDATE, 4 = NAME_QUERY, 5 = NAME_REPLY
uint8_t  resv[3];
uint32_t s_eid;          EID of the flow at the sender
uint32_t d_eid;          EID of the flow at the receiver
int32_t  response;       result of a flow allocation
uint64_t bandwidth;      QoS of the flow
uint32_t delay;
uint32_t loss;
uint32_t ber;
uint32_t max_gap;
uint32_t timeout;
uint8_t  availability;
uint8_t  service;
uint16_t data_len;       length of the data after this header

The first frame is a NAME_QUERY (code 04), sent by uni-2 to the broadcast address. Its data (data_len 0020 = 32 bytes) is the SHA3-256 hash of unicast-layer, which starts right after it: 23 95 8d a5 ... dd 7b. uni-1 is bound to that name, so it sends a NAME_REPLY (code 05) with the same hash back to uni-2. From the source address of that frame, uni-2 now knows the MAC address of uni-1.

Allocating the PoA flow

Next, uni-2 sends a FLOW_REQ to that MAC address, and uni-1 answers with a FLOW_REPLY: 

08:28:06.912571 76:f3:91:bf:5e:bb (oui Unknown) > 8e:a8:49:f8:b0:e6 (oui Unknown), ethertype Unknown (0xa000), length 135:
        0x0000:  0000 0074 9501 0000 0000 0000 0100 0000  ...t............
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0003  ................
        0x0020:  e800 0000 0000 0000 0000 0007 d000 01d4  ................
        0x0030:  c000 0100 4423 958d a531 0c98 d039 8d50  ....D#...1...9.P
        0x0040:  acfc 3fac 986f 44e7 bb8c 73e4 31a6 157f  ..?..oD...s.1...
        0x0050:  252d e4dd 7b21 94ec 9557 5d7d 3ff7 9a96  %-..{!...W]}?...
        0x0060:  9a49 073f 1018 d920 28e7 6ee1 2700 0000  .I.?....(.n.'...
        0x0070:  0002 a000 0000 0000 00                   .........
08:28:06.914044 8e:a8:49:f8:b0:e6 (oui Unknown) > 76:f3:91:bf:5e:bb (oui Unknown), ethertype Unknown (0xa000), length 135:
        0x0000:  0000 0074 9502 0000 0000 0000 0100 0000  ...t............
        0x0010:  0100 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 4421 94ec 9557 5d7d 3ff7 9a96  ....D!...W]}?...
        0x0040:  9a49 073f 1018 d920 28e7 a247 0600 0000  .I.?....(..G....
        0x0050:  0000 0000 0000 0000 00e7 9334 dc28 7822  ...........4.(x"
        0x0060:  ec32 ca57 b5ea 3770 6e7b 68f9 6756 d721  .2.W..7pn{h.gV.!
        0x0070:  f6f8 755a bca9 7480 b1                   ..uZ..t..

The request carries uni-2's EID for the flow (s_eid = 1), and the QoS of the enrollment flow: a delay of 1000 ms, no loss (0), a maximum gap of 2000 ms, a timeout of 120000 ms (0001 d4c0) and service 1, a message flow with FRCT. As no loss is allowed, the flow is reliable. The 68 bytes of data (data_len 0044) are the SHA3-256 hash of unicast-layer, the same as in the query, followed by the 36-byte OAP request: the 16-byte OAP ID, of which the logs show the first 8 bytes (2194 ec95 575d 7d3f), a timestamp, the identifiers of the cipher and the key derivation function (both zero, so no encryption) and of the signature digest (02a0, SHA-256), and the lengths of the certificate, the key exchange and the data, which are all zero. The reply carries uni-1's EID (s_eid = 1), echoes uni-2's (d_eid = 1), and has response 0: success. Its data is the OAP response, which adds a 32-byte hash of the request. Tutorial 07 explains OAP in detail.

The PoA flow is now allocated, and both sides use its EID in the PoA header. The first frame on EID 1 is uni-2 connecting to the enrollment component: 

08:28:06.914887 76:f3:91:bf:5e:bb (oui Unknown) > 8e:a8:49:f8:b0:e6 (oui Unknown), ethertype Unknown (0xa000), length 62:
        0x0000:  0001 002b 49c3 002e 6a00 0000 00f6 e35d  ...+I...j......]
        0x0010:  3600 0000 000a 0a45 6e72 6f6c 6c6d 656e  6......Enrollmen
        0x0020:  7412 034f 4550 1801 2800 4000 5ed1 74f7  t..OEP..(.@.^.t.

The enrollment flow is reliable, so the payload starts with a 16-byte header for the Flow and Retransmission Control Protocol (FRCT), followed by the OCEP message. You can read the component name (Enrollment) and the protocol (OEP) in the ASCII column. The management flow and the data transfer flow are set up with the same exchange of query, request and reply. The management flow uses EID 1 again, as the enrollment flow is released by then, and the data transfer flow gets EID 2.

Finding the peer and allocating the enrollment flow on the wire. The sizes are those of the frames in the capture.

The ping

All traffic between the unicast IPCPs now goes over the data transfer flow, EID 2: the DHT lookup for oping-server, the flow allocation, and the pings themselves. This is the first ping request: 

08:28:17.875507 76:f3:91:bf:5e:bb (oui Unknown) > 8e:a8:49:f8:b0:e6 (oui Unknown), ethertype Unknown (0xa000), length 99:
        0x0000:  0002 0050 4615 6d36 8100 3c00 0042 0000  ...PF.m6..<..B..
        0x0010:  0061 de0f 0200 0000 0000 0000 006b 2200  .a...........k".
        0x0020:  0000 0000 00c9 3140 3000 0000 0000 0000  ......1@0.......
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 00                             .....

After the PoA header (eid 0002, length 0050 = 80 bytes), there is a 16-byte header of the data transfer protocol (DTP): 

uint8_t  dst_addr[4];  15 6d 36 81               address of uni-1
uint8_t  qos_cube;     00                        best effort
uint8_t  ttl;          3c                        60, the maximum
uint8_t  ecn;          00                        no congestion
uint8_t  cap;          00                        link capacity code, 0 is unknown
uint64_t eid;          42 00 00 00 61 de 0f 02   endpoint ID of the flow at uni-1

These fields are in host byte order: the address bytes are printed in the same order in the logs (Flat address initialized to 15.6d.36.81). The flow allocator assigns 64-bit EIDs, with a random number in the upper 32 bits and the IPCP's flow descriptor for the flow in the lower 32 bits (0x42 = 66). The remaining 64 bytes are the oping message: 

uint32_t type;     0000 0000            0, a ping request
uint32_t id;       0000 0000            0, the sequence number
uint64_t tv_sec;   6b22 0000 0000 0000  monotonic clock time of the client
uint64_t tv_nsec;  c931 4030 0000 0000

followed by 40 bytes of zeroes, to make the default payload of 64 bytes. The timestamp is only read by the client to compute the round-trip time, so it is in host byte order. The reply goes back to the address of uni-2 (59 ce de 45), to uni-2's EID for the flow, with type 1 (a ping reply) and the same timestamp: 

08:28:17.875838 8e:a8:49:f8:b0:e6 (oui Unknown) > 76:f3:91:bf:5e:bb (oui Unknown), ethertype Unknown (0xa000), length 99:
        0x0000:  0002 0050 4659 cede 4500 3c00 0042 0000  ...PFY..E.<..B..
        0x0010:  00bb 2373 8e00 0000 0100 0000 006b 2200  ..#s.........k".
        0x0020:  0000 0000 00c9 3140 3000 0000 0000 0000  ......1@0.......
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 00                             .....

Between the pings, you will also see 94-byte frames on EID 2. These are heartbeats and acknowledgements between the flow allocators, which the congestion avoidance uses to measure the round-trip time. The small 25-byte frames on EID 1 are link-state updates on the management flow.

Encrypted flows

Our security configuration for oping-server disabled encryption. Let's remove that line, and only keep auth=optional: 

$ printf 'auth=optional\n' | sudo tee /etc/ouroboros/security/client/oping-server/sec.conf

and ping once more: 

$ oping -n oping-server -c 1
Pinging oping-server with 64 bytes of data (1 packets):

64 bytes from oping-server: seq=0 time=2.825 ms

--- oping-server ping statistics ---
1 packets transmitted, 1 received, 0 out-of-order, 0% packet loss, time: 1001.404 ms
rtt min/avg/max/mdev = 2.825/2.825/2.825/NaN ms

The IRMd now enables encryption, and performs a key exchange between the client and the server side during the flow allocation. By default, this is an ephemeral Diffie-Hellman exchange on the prime256v1 curve, and the flow is encrypted with AES-256 in GCM mode: 

==82131== irmd(II): Allocating flow for 82307 to oping-server (uid 967).
==82131== irmd/oap(DB): [e6f2acf0118a6c71] Preparing OAP request for oping-server.
==82131== irmd/oap(II): Encryption enabled for oping-server.
==82131== irmd/oap(DB): [e6f2acf0118a6c71] Generated ephemeral prime256v1 keys (91 bytes).
==82131== irmd/oap(DB): [e6f2acf0118a6c71] OAP request prepared for oping-server.
==82131== irmd(DB): UID mismatch, falling back.
==82195== dht(DB): K<f6c93ff23bd91094> No local values.
==82131== irmd/oap(DB): [07013fae7619ed34] Preparing OAP request for oping-server.
==82131== irmd/oap(II): Encryption enabled for oping-server.
==82131== irmd/oap(DB): [07013fae7619ed34] Generated ephemeral prime256v1 keys (91 bytes).
==82131== irmd/oap(DB): [07013fae7619ed34] OAP request prepared for oping-server.
==82195== ipcpd/ipcp(II): Allocating flow 8 for 82307 to f6c93ff2.
==82195== dht(DB): K<f6c93ff23bd91094> No local values.
==82195== ipcpd/ipcp(II): Finished allocating flow 8 to f6c93ff2.
==82131== irmd(DB): Flow req arrived from IPCP 82139 for f6c93ff2.
==82131== irmd(II): Flow request arrived for oping-server.
==82131== irmd(DB): IPCP 82278 accepting flow 9 for oping-server.
==82131== irmd/oap(DB): Processing OAP request for oping-server.
==82131== irmd/oap(DB): [07013fae7619ed34] No crt provided.
==82131== irmd/oap(DB): [07013fae7619ed34] Selected client cipher aes-256-gcm.
==82131== irmd/oap(DB): [07013fae7619ed34] Selected client KDF sha256.
==82131== irmd/oap(II): [07013fae7619ed34] Negotiated prime256v1 + aes-256-gcm.
==82131== irmd/oap(DB): [07013fae7619ed34] Generated prime256v1 ephemeral keys.
==82131== irmd/oap(II): [07013fae7619ed34] OAP request processed for oping-server.
==82139== ipcpd/ipcp(II): Responding 0 to alloc on flow_id 9.
==82139== ipcpd/ipcp(II): Finished responding 0 to allocation request.
==82131== irmd(II): Flow 9 accepted by 82278 for oping-server (uid 0).
==82195== flow-allocator(DB): IPCP received msg response 0 for flow on fd 66.
==82131== irmd(DB): Response for flow 8 to oping-server.
==82131== irmd/oap(DB): [07013fae7619ed34] Completing OAP for oping-server.
==82131== irmd/oap(DB): [07013fae7619ed34] Accepted server cipher aes-256-gcm.
==82131== irmd/oap(DB): [07013fae7619ed34] Accepted server KDF sha256.
==82131== irmd/oap(DB): [07013fae7619ed34] DHE: derived shared secret.
==82131== irmd/oap(II): [07013fae7619ed34] Negotiated prime256v1 + aes-256-gcm.
==82131== irmd/oap(DB): [07013fae7619ed34] No crt provided.
==82131== irmd/oap(II): [07013fae7619ed34] OAP completed for oping-server.
The key exchange of an encrypted flow. The OAP request and response carry the public keys, each side of the IRMd derives the same key and hands it to its oping process, and the IPCPs forward packets they cannot read.

On the wire, the flow allocation request is now 253 bytes instead of 162, as it carries the client's ephemeral public key (91 bytes). The ping request looks like this: 

08:28:23.944348 76:f3:91:bf:5e:bb (oui Unknown) > 8e:a8:49:f8:b0:e6 (oui Unknown), ethertype Unknown (0xa000), length 121:
        0x0000:  0002 0066 7915 6d36 8100 3c00 0043 0000  ...fy.m6..<..C..
        0x0010:  00b4 7a5b 0300 0000 0000 00e4 8d63 e93b  ..z[.........c.;
        0x0020:  c898 bd8b 64f7 6736 7b4b ec45 ee06 818e  ....d.g6{K.E....
        0x0030:  80cf e6bc f26e 1416 7ca7 f0d5 778f 30fb  .....n..|...w.0.
        0x0040:  cc61 2095 754a c4be f922 4050 78fb 8f20  .a..uJ..."@Px...
        0x0050:  14e0 a6fd 6b12 ceee 4e2f 1824 0794 f64d  ....k...N/.$...M
        0x0060:  19f2 3f38 9ba4 1208 6538 b2              ..?8....e8.

The DTP header is still readable, as the unicast IPCPs need it to deliver the packet. The payload is now 86 bytes instead of 64: a 6-byte key selector (0000 0000 0000 here), which tells the receiver which key encrypted the packet, followed by the 64 encrypted bytes of the oping message and a 16-byte authentication tag. The IRMd derives the key from the key exchange and hands it to the two oping processes, which encrypt and decrypt the packets themselves; the unicast IPCPs forward the packet without being able to read it. Tutorial 06 shows how to add authentication with certificates.

If you run the metrics exporter during this tutorial, the PoA statistics show the queries and the traffic. These panels come from a separate run, with 20 pings instead of 3, and 20 more after encryption is enabled:

The dotted lines mark the start of the plain pings and of the encrypted pings. uni-2 sent three name queries, for the enrollment flow, the management flow and the data transfer flow, and uni-1 answered all three. Each IPCP keeps two PoA flows. During the encrypted pings, every ping is 22 bytes longer on the wire.

Two machines

To run this tutorial on two machines on the same Ethernet LAN, split the configuration file. We call the machine with the oping server the server, and the other one the client. Replace eth0 with the name of your Ethernet device (ip link lists them). On the server: 

###  Ouroboros Tutorial 04, server machine
[name.oping-server]
prog=["/usr/bin/oping"]
args=["--listen"]

[unicast.uni-1]
bootstrap="unicast-layer"
autobind=true
eth=["eth0"]
reg=["oping-server"]

On the client: 

###  Ouroboros Tutorial 04, client machine
[name.oping-server]

[unicast.uni-2]
enrol={dst="unicast-layer", eth={}}
autobind=true
eth=["eth0"]
conn=[{dst="uni-1", eth={}}]

The client configuration also creates the name oping-server, without a program. The IRMd only allocates flows to names it knows; without this stanza, the oping client would fail with Failed to allocate flow: -1007, and the IRMd would log Name oping-server does not exist. On a single machine, the stanza for the server did that for us.

The client IPCP and the oping client run on the client, so the client security configuration for unicast-layer, uni-1 and oping-server goes on the client. The server does not allocate any flows, so it needs no security configuration. Start the IRMd on the server first, and then on the client. The traffic is the same as on lo, with the real MAC addresses of the machines.

Notes

  1. ↑ Before version 0.25, Ethernet was a separate IPC process type (the ipcpd-eth-dix and ipcpd-eth-llc) with its own Layer underneath the unicast Layer. These are replaced by points of attachment, and the LLC variant is gone.
  2. ↑ As in the previous tutorials, we assume the oping client does not run as root or as a member of the ouroboros group. Otherwise, the client and the server share a packet pool, the IRMd creates a direct flow, and the pings will not show up on lo. If you want to run the client as root, you can build Ouroboros with -DDISABLE_DIRECT_IPC=ON.
Retrieved from "https://ouroboros.rocks/mediawiki/index.php?title=Ouroboros_Tutorial_04&oldid=2061"