Ouroboros Tutorial 04
In this tutorial, we show how to run oping between two machines connected to the same Ethernet LAN. It requires OpenSSL to be installed before building O7s to enable encrypted flows, and as usual we use a debug build to show some extra log output.
For the second machine we use a Raspberry Pi Model 4B+ (ARM64, 32-bit Raspbian 11) connected to the same WiFi LAN as my desktop (Intel 13900KF, ArchLinux).
We chose the Ethernet DIX option, because our LLC packets sometimes get dropped on wireless LANs (protocol ossification at work). To use the LLC Ethernet 0-Layer, all that is needed is to change eth-dix to eth-llc in (both) the configuration files in this tutorial (left to the reader).
We run the server on the raspberry pi. The O7s config file for the Raspberry Pi connects to the wlan0 device and registers the "oping-server" application with the Ethernet 0-Layer. You will have to replace wlan0 with the correct ethernet device! The name wifi1 and wifi-network don't really matter, so you can change them to your liking.
[name."oping-server"] prog=["/usr/bin/oping"] args=["--listen"] [eth-dix.wifi1] bootstrap="wifi-network" dev="wlan0" reg=["oping-server"]
The configuration for the desktop machine (O7s client) is quite simple:
[eth-dix.wifi1] bootstrap="wifi-network" dev="wlan0"
If you do not have two machines, you can run this on a single machine using the loopback adapter. In that case, set the following in your configuration file:
[name."oping-server"] prog=["/usr/bin/oping"] args=["--listen"] [eth-dix.lo] bootstrap="loopback-network" dev="lo" reg=["oping-server"]
The description assumes you are running on 2 machines, on a single machine. Save these configuration as a file on each machine - we chose to save them at /etc/ouroboros/tut04.conf - and start an IRMd on each machine with that config file.
The IRMd will log the use of the configuration file, on the raspberry pi it will create the service name for oping-server:
dstaesse@raspberrypi:~ $ sudo irmd --stdout --config /etc/ouroboros/tut04.conf ==28267== irmd(II): Ouroboros IPC Resource Manager daemon started... ==28267== irmd/configuration(II): Reading configuration from file /etc/ouroboros/tut04.conf ==28267== irmd/configuration(DB): Found service name oping-server in configuration file. ==28267== irmd(II): Created new name: oping-server. ==28267== irmd(II): Bound program /usr/bin/oping to name oping-server.
The IRMd then bootstraps the IPCP (this happens on both machines):
==28267== irmd/configuration(DB): Found IPCP wifi1 in configuration file. ==28267== irmd(II): Created IPCP 28279. ==28267== irmd/configuration(DB): No hash specified, using default. ==28279== ipcpd/ipcp(II): Bootstrapping... ==28279== ipcpd/eth-dix(DB): Device MTU is 1500. ==28279== ipcpd/eth-dix(DB): Layer MTU is 1500. ==28279== ipcpd/eth-dix(II): Using raw socket device. ==28279== ipcpd/eth-dix(DB): Bootstrapped IPCP over DIX Ethernet with pid 28279 and Ethertype 0xA000. ==28279== ipcpd/ipcp(II): Finished bootstrapping: 0. ==28267== irmd(II): Bootstrapped IPCP 28279 in layer wifi-network.
The ipcpd-eth-* detects the device MTU (which is usually 1500 bytes, but can be lower on virtual machine guests, or higher, for instance the loopback interface). The ipcpd-eth-* Layer MTU will be set to the device MTU, or to IPCP_ETH_LO_MTU when using the loopback adapter. We will probably make the Layer MTU a runtime configuration parameter in the future, so it can be configured in the configuration file. The Raspberry Pi is using Linux, which defaults to a raw socket device for interfacing with Ethernet devices. Other options for interfacing with Ethernet are Berkeley Packet Filter (BPF) (for *BSD and OS X) or netmap (if installed). Our IPCP will process all packets on a specified Ethertype. The default value is 0xA000.
The IRMd on the raspberry Pi will make the oping service available on the Ethernet 0-Layer:
==28267== irmd/configuration(DB): Registering oping-server in 28279 ==28267== irmd(WW): Name oping-server already exists. ==28279== ipcpd/ipcp(II): Registering f6c93ff2... ==28279== ipcpd/ipcp(II): Finished registering f6c93ff2 : 0. ==28267== irmd(II): Registered oping-server with IPCP 28279 as f6c93ff2.
With the IRMd started on both machines, we can start the oping server. We use the --quiet (-Q) option to reduce output in the example (oping -l -Q).
In addition to demonstrating the use of the ipcpd-eth-dix, we will also show the use of encrypted flows. To make this more visible, we dump the traffic on the Raspberry Pi's wlan0 device for ethertype 0xA000 using tcpdump (change wlan0 to the correct ethernet device).
dstaesse@raspberrypi:~ $ sudo tcpdump -i wlan0 ether proto 0xa000 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on wlan0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
Our desktop machine has MAC address c8:cb:9e:d0:04:08, the Raspberry Pi has MAC address d8:3a:dd:39:0a:b5.
Our ipcpd-eth-dix has the following packet header, note that the dst and src hardware (MAC) addresses and the ethertype are shown on the first line of the tcpdump output. The hex output starts at eid.
uint8_t dst_hwaddr[6]; uint8_t src_hwaddr[6]; uint16_t ethertype;
uint16_t eid; uint16_t length;
The endpoint identifier (EID, eid field in the header above) is used for multiplexing flows, in the ipcpd-eth-* it is 16 bits wide and set to the flow descriptor used by the application (in the ipcpd-unicast the EID is 64 bits and assigned in a random way). The length field is needed to deal with Ethernet frames that would be shorter than 64 bytes, which are padded with zero bytes, and we need to know if those zeroes are padding or payload. These are known as runt frames.
For readers also trying the ipcpd-eth-llc, the header format is a little bit different: no eid instead we use LLC SAPs as endpoint identifiers and no length (Ethernet LLC has its own length field instead of the Ethertype to deal with runt frames). The control field cf is fixed and set to LLC type 1 (0x03).
When we ping (one ping only) from the desktop (oping -n oping-server -c 1), we see the traffic exchange on the wlan. Let's break it down:
The client initiates the communications using the flow_alloc() function call. This will send a Flow Allocation Request message to the server. We have not requested anything special, so this will be a request for a vanilla best-effort raw flow:
09:57:08.998005 c8:cb:9e:d0:04:08 (oui Unknown) > d8:3a:dd:39:0a:b5 (oui Unknown), ethertype Unknown (0xa000), length 88:
0x0000: 0000 0046 0040 0000 0000 0001 0000 0000 ...F.@..........
0x0010: 0000 0000 0000 0001 ffff ffff ffff ffff ................
0x0020: 0001 d4c0 0000 0000 0000 f6c9 3ff2 3bd9 ............?.;.
0x0030: 1094 3a77 8e7b 3faf 7bff ec03 796d b676 ..:w.{?.{...ym.v
0x0040: 1bfa 5f70 758f 9ee2 e6fa .._pu.....
The packet is a flow allocation message (fixed packet format for the flow allocator). The full description of what's in this packet is:
uint16_t eid: 0000 = 0 (This message is for the flow allocator) uint16_t length: 0046 = 70 (Length of the payload) uint16_t src eid: 0040 = 64 (EID assigned to the oping client) uint16_t dst eid: 0000 = 0 (not used in allocation request) uint32_t loss: 0000 0001 = 1 (disables ARQ) uint64_t bandwidth: 0000 0000 0000 0000 = 0 (no reservation) uint32_t ber: 0000 0001 = 1 (disables CRC) uint32_t max_gap: ffff ffff = MAX_INT (any inter-packet gap is fine) uint32_t delay: ffff ffff = MAX_INT (any delay is fine) uint32_t timeout: 0001 d4c0 = 120000 (2 minutes) uint16_t cypher_s: 0000 = 0 (no encryption) uint8_t in_order: 00 = 0 (no in-order delivery) uint8_t code: 00 = 0 (flow allocation request) uint8_t availability: 00 = 0 (9's, any availability will do) int8_t response 00 = 0 (not used in flow allocation request)
The final 32 bytes of the message (f6c9 3f.. ..e2 e6fa) is the SHA3-256 hash of oping-server. When the flow is accepted at the server side (on the Raspberry Pi), the ipcpd-eth-dix responds with a flow allocation response.
The format is the same as above, without the destination has, and the QoS is set to 0 (QoS negotiation is not yet implemented).
09:57:09.000685 d8:3a:dd:39:0a:b5 (oui Unknown) > c8:cb:9e:d0:04:08 (oui Unknown), ethertype Unknown (0xa000), length 56:
0x0000: 0000 0026 0040 0040 0000 0000 0000 0000 ...&.@.@........
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0001 0000 ..........
The full explanation of the response is:
uint16_t eid: 0000 = 0 (This message is for the flow allocator) uint16_t length: 0026 = 38 (Length of the payload) uint16_t src eid: 0040 = 64 (EID assigned to the oping server at the server IPCP) uint16_t dst eid: 0040 = 64 (EID assigned to the oping client at the client IPCP) /* QoS all zero unused in response */ uint8_t in_order: 00 = 0 (part of QoS, zero) uint8_t code: 01 = 1 (flow allocation response) uint8_t availability: 00 = 0 (part of QoS, zero) int8_t response 00 = 0 (flow allocation request was successful)
The next message is the actual oping message. Note the first two fields are 0040, the first is the destination EID (64), the second is not the source EID, but the payload length (both are also 64).
09:57:10.020082 c8:cb:9e:d0:04:08 (oui Unknown) > d8:3a:dd:39:0a:b5 (oui Unknown), ethertype Unknown (0xa000), length 82:
0x0000: 0040 0040 0000 0000 0000 0000 e307 0000 .@.@............
0x0010: 0000 0000 aa02 f70d 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000 ....
The format of an oping message is:
uint32_t type: 0000 0000 = 0 (oping request) uint32_t id: 0000 0000 = 0 (sequence number, first message) uint64_t tv_sec: e307 0000 0000 0000 = - (monotonic clock timestamp) uint64_t tv_nsec: aa02 f70d 0000 0000 = - (monotonic clock timestamp)
The remainder are 40 bytes worth of zeroes to get the payload length to 64 bytes (ping default). Note that the timestamp is not converted to network byte order as it is only set and read at the client to calculate the round-trip time.
The oping response has the message type set to 0000 0001 (1 = oping response) below:
09:57:10.020890 d8:3a:dd:39:0a:b5 (oui Unknown) > c8:cb:9e:d0:04:08 (oui Unknown), ethertype Unknown (0xa000), length 82:
0x0000: 0040 0040 0000 0001 0000 0000 e307 0000 .@.@............
0x0010: 0000 0000 aa02 f70d 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0040: 0000 0000
Now, let's request an encrypted flow. To do that, start the oping client again, but now give it a raw_crypt option (oping -n oping-server -c 1 -q raw_crypt).
The first message is again a flow allocation request. The formatting is the same as above, so let's highlight the differences.
09:57:20.057379 c8:cb:9e:d0:04:08 (oui Unknown) > d8:3a:dd:39:0a:b5 (oui Unknown), ethertype Unknown (0xa000), length 179:
0x0000: 0000 00a1 0040 0000 0000 0001 0000 0000 .....@..........
0x0010: 0000 0000 0000 0001 ffff ffff ffff ffff ................
0x0020: 0001 d4c0 0100 0000 0000 f6c9 3ff2 3bd9 ............?.;.
0x0030: 1094 3a77 8e7b 3faf 7bff ec03 796d b676 ..:w.{?.{...ym.v
0x0040: 1bfa 5f70 758f 9ee2 e6fa 3059 3013 0607 .._pu.....0Y0...
0x0050: 2a86 48ce 3d02 0106 082a 8648 ce3d 0301 *.H.=....*.H.=..
0x0060: 0703 4200 0476 5de7 77ed 2802 2ad0 d02a ..B..v].w.(.*..*
0x0070: 2d63 98df 7ad2 454f dea9 043f 7a50 4a5e -c..z.EO...?zPJ^
0x0080: 1e33 1bf8 6fe9 8f06 f5b7 89b1 1b8b d1d9 .3..o...........
0x0090: 0084 c0e4 df5e 5833 19ab ab0f db54 8882 .....^X3.....T..
0x00a0: 1b7a 7cb8 ee .z|..
The QoS now specified the cypher strength as 0100 (256). The value doesn't matter at the moment, when it's non-zero it enables encryption. We will revise this option entirely. For now, it's enough for demoing the concept of encrypted flows.
With encryption enabled, not only the destination hash is sent, but also a source (ephemeral) public key (3059 30.. 7cb8 ee) is attached to the flow allocation request to perform a Diffie-Hellman exchange. The destination uses the source public key (and its own private key) to generate a shared secret and then hashes this secret (SHA3-256) to derive a unique 256-bit symmetric encryption key (AES-256).
The oping server then sends the response, with its own (ephemeral) public key (3059 30.. 45e9 ad):
09:57:20.060233 d8:3a:dd:39:0a:b5 (oui Unknown) > c8:cb:9e:d0:04:08 (oui Unknown), ethertype Unknown (0xa000), length 147:
0x0000: 0000 0081 0041 0040 0000 0000 0000 0000 .....A.@........
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0001 0000 3059 3013 0607 ..........0Y0...
0x0030: 2a86 48ce 3d02 0106 082a 8648 ce3d 0301 *.H.=....*.H.=..
0x0040: 0703 4200 045b b39c 9f17 ce44 005a e2cb ..B..[.....D.Z..
0x0050: d2c9 5207 8cad 2aa5 37fc 727b 71de 77e6 ..R...*.7.r{q.w.
0x0060: e0e3 2164 b619 a8a3 94c3 d6de ce4b dda8 ..!d.........K..
0x0070: d31e b218 5a59 24c9 685f 0943 55f5 128a ....ZY$.h_.CU...
0x0080: 0eba 46e9 ad ..F..
The client can now use the server's public key and its own private key to derive the same shared secret.
The source ping is then sent encrypted over the Ethernet network. The destination EID is 0041 (= 65). I was quick and the previous flow on the server had not timed out yet. It will be more common to have the same EID as before (0040, 64) if you follow this tutorial for the first time. Everything after the EID is encrypted. The packet length is now 114 bytes instead of 82 for the non-encrypted case (payload length is 0060 = 96 bytes). This is because we add a 128-bit (32 byte) random Initialization Vector (IV) to each packet to prevent the same data plaintext to always result in the same cypher text.
09:57:21.075534 c8:cb:9e:d0:04:08 (oui Unknown) > d8:3a:dd:39:0a:b5 (oui Unknown), ethertype Unknown (0xa000), length 114:
0x0000: 0041 0060 c180 1f26 71d1 3911 3e7c 0d29 .A.`...&q.9.>|.)
0x0010: 7287 17c8 708a cc5b bc77 9b8e af55 7594 r...p..[.w...Uu.
0x0020: 5efb 00a8 aa8d 1749 440f 6c68 6d27 cd33 ^......ID.lhm'.3
0x0030: 630a c34b a500 ea78 7d4e 1fd2 47d1 f5ab c..K...x}N..G...
0x0040: 2e17 5dc0 581a bee8 efbf dea4 fe41 2c12 ..].X........A,.
0x0050: 37f1 9704 d2da a94c 0e8a 0a00 2844 4ea2 7......L....(DN.
0x0060: b177 58b1 .wX.
09:57:21.076497 d8:3a:dd:39:0a:b5 (oui Unknown) > c8:cb:9e:d0:04:08 (oui Unknown), ethertype Unknown (0xa000), length 114:
0x0000: 0040 0060 8708 787f cf81 c7aa a006 4195 .@.`..x.......A.
0x0010: 64d8 d977 24bc bc38 cb16 138f 181e 78cd d..w$..8......x.
0x0020: 3877 c8de fed1 f0a1 902f aee6 245b a291 8w......./..$[..
0x0030: 5c7f 5e35 0f85 858b 0fb0 c55c 60e5 c38a \.^5.......\`...
0x0040: 06ac 2c11 e582 7de2 5371 32fc b58a 6c83 ..,...}.Sq2...l.
0x0050: 129d dc90 c4c8 ab5a a3e7 efd2 e8de 6ffb .......Z......o.
0x0060: 7932 7439
This was a brief tutorial demoing encrypted flows. The implementation is still rudimentary at this point, and support for other encryption methods (galois counter mode for reliable encrypted flows) and authenticated flows are on the roadmap.