Add blogpost on RINA vs Ouroboros

1 files changed, 912 insertions, 0 deletions

diff --git a/content/en/blog/20210320-ouroboros-rina.md b/content/en/blog/20210320-ouroboros-rina.md
new file mode 100644
index 0000000..04fac33
--- /dev/null
+++ b/content/en/blog/20210320-ouroboros-rina.md
@@ -0,0 +1,912 @@
+---
+date: 2021-03-20
+title: "How does Ouroboros relate to RINA, the Recursive InterNetwork Architecture?"
+linkTitle: "Is Ouroboros RINA?"
+description: "TL;DR: Ouroboros isn't RINA."
+author: Dimitri Staessens
+---
+
+```
+There are two kinds of researchers: those that have implemented
+something and those that have not. The latter will tell you that there
+are 142 ways of doing things and that there isn't consensus on which
+is best.The former will simply tell you that 141 of them don't work.
+  -- David Cheriton
+```
+
+When I talk to someone that's interested in Ouroboros, a question that
+frequently pops up is how the project relates to the
+[Recursive InterNet(work) Architecture](https://en.wikipedia.org/wiki/Recursive_Internetwork_Architecture),
+or **RINA**. I usually steer away from going into the technical
+aspects of how the architectures differ, mostly because not many
+people know the details of how RINA works. But the origin of Ouroboros
+definitely lies with our research and our experiences implementing
+RINA, so it's a good question. I'll address it as best as I can,
+without going overboard on a technical level. I will assume the reader
+is at least somewhat familiar with RINA. Also keep in mind that both
+projects are ongoing and should not be considered as "done"; things
+may change in the future. These are my -- inevitably subjective and
+undoubtedly somewhat inaccurate -- recollections of how it went down,
+why Ouroboros exists, and how it's different from RINA.
+
+And a quick note here: Ouroboros _the network prototype_ has no
+relation to the Ouroboros Proof-of-Stake protocol in the Cardano
+blockchain. That some of the Cardano guys are also interested in RINA
+doesn't help to ease any confusion.
+
+### IBBT meets RINA
+
+I first came into contact with RINA somewhere in 2012, while working
+as a senior researcher in the field of telecommunication networks at
+what was then known as IBBT (I'll save you the abbreviation), would
+soon be known as iMinds, and is now integrated into
+[IMEC](https://www.imec-int.com).  A new research project was going to
+start and our research group was looking for someone to be responsible
+for the IBBT contributions. That project, called
+[IRATI](https://cordis.europa.eu/project/id/317814) was a relatively
+short (2 years duration) project in the "Future Internet Research and
+Experimentation" (FIRE) area of the _7th framework programme_ of the
+European Commission. I won't go into the details and strategies of
+research funding; the important thing to know is that the objectives
+of FIRE are "hands-on", aimed at building and deploying Internet
+technologies. Given that I had some experience deploying experiments
+(at that time OpenFlow prototypes) on our lab testbeds, I listened to
+the project pitch, an online presentation with Q&A given by the
+project lead, Eduard Grasa from [i2cat]((https://i2cat.net/)), who
+explained the concepts behind RINA, and got quite excited about how
+elegant this all looked. So I took on the project and read John Day's
+[Patterns in Network Architecture](https://www.oreilly.com/library/view/patterns-in-network/9780132252423/),
+which we later usually referred to as _PNA_.  It was also the time
+when I was finishing my PhD thesis, so my PostDoc track was going to
+be for a substantial part on computer network architecture and RINA.
+Unifying
+[Inter-Process Communication](https://en.wikipedia.org/wiki/Inter-process_communication) (IPC)
+and networking. How exciting was that!
+
+IRATI -- Investigating RINA as an Alternative to TCP/IP -- was
+something different from the usual research projects, involving not
+only some substantially new and unfamiliar ideas, but it also relied
+very heavily on software development. Project work was performed as
+part of PhD tracks, so who would do the work? There was a PhD student
+under my guidance working mostly on OpenFlow, Sachin -- one of the
+kindest people I have ever met, and now a professor at TU Dublin --
+and there was also a master thesis student, Sander Vrijders, that was
+interested in pursuing a PhD in our research group. After a couple of
+internal sessions where we explained possible research tracks aligned
+to ongoing and upcoming projects in our group, Sander decided to take
+on the challenge of IRATI and start a PhD track on RINA.
+
+### IRATI
+
+**IRATI** kicked off in January 2013 at i2cat in Barcelona. It was
+followed by a RINA workshop, bringing the project in touch with the
+RINA community, which had its epicenter at Boston University
+(BU). It's where I first met John Day, who gave a 2-day in-depth
+tutorial of RINA. Eduard also presented an outline of the IRATI
+objectives. The project promised an implementation of RINA in Linux
+_and_ FreeBSD/JunOS, with detailed comparisons of RINA against TCP/IP
+in various scenarios, and also demonstrate interoperability with other
+RINA prototypes: the
+[TINOS prototype](https://github.com/PouzinSociety/tinos) and the
+[TRIA](http://trianetworksystems.com/) prototype. IRATI would also
+prepare the European FIRE testbeds for RINA experiments using the
+prototype. In 2 years, on 870k Euros in research funding. A common
+inside joke at project kick-off meetings in our field was to put a
+wager on the number slides that the presentation deck at the final
+project review meeting would differ from the slide decks presented at
+the initial kick-off meeting. IRATI was _not_ going to be one of those
+projects!
+
+With the RINA community gathered at the workshop, there were initial
+ideas for a follow-up research proposal to IRATI. Of course, almost
+every potential participant present was on board.
+
+Three partners were responsible for the implementation: i2cat, who had
+experience on RINA; [Nextworks](https://www.nextworks.it) a
+private-sector company with substantial experience on implementing
+networking solutions, and iMinds/imec, bringing in our testbed
+experience. Interoute (now part of [GTT](https://gtt.net)) validated
+the test scenarios that we would use for evalutions. Boston University
+had an advisory role in the project.
+
+The first work was determining the software design of the
+implementation. IRATI was going to build an in-kernel implementation
+of RINA. A lot of the heavy lifting on the design was already done
+during the project proposal preparation phase, and the components to
+be implemented were
+[well-defined](https://core.ac.uk/download/pdf/190646748.pdf).
+Broadly speaking, there were 3 things to implement: the IPCPs that
+make up the RINA layers (Distributed IPC Facilities, DIFs), the
+component that is responsible for creating and starting these IPCPs
+(the IPC manager, which had a user space and a kernel space part), and
+the core library to communicate between these components, called
+_librina_. The prototype would be built in 3 phases over the course of
+2 years.
+
+i2cat was going to get started on most of the management parts (IPC
+Manager, based on their existing Java implementation; librina,
+including the Common Distributed Application Protocol (CDAP) and the
+DIF management functions in the normal IPCP) and the Data Transfer
+Protocol (DTP). iMinds was going to be responsible for the kernel
+modules that will allow the prototype to run on top of
+Ethernet. Nextworks was taking a crucial software-architectural role
+on kernel development and software integration. For most of these
+parts we had access to a rough draft of what they were supposed to do,
+John Day's RINA reference model, which we usually referred to as _the
+specs_.
+
+i2cat had a vested interest in RINA and was putting in a lot of
+development effort with 3 people working on the project: Eduard,
+Leonardo Bergesio and Miquel Tarzan. Nextworks assigned Francesco
+Salvestrini, an experienced kernel developer to the project. From
+iMinds, the development effort would come from Sander. My personal
+involvement in the project software development was limited, as I
+still had other ongoing projects (at least until the end of 2014) and
+my main role would be in the experimentation work, which was only
+planned start after the initial development phase.
+
+The project established efficient lines of communications, mostly
+using Skype and the mailing lists and the implementation work got
+underway swiftly. I have been fortunate to be a part of a couple of
+projects where collaboration between partners was truly excellent, but
+the level of teamwork in IRATI was unprecedented. There was a genuine
+sense of excitement in everybody involved in the project.
+
+So, Sander's first task was to implement the
+[_shim DIF over Ethernet_](https://ieeexplore.ieee.org/document/6798429).
+This is a Linux loadable kernel module (LKM) that wraps the Ethernet
+802.1Q VLAN with a thin software layer to present itself using the
+RINA API. The VLAN ID would be used as the layer name. No
+functionality would be added to the existing Ethernet protocol so with
+only the src and dst address fields left, this _shim DIF_ was
+restricted to having only a single application registerd at a time,
+and to a single RINA "flow" between the endpoints. We could deploy
+about 4000 of these _shim DIFs_ in parallel to support larger RINA
+networks. The name resolution for endpoint applications was planned to
+be using the Address Resolution Protocol (ARP), which was readily
+available in the Linux kernel. Or so we thought.
+
+The ARP implementation in the kernel assumed IPv4 as the only L3
+protocol (IPv6 doesn't use ARP), so it could not handle the resolution
+of RINA _application names_ to MAC addresses, which we needed for the
+shim DIF. So after some deliberation, we decided to implement an RFC
+826 compliant version of ARP to support the shim DIF.
+
+In the meantime, we also submitted a small 3-partner project proposal
+the GEANT framework, tailored to researching RINA in an NREN (National
+Research and Education Networks) environment. The project was lead by
+us, partnering with i2cat, and teaming up with
+[TSSG](https://tssg.org/).  [IRINA](https://i2cat.net/projects/irina/)
+would kick off in October 2013, meaning we'd have 2 parallel projects
+on RINA.
+
+The project had made quite some progress in its first 6 months, there
+were initial implementations for most of the components, and in terms
+of core prototype functionality, IRATI was quickly overtaking the
+existing RINA prototypes. However, the pace of development in the
+kernel was slower than anticipated and some of the implementation
+objectives were readjusted (and FreeBSD/JunOS was dropped in favor of
+a _shim DIF for Hypervisors_). With the eye on testbed deployments,
+Sander started work on the design of a second _shim DIF_, one that
+would allow us to run the IRATI prototype over TCP/UDP.
+
+In the meantime, the follow-up project that was coined during the
+first RINA workshop took shape and was submitted. Lead by our IRINA
+partner TSSG, it was envisioned to be a a relatively large project,
+about 3.3 million Euros in EC contributions, running for 30 months and
+bringing together 13 partners with the objective to build the IRATI
+prototype into what was essentially a carrier network demonstrator for
+RINA, adding _policies_ for mobility, security and reliability.
+[**PRISTINE**](https://cordis.europa.eu/project/id/619305) got
+funded. This was an enormous boon to the RINA community, but also a
+bit of a shock for us as IRATI developers, as the software was already
+a bit behind schedule with a third project on the horizon. The
+furthest we could push forward the start of PRISTINE was January 2014.
+
+As the IRATI project was framed within
+[FIRE](https://dl.acm.org/doi/10.1145/1273445.1273460), there was a
+strong implied commitment to get experimental results with the project
+prototype. By the last quarter of 2013, the experimentation work got
+started, and the prototype was getting its first deployment trials on
+the FIRE testbeds. This move to real hardware brought more problems to
+light. The network switches in the OFELIA testbed wasn't agreeing very
+well with our RFC-compliant ARP implementation, dropping everything
+that wasn't IPv4. One of the testbeds also relied on VLANs to seperate
+experiments, which didn't fare well with our idea to (ab)use them
+within an experiment for the _shim DIF_. While Sander did the
+development of the _shim DIFs_ using the actual testbed hardware,
+other components had been developed predominantly in a virtual machine
+environment and had not been subjected to the massive parallellism
+that was available on dual-Xeon hardware. The stability of the
+implementation had to be substantially improved to get stable and
+reliable measurements. These initial trials in deploying IRATI also
+showed that configuring the prototype was very time consuming. The
+components used json configuration files which were to be created for
+each experiment deployment, causing substantial overhead.
+
+The clock was ticking and while the IRATI development team was working
+tirelessly to stabilize the stack, I worked on some (kernel) patches
+and fixes for the testbeds so we could use VLANs (on a different
+Ethertype) in our experiment. We would get deployment and stability
+testing done and (internally) release _prototype 1_ before the end of
+the year.
+
+### PRISTINE
+
+January 2014. The PRISTINE kick-off was organized together with a
+workshop, where John Day presented RINA, similar to the IRATI kick-off
+one year earlier, except this time it was in Dublin and the project
+was substantially bigger, especially in headcount. It brought together
+experts in various fields of networking with the intent of them
+applying that experience into developing polcies for RINA. But many of
+the participants to the PRISTINE project were very new to RINA, still
+getting to grips with some of the concepts (and John didn't shy away
+from making that abundantly clear).
+
+The first couple of months of PRISTINE was mostly about getting the
+participants up-to-speed with the RINA architecture and defining the
+use-case, which centered on a 5G scenario with highly mobile end-users
+and intelligent edge nodes. It was very elaborate, and the associated
+deliverables were absolute dreadnoughts.
+
+During this PRISTINE ramp-up phase, development of the IRATI prototype
+was going on at a fierce pace. The second project brought in some
+extra developers to work on the IRATI core Bernat Gaston (i2cat),
+Vincenzo Maffione (Nextworks), and Douwe de Bock (a master student at
+iMinds). i2cat focusing on management and flow control and was also
+porting the Java user-space parts to C++, Vincenzo was focusing on the
+_shim Hypervisor_, which would allow communications between processes
+running over a VM host and guest, and we were building the shim layer
+to run RINA over TCP and UDP.
+
+By this time, frustrations were starting to creep in. Despite all the
+effort in development, the prototype was not in a good shape.  The
+development effort was also highly skewed, with i2cat putting in the
+bulk of the work. The research dynamic was also changing. At the start
+of IRATI, there was a lot of ongoing architectural discussions about
+what each component should do, to improve the _specs_, but due to the
+ever increasing time pressure, the teams were working more and more in
+isolation. Getting it _done_ became a lot more important than getting
+it _right_.
+
+All this development had led to very little dissemination output,
+which didn't go unnoticed at project reviews. The upshot of the large
+time-overlap between the two projects was that, in combination with
+the IRATI design paper that got published early-on in the project, we
+could afford to lose out a bit on dissemination in IRATI and try to
+catch up in PRISTINE. But apart from the relatively low output in
+research papers, this project had no real contributions to
+standardization bodies.
+
+In any case, the project had no choice but to push on with
+development, and, despite all difficulties, somewhere mid 2014 IRATI
+had most basic functionalities in place to bring the software in a
+limited way into PRISTINE so it could start development of the
+_PRISTINE software developement kit (SDK)__ (which was developed by
+people also in IRATI).
+
+Mostly to please the reviewers, we tried to get some standardization
+going, presenting RINA at an ISO SC6 JTC1 meeting in London and also
+at IETF91. Miquel and myself would continue to follow up on
+standardization in SC6 WG7 on "Future Network" as part of PRISTINE,
+gathering feedback on the _specs_ and getting them on the track
+towards ISO RINA standards. I still have many fond memories of my
+experiences discussing RINA within WG7.
+
+The IRATI pro was officially ending soon, and the development was now
+focusing on the last functions of the Data Transfer Control Protocol
+(DTCP) component of EFCP, such as retransmission logic
+(delta-t). Other development was now shifted completely out of IRATI
+towards the PRISTINE SDK.
+
+In the meantime, we also needed some experimental
+results. Experimentation with the prototype was a painful and very
+time-consuming undertaking. We finally squeezed a publication at
+Globecom 2014 out of some test results and could combine that with a
+RINA tutorial session.
+
+January 2015, another new year, another RINA workshop. This time in
+Ghent, as part of a Flemish research project called RINAiSense --
+which should be pronounced like the French _renaissance_ -- that would
+investigate RINA in sensor networks (which now falls under the nomer
+"Internet of Things" (IoT). After the yearly _John Day presents RINA_
+sessions, this was also the time to properly introduce the IRATI
+prototype to everyone with a hands-on VM tutorial session, and to
+introduce [RINAsim](https://rinasim.omnetpp.org/), an OMNET++ RINA
+simulator developed within PRISTINE.
+
+After the workshop, it was time to wrap up IRATI. For an external
+observer it may lack impact and show little output in publications,
+and it definitely didn't deliver a convincing case for _RINA as an
+alternative for TCP/IP_. But despite that, I think the project really
+achieved a lot, in terms of bringing for the first time some tools
+that can be used to explore RINA, and for the people that worked on
+it, an incredible experience and deeps insights into computer networks
+in general. This would not have been possible without the enthousiasm
+and hard work put in by all those involved, but especially Eduard and
+the i2cat team.
+
+As IRINA was wrapping up, a paper on the how the _shim DIF over
+Hypervisors_ could be used to [reduce complexity of VM
+networking](https://ieeexplore.ieee.org/document/7452280) was
+submitted for IEEE COMMAG.
+
+We're approaching the spring of 2015, and IRATI was now officially
+over, but there was no time to rest as the clock was ticking on
+PRISTINE. The project was now already halfway its anticipated 30-month
+runtime, and its first review, somewhere end of 2014, wasn't met with
+all cheers, so we had to step up. This was also the period where some
+of my other (non-RINA) projects were running out. Up to then, my
+personal involvement on RINA had been on (software) design our
+components, reviewing the _specs_, and the practical hands-on was in
+using the software: deploying it on the testbeds and validating its
+functionality.  But now I could finally free up time to help Sander on
+the development of the IRATI prototype.
+
+Our main objective for PRISTINE was on _resilient routing_: making
+sure the _DIF_ survives underlying link failures. This has been a
+long-time research topic in our group, so we pretty much quickly know
+_how_ to do it at a conceptual level. But there were three
+requirements: first and foremost, it needed _scale_: we needed to be
+able to run something that could be called a network, not just 3 or 4
+nodes and not just a _couple_ of flows in the network. Second, it
+needed _stability_: to measure the recovery time, we needed to send
+packets at small but -- more importantly -- steady intervals and
+thirdly, we needed measurement _tools_.
+
+As part of IRINA, we developed a basic traffic-generator, which would
+be extended for PRISTINE and tailored to suit our needs. Stability was
+improving gradually over time. Our real problem was _scale_, to which
+the biggest hurdle was the configuration of the IRATI stack. It was a
+complete nightmare. Almost anything and everything had to be
+preconfigured in _json_. I remember that by that time, Vincenzo had
+developed a tool called the _demonstator_ based on tiny buildroot VMs
+to create setups for local testing, but this wasn't going to help us
+deploy it on the Fed4FIRE testbeds. So Sander developed one of the
+first orchestrators for RINA, called the _configurator_ for deploying
+IRATI on [emulab](https://www.emulab.net/portal/frontpage.php).
+
+Somewhere around that time, the _one-flow-only-limitation_ of the
+_shim DIF over VLAN_ was showing and a _shim DIF over Ethernet Link
+Layer Control (LLC)_ was drafted and developed. By mapping endpoints
+to LLC Service Access Points (SAPs), this _shim DIF_ could support
+parallel flows (data flows and management flows) between the client
+IPCPs in the layer above.
+
+With the PRISTINE SDK released as part of "openIRATI" somewhere after
+the January workshop a good month prior, there was another influx of
+code into the prototype for all the new features
+(a.k.a. _policies_). Francesco, who had been managing a lot of the
+software integration, was also leaving the RINA projects. This is the
+point where I really noticed that Sander and Vincenzo were quickly
+losing faith in the future of the IRATI codebase, and the first ideas
+of branching off -- or even starting over -- began to emerge.
+
+The next Horizon-2020-proposal deadline was also approaching, so our
+struggles at that point also inspired us to propose developing a more
+elaborate RINA orchestrator and make deployment and experimentation
+with (open)IRATI a much more enjoyable experience. That project,
+[ARCFIRE](https://ict-arcfire.eu/) would start in 2016.
+
+Now, we were still focusing on the basics: getting link state routing
+running, adding some simple _loop-free alternates_ policy to it, based
+on the operation of [IP FRR](https://tools.ietf.org/html/rfc5286) and
+running a bunch of flows over that network to measure packet loss when
+we break a link. Sander was focusing on the policy design and
+implementation, I was going to have a look at the IRATI code for
+scaling up the flow counts, which needed non-blocking I/O. I won't go
+into the details, but after that short hands-on stint in the IRATI
+codebase, I was onboard with Sander to starting looking to options for
+a RINA implementation beyond IRATI.
+
+It was now summer 2015, PRISTINE would end in 12 months and the
+project was committed to openIRATI, so at least for PRISTINE, we again
+had no choice but to plow on. A couple of frustrating months lied
+ahead of us, trying to get experimental results out of a prototype
+that was nowhere near ready for it, and with a code base that was also
+becoming so big and complex that it was impossible to fix for anyone
+but the original developers. This is unfortunately the seemingly
+inescapable fate of any software project whose development cycle is
+heavily stressed by external deadlines, especially deadlines set
+within the rigid timeline of a publicly funded research project.
+
+By the end of summer, we were still a long way off the mark in terms
+of what we hoped to achieve. The traffic generator tool and
+configurator were ready, and the implementation of LFA was as good as
+done, so we could deploy the machines for the use case scenarios,
+which were about 20 nodes in size, on the testbeds. But the deployment
+that actually worked was still limited to a 3-node PoC in a triangle
+that showed the traffic getting routed over the two remaining link if
+a link got severed.
+
+In the meantime, Vincenzo had started work on his own RINA
+implementation, [rlite](https://github.com/vmaffione/rlite), and
+Sander and myself started discussing options on a more and more
+regular basis on what to do. Should we branch off IRATI and try to
+clean it up? Keep only IRATI kernel space and rewrite user space? Hop
+on the _rlite_ train? Or just start over entirely? Should we go
+user-space entirely or keep parts in-kernel?
+
+In the last semester of 2015, Sander was heading for a 3-month
+research stint in Boston to work on routing in RINA with John and the
+BU team. By that time, we had ruled out branching off of openIRATI.
+Our estimate was that cleaning up the code base would be more work
+than starting over. We'd have IRATI as an upstream dependency, and
+trying to merge contributions upstream would lead to endless
+discussions and further hamper progress for both projects. IRATI was
+out.  Continuing on rlite was still a feasible option. Vincenzo was
+making progress fast, and we knew he was extremely talented. But we
+were also afraid of running into disagreements of how to proceed. In
+the meantime, Sander's original research plans in Boston got subverted
+by a 'major review' decision on the _shim Hypervisor_ article, putting
+priority on getting that accepted and published. When I visited Sander
+in Boston at the end of October, we were again assessing the
+situation, and agreed that the best decision was to start our own
+prototype, to avoid having _too many cooks in the kitchen_.
+Development was not part of some funded project, so we were free to
+evaluate and scrutinize all design decisions, and we could get
+feedback on the RINA mailing lists on our findings. When all
+considerations settled, our own RINA implementation was going to be
+targeting POSIX and be user space only.
+
+We were confident we could get it done, so we took the gamble. ARCFIRE
+was going to start soon, but the first part of the project would be
+tool development. Our experimentation contributions to PRISTINE were
+planned to wrap up by April -- the project was planned to end in June,
+but a 4-month extension pushed it to the end of October. But starting
+May, we'd have some time to work on Ouroboros relatively
+undisturbed. In the very worst case, if our project went down the
+drain, we could still use IRATI or rlite to meet any objectives for
+ARCFIRE. We named our new RINA-implementation-to-be _Ouroboros_, the
+mythical snake that eats its own tail represented recursion, and also
+-- with a touch of imagination -- resembles the operation of a _ring
+buffer_.
+
+### ARCFIRE
+
+Another year, another RINA project kick-off, this time it was again in
+Barcelona, but this time without a co-located workshop. ARCFIRE (like
+IRATI before it) was within the FIRE framework, and the objective was
+to get some experiments running with a reasonable number of nodes (on
+the order of 100) to demonstrate stability and scale of the prototypes
+and also to bring tooling to the RINA community. The project was
+coordinated by Sven van der Meer (Ericsson), who had done significant
+work on the PRISTINE use cases, and would focus on the impact of RINA
+on network management. The industry-inspired use cases were brought by
+Diego Lopez (Telefónica), _acteur incontournable_ in the
+Network Functions Virtualization (NFV) world. The project was of
+course topped off with i2cat, Nextworks, and ourselves, as we were
+somewhere in the process of integration into IMEC. The order at hand
+for us was to develop an fleshed-out testbed deployment framework for
+RINA, which we named [Rumba](https://gitlab.com/arcfire/rumba). (A
+rhumba is a bunch of rattlesnakes, and Ouroboros is a snake, and it
+was written in Python -- rhumba already existed, and rumba was an
+accepted alternate spelling).
+
+In early 2016, the RINA landscape was very different from when we
+embarked on IRATI in 2013. There were 2 open source prototypes, IRATI
+was the de-facto standard used in EC projects, but Vincenzo's rlite
+was also becoming available at the time and would be used in
+ARCFIRE. And soon, the development of a third prototype -- _ouroboros_
+-- would start. External perception of RINA in the scientific
+community had also been shifting, and not in a positive direction. At
+the start of the project, we had the position paper with project plans
+and outlines, and the papers on the _shims_ showed some ways on how
+RINA could be deployed. But other articles trying to demonstrate the
+benefits of RINA were -- despite all the efforts and good will of all
+people involved -- lacking in quality, mostly due to the limitations
+of the software. All these subpar publications did more harm than
+good, as the quality of the publications rubbed off on the perceived
+merits of the RINA architecture as a whole. We were always feeling
+this pressure to publish _something_, _anything_ -- and reviewers were
+always looking for a value proposition -- _Why is this better than my
+preferred solution?_, _Compare this in depth to my preferred solution_
+-- that we simply couldn't support with data at this point in
+time. And not for lack of want or a lack of trying. But at least,
+ARCFIRE had at 2 years to look forward to, a focused scope and by now,
+the team had a lot of experience in the bag. But for the future of
+RINA, we knew the pressure was on -- this was a _now or never_ type of
+situation.
+
+### Ouroboros
+
+We laid the first stone on Ouroboros on Friday February 12th, 2016. At
+that point in time Ouroboros was still planned as a RINA
+implementation, so we started from the beginning: an empty git
+repository, renewed enthousiasm, fresh _specs_ -- still warm from the
+printer and smelling of toner -- in our hands, and Sanders initial
+software design and APIs in colored marker on the whiteboard. Days
+were long -- we still had work to do on PRISTINE, mind you -- and
+evenings were short. I could now imagine the frustration of the i2cat
+people, who a couple of years prior were probably also spending their
+evenings and nights enthousiastically coding on IRATI while, for us,
+IRATI was still a (very interesting) job rather than a passion. We
+would feel no such frustrations as we knew from the onset that the
+development of Ouroboros was going to be a two-man job.
+
+While we were spending half our days gathering and compiling results
+from our _LFA_ experiments for PRISTINE, which -- fortunately or
+unfortunately depending on the way I look at it -- did not result in a
+publication, and half our days on the rumba framework, our early
+mornings and early evenings were filled with discussions on the RINA
+API used in Ouroboros. It was initially based on IRATI. Flow
+allocation used source and destination _naming information_ -- 4
+objects that the RINA _specs_ (correctly, might I add) say should be
+named: Application Process Name, Application Process Instance Id,
+Application Entity Name and Application Entity Instance Id. This
+_naming information_ as in IRATI, was built into a single structure --
+a 4-tuple -- and we were quickly running into a mess, because, while
+these names need to be identified, they are not resolved at the same
+time, nor in the same place. Putting them in a single struct and
+passing that around with NULL values all the time was really ugly. The
+naming API in Ouroboros changed quickly over time, initially saving
+some state in an _init_ call (the naming information of the current
+application, for instance) and later on removing the source naming
+information from the flow allocation protocol alltogether, because it
+could so easily be filled with fake garbage that one shouldn't rely on
+it for anything. The four-tuple was then broken up to pass two 2-tuple
+name and instance-id, using one for the Process, the other for the
+Entity. But we considered these changes to be just a footnote in the
+RINA service definition, -- taste, one could take it or leave it, no
+big deal. Little did we know that these small changes were just the
+start -- the first notes of a gentle, breezy prelude that was slowly
+building towards a fierce, stormy cadenza that would signify the
+severance of Ouroboros from RINA almost exactly one year later.
+
+Another such change was with the _register_ function. To be able to
+reach a RINA application, you need to register it in the _DIF_. When
+we were implementing this, it just struck us that this code was being
+repeated over and over again in applications. And just think about it,
+_how does an application know which DIFs there are in the system?_.
+And if new DIFs are created while the application is running, how do I
+feed that information? That's all functionality that would have to be
+included in _every_ RINA application. IRATI has this as whole set of
+library calls. But we did something rather different. We moved the
+registering of applications _outside_ of the applications
+themselves. It's _application management_, not _IPC_. Think about how
+much simpler this small change makes life for an application
+developer, and a network administrator. Think about how it would be if
+-- in the IP world -- you could create a socket on port 80 or port 443
+_from the shell_, and set options on that socket _from the shell_, and
+then tell your kernel that incoming connections on that socket should
+be sent to this Apache or that Nginx program _from the shell_, and all
+that the Apache or Nginx developers would need to do is call accept()
+and read/write/select/epoll etc calls, instead of having to handle
+sockets and all their options. That's what the bind() and register()
+calls in Ouroboros do for Ouroboros applications: you bind some
+program to that name _from the command line_, you register the name in
+the layer (equivalent of creating the socket) _from the command line_
+, and all the (server) program has to do is call _flow_accept()_ and
+it will receive incoming flows.  It is this change in the RINA API
+that inspired us to name our first very first public presentation
+about Ourobors, at FOSDEM 2018,
+[IPC in 1-2-3](https://archive.fosdem.org/2018/schedule/event/ipc/).
+
+When we tried to propose them to the RINA community, these changes
+were not exactly met with cheers. The interactions with that community
+was alse beginning to change. RINA was the _specs_. Why are we now
+again asking questions about basic things that we implemented in IRATI
+years ago? IRATI shows its works. Want to change the _specs_: talk to
+John.
+
+We had also implemented our first _shim DIF_, which would allow to run
+the Ouroboros prototype over UDP/IPv4. We started with a UDP shim
+because there is a POSIX sockets API for UDP. Recall that we were
+targeting POSIX, including FreeBSD and MacOS X to make the Ouroboros
+prototype more accessible. But programming interfaces into Ethernet,
+such as _raw sockets_, were not standard between operating systems, so
+we would implement an Ethernet _shim DIF_ later. Now, the Ouroboros
+_shim DIF_ stopped being a _shim_ pretty fast. When we were developing
+the _shim DIFs_ for IRATI, there was one very important rule: we were
+not allowed to add functionality to the protocol we were wrapping with
+the RINA API, we could only _map_ functions that were existing in the
+(Etherent/UDP) protocol. This -- was the underlying reasoning -- would
+show that the protocol/layers in the current internet were
+_incomplete_ layers. But that also meant that the functions that were
+not present -- the flow allocator in particular -- would need to be
+circumvented through manual configuration at the endpoints. We weren't
+going to have any of that -- the Ouroboros IPCP daemons all implement
+a flow allocator. You may also be wondering why none of the prototypes
+have a _shim DIF_ directly over IP. It's perfectly possible! But the
+reason is simple: it would use a non-standardized value for the
+_protocol_ field in the IP header, and most IP routers simply drop
+such packets.
+
+Somewhere around April, we were starting the implementation of a
+_normal_ IPCP in Ouroboros, and another RINA component was quickly
+becoming a nuisance to me: the _Common Distributed Application
+Protocol_ or _CDAP_. While I had no problem with the objectives of
+CDAP, I was -- to put it mildly -- not a big fan of the
+object-oriented paradigm that was underneath it. Its methods,
+_read/write, create/destroy, start/stop_ make sense to many, but just
+like the HTTP methods PUT/GET/DELETE/POST/... there is nothing
+_fundamental_ about it. It might as well have just one method,
+_[execute](http://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html)_.
+It's taste, and it definitely wasn't _my_ taste. I found that it only
+proved my long-holding observation that for every engineer there were
+at least three overengineers. I made a bold prediction to Sander: one
+day, we would kick CDAP out of the Ouroboros prototype.
+
+Summer was approaching again. Most of the contributions to PRISTINE
+were in, so the ARCFIRE partners could start to focus on that
+project. And there was a risk: ARCFIRE depended on the Fed4FIRE
+testbeds, which was ending and its future was not certain. The
+projected target API for _rumba_ was
+[jFed](https://jfed.ilabt.imec.be/). To mitigate the risk, we made an
+inventory of other potential testbeds, and to accomodate for the wait
+for the results of the funding calls, we proposed (and got) an
+extention to ARCFIRE with 6 months to a 30-month project duration. In
+the end, Fed4FIRE was funded, ARCFIRE had some breathing space --
+after all, we had to fire on all cylinders to get the best possible
+results and make a case for RINA -- and Sander and myself had some
+extra time to get Ouroboros up and running.
+
+Sander quickly developed an Ethernet LLC _shim DIF_ based on the UDP
+one, and after that, we both moved our focus on the key components in
+the _normal IPCP_, implementing the full flow allocator and building
+the data transfer protocol (DTP), and the routing and forwarding
+functionality. CDAP was getting more and more annoying, but apart from
+that, this part of the RINA _specs_ were fairly mature following the
+implementation work in IRATI, and the implementation progress was
+steady and rather uneventful. For now.
+
+Work on the PRISTINE project was wrapped up, and the final
+deliverables were submitted at the end of October. PRISTINE was a
+tough project for us, with very little outcomes. Together with Miquel,
+I did make some progress with RINA standardization in ISO
+JTC1/SC6. But Sander and myself could show few research results, no
+published papers where we were the main authors. PRISTINE as a whole
+also fell short a bit in its main objectives, the RINA community
+hadn't substantially grown, and its research results were still --
+from an external vantage point -- mediocre. For us, it was a story of
+trying to do too much, too soon. Everyone tried their best, and I
+think we achieved what was achieveable given the time and resources we
+had. The project definitely had some nice outcomes. Standardization at
+least got somewhere, with a project in ISO and also some traction
+within the Next Generation Protocols (NGP group at
+[ETSI](www.etsi.org). RINAsim was a nice educational tool, especially
+for visualizing the operation of RINA.
+
+Our lack of publication output was also noticed by our direct
+superiors at the University, who got more and more anxious. The
+relationship deteriorated steadily, we were constantly nagged about
+publications, _minimum viable papers_, and the _value proposition_ of
+RINA: _killer features_, _killer apps_. For us, the simplicity and
+elegance of the design was all we needed as a motivation to
+continue. There were some suggestions to build a simulator instead of
+a full prototype. My feeling was that a simulator would be
+unconvincing to show any _benefits of RINA_ -- I can't express in
+words how much I hated that phrase. To prove anything, simulators need
+to be validated against the real thing. And there are certain pitfalls
+that can only be found in an implementation. This is the reason why I
+chose that particular quote at the top of this blog post. Both parties
+started to sound like broken records to eachother, every meeting was
+devolving into a pointless competition in
+who-knows-the-most-workarounds. As the saying goes, arguing with an
+engineer is like wrestling a pig in the mud. There wasn't anything
+constructive or useful to those interactions, so we stopped giving a
+shit -- pardon my French. The Ouroboros prototype was coming along, we
+were confident that we knew what we were doing. All we needed was time
+to get it done. We'll write a paper on Ouroboros when we had one worth
+writing.
+
+By January 2017, we had a minimal working _normal_ IPCP. Sander was
+looking into routing, working on a component we called the _graph
+adjacency manager_ (GAM). As its name suggest, the GAM would be
+responsible for managing links in the network, what would be referred
+to as the _network topology_, and would get policies that instruct it
+how to maintain the graph based on certain parameters. This component,
+however, was short-lived and replaced by an API to connect IPCPs so
+the actual layer management logic could be a standalone program
+outside of the IPCPs instead of a module inside the IPCPs, which is
+far more flexible.
+
+### Ouroboros diverges from RINA
+
+In the meantime, I was implementing and revising _CACEP_, the Common
+Application Connnection Establishment Protocol that was accompanying
+CDAP in RINA. Discussions on CACEP between Sander and myself were
+interesting and sometimes heated -- whiteboard markers have
+experienced flight and sudden deceleration. CDAP was supposed to
+support different encoding schemes -- the OSI _presentation layer_. We
+were only going to implement Google Protocol Buffers, which was also
+used in IRATI, but the support for others should be there. The flow
+allocator and the RIB were built on top of our CDAP
+implementation. And something was becoming more and more obvious. What
+we were implementing -- agreeing on protocol versions, encoding etc --
+was something rather universal to all protocols.  Now, you may
+remember that the flow allocator is passing something -- the
+information needed to connect to a specific Application Entity or
+Application Entity Instace -- that was actually only needed after the
+flow allocation procedure was basically established.  But after a
+while, it was clear to me that this information should be _there_ in
+that CACEP part, and was rather universal for all application
+connections, not just CDAP. After I presented this to Sander
+_<despair>_ over IRC, he actually recognized how this -- to me
+seemingly small -- change impacted the prototype. Now, I will never
+forget the exchange, and I actually saved that conversation as a text
+file. The date was February 24th, 2017.
+
+```
+...
+<despair> nice, so then dev.h is even simpler
+<despair> ae name is indeed not on the layer boundary
+<dstaesse> wait why is dev.h simpler?
+<despair> since ae name will be removed there
+<dstaesse> no
+<dstaesse> would you?
+<despair> yes
+<despair> nobody likes balls on the line
+<despair> it's balls out
+...
+```
+
+Now, RINA experts will (or should) gasp for air when reading this. It
+refers to something that traces back to John's ISO JTC1/SC6 days
+working on Open Systems Interconnect (OSI), when there was a heavy
+discussion ongoing about the "Application Entity": _where was it
+located_? If it was in the _application_, it would be outside of SC6,
+which was dealing with networks, if it was in the network, it would be
+dealt with _only_ in SC6. It was a turf battle battle between two ISO
+groups, and because Application Entities were usually drawn as a set
+of circles, and the boundary between the network application as a
+line, that battle was internally nicknamed -- boys will be boys -- the
+_balls-in, balls-out_ question. If you ever attended one of John's
+presentation, he will take a short pause and then continue that this
+was the only time that a major insight came from a turf war: _the
+balls were on the line_. The Application Entity needed to be known in
+both the application and the network. Alas! Our implementation was
+clearly showing that this was not the case. The balls were _above_ the
+line, the _network_ (or more precise: the flow allocator) doesn't need
+to know _anything_ about application entities! Then and there, Sander
+had hammered a first nail in RINA's coffin.
+
+Ouroboros now had a crisp and clear boundary between the flow in a
+_DIF_, and any connections using that flow in the layer above. Flow
+allocation creates a flow between _Application Instances_ and after
+that, a connection phase would create a _connection_ between
+_Application Entity Instances_. So roughtly speaking -- without the
+OSI terminology -- first the network connects the running programs,
+and after that, the programs decide which protocol to use (which can
+be implicit). What was in the _specs_ , what the RINA API was actually
+doing, was piggybacking these exchanges! Now, we have no issues with
+that from an operational perspective, in effect the Ouroboros flow
+allocator has a _piggyback API_. But the contents of the piggybacked
+information in Ouroboros is _opaque_. And all this has another, even
+bigger, implication. One that I would come to via another line of
+reasoning some time later.
+
+With ARCFIRE rolling along and the implementation of the _rumba_
+framework in full swing, Sander was working on the link-state routing
+policy for Ouroboros, and I started implementing a _Distributed Hash
+Table (DHT)_ that would serve as the directory -- think of the
+equivalent of [DNS-SRV](https://en.wikipedia.org/wiki/SRV_record) for
+a RINA DIF -- a key-value store mapping _application names_ to
+_addresses_ in the layer. The link-state routing component was
+something that was really closely related to the Resource Information
+Base -- the RIB. That RIB was closely coupled with CDAP. Remember that
+prediction that I made about a year prior, somewhere in April 2016? On
+September 9th 2017, two weeks before the ARCFIRE RINA hackathon, CDAP
+was removed from Ouroboros. I still consider it the most satisfying
+[git commit](https://ouroboros.rocks/cgit/ouroboros/commit/?id=45c6615484ffe347654c34decb72ff1ef9bde0f3&h=master)
+of my life, removing 3700 lines of utter uselessness -- CDAP got 3 out
+of 4 characters right. From that day, Ouroboros could definitely not
+be considered a RINA implementation anymore.
+
+It was time to get started on the last big component: DTCP -- the
+_Data Transfer Control Protocol_. When implementing this, a couple of
+things were again quickly becoming clear. First, the implementation
+was proving to be completely independent of DTP. The RINA _specs_, you
+may recall, propose a state vector between DTP and DTCP. This solves
+the _fragmentation problem_ in TCP: If an IP fragment gets lost, TCP
+would resend all fragments. Hence TCP needs to know about the
+fragmentation in IP and only retransmit the bytes in that fragment.
+But the code was again speaking otherwise. It was basically telling
+us: TCP was independent of IP. But fragmentation should be in TCP, and
+IP should specify its maximum packet size. Anything else would result
+in an untolerable mess.  So that's how we split the _Flow and
+Retransmission Control Protocol_ (FRCP) and the _Data Transfer
+Protocol_ (DTP) in Ouroboros. The second nail in RINA's coffin.
+
+With FRCP split from DTP in roughly the same way as TCP was split from
+IP, we had a new question: where to put FRCP? RINA has DTCP/DTP in the
+layer as EFCP. And this resulted in something that I found rather
+ugly: a normal layer would "bootstrap" its traffic (e.g. flow
+allocator) over its own EFCP implementation to deal with underlying
+layers that do not have EFCP (such as the _shim DIFs_). Well, fair
+enough I guess. But there is another thing. One that bugged me even
+more. RINA has an assumption on the _system_, one that has to be
+true. The EFCP implementation -- which is the guarantee that packets
+are delivered, and that they are delivered in-order -- is in the
+IPCP. But the application process that makes use of the IPCP is a
+_different process_. So, in effect, the transfer of data, the IPC,
+between the Application Process and the IPCP has to be reliable and
+preserver data order _by itself_. RINA has no control over this
+part. RINA is not controlling _ALL_ IPC; there is IPC _outside of
+RINA_. Another way of seeing it, is like this: If a set of processes
+(IPCPs) are needed to provide reliable state synchronization between
+two applictions A and B, who is providing reliable state
+synchronization between A and the first IPCP? If it's again an IPCP,
+that's _infinite_ recursion! Now -- granted -- this is a rather
+_academic_ issue, because most (all?) computer hardware does provide
+this kind of preserving IPC. However, to me, even theoretical issues
+were issues. I wanted Ouroboros to do _ALL_ IPC, even between its own
+components, and not make _any_ assumptions! Then, and only then, it
+would be universal. Only then, the _unification of networking and IPC_
+would be complete.
+
+The third nail in RINA's coffin was _the big one_. And in hindsight,
+we should already have seen that coming with our realization that the
+application entity was _above the line_. We moved FRCP into the
+application. It would be implemented in the library, not in the IPCP,
+as a set of function calls, just like HTTP libraries. Sander was
+initially skeptic, because to his taste, if a single-threaded
+application uses the library, it should remain single-threaded. How
+could it send acknowledgements, restransmit packets etc? I agreed, but
+said I was confident that it would work by running the functionality
+as part of the IPC calls, read/write/fevent. And that's how it's
+implemented now.  All this meant that Ouroboros layers were not DIFs,
+and we stopped using that terminology.
+
+By now, the prototype was running stable enough for us to go _open
+source_. We got approval from IMEC to release it to the public under
+the GPLv2 / LGPL license, and in early 2018, almost exactly 2 years
+after we started the project, we presented the first public version of
+Ouroboros at FOSDEM 2018 in Brussels.
+
+But we were still running against the clock. ARCFIRE was soon to end,
+and Ouroboros had undergone quite some unanticipated changes that
+meant the implementation was facing the reality of [Hofstadter's
+Law](https://en.wikipedia.org/wiki/Hofstadter%27s_law).
+
+We were again under pressure to get some publications out; in order to
+meet ARCFIRE objectives, and Sander had to meet some publication quota
+to finish his PhD. The design of Rumba was interesting enough for a
+[paper](https://www.geni.net/), the implementation allowed us to
+deploy 3 Recursive Network prototypes (IRATI, rlite and Ouroboros) on
+testeds using different APIs: jFed for Fed4Fire and
+[GENI](https://www.geni.net/), Emulab for iMinds virtual wall testbed,
+QEMU using virtual machines, docker using -- well -- docker
+containers, and a local option only for Ouroboros. But we needed more
+publications, so for ARCFIRE Sander had implemented Loop-Free
+Alternates routing in Ouroboros and was getting some larger-scale
+results with them. And I reluctantly started working on a paper on
+Ouroboros -- I still felt the time wasn't right, and we first needed
+to have a full FRCP implementation and full congestion avoidance to
+make a worthwile analysis. By then I long had a feeling that my days
+at the university were numbered, it was time to move on, and I was
+either leaving after submitting a publication on Ouroboros, or without
+a publication on Ouroboros.
+
+We finished the experiments for ARCFIRE, but as with PRISTINE, the
+results were not accepted for publication. During the writing of the
+paper, a final realization came. We had implemented our link-state
+routing a while ago, and it was doing something interesting, akin to
+all link-state routing protocols: a link-state packet that came in on
+some flow, was sent out on all other flows. It was -- in effect
+--doing broadcast. But... OSPF is doing the same. Wait a minute. OSPF
+uses a multicast IP address. But of course! Multicast wasn't what it
+seemed to be. Multicast was broadcast on a layer, creating a multicast
+group was enrollment in that layer. A multicast IP address is a
+broadcast layer name! Let that one sink in. Based on the link-state
+routing code in the _normal IPCP_, I implemented the broadcast IPCP in
+a single night. The _normal IPCP_ was renamed _unicast IPCP_. It had
+all fallen into place, the Ouroboros architecture was shaped.
+
+But we had no value proposition to give, no value-added feature, no
+killer app, no unique selling point. Elegance? I received my notice on
+Christmas Eve 2018. Life as a researcher would be over. But what a
+ride those last 3 years had been. I'd do the same all over again.
+
+The [paper](https://arxiv.org/abs/2001.09707) was submitted in January
+2019. We haven't received any word from it since.
+
+With the GPL license on Ouroboros, Sander and myself decided to
+continue to update the prototype and build a bit of a website for
+it. So, if you made it all the end to this blog post: thank you for
+your interest in the project, that's why we did what we did, and
+continue to do what we do.
+
+Stay curious,
+
+Dimitri
\ No newline at end of file