Project Iris

Peer-to-peer messaging for backend decentralization


Completely decentralized.
Secure against passive and active attacks.
Beautiful, simple and language agnostic API.


Want to learn more? The book of Iris!

 
February 3, 2014: Iris at FOSDEM '14

Iris v0.2.0 released!

Published: March 31, 2014

After a long wait, I am thrilled to announce the release of version 0.2.0 of the Iris cloud messaging system. It is a massive stability release, with over 80% of the code base cleaned up and rewritten, slowly but surely transitioning from a research project into a stable product. There is still plenty to do, but also already plenty to see. Beside the countless fixes, notable additions under the hood are:

  • Completely redesigned tunnels based on direct connections, both simplifying the code and improving performance.
  • System and data traffic uses separate network links, ensuring that clogged up data links don't delay control messages.
  • Graceful termination was added up until pastry-peer level, leading to fewer lost messages during churn; still enough to do.

As usual, you can find the pre-built Iris binaries at the projects downloads section for Linux, Mac OS X and Windows; as well as the full source code at the projects GitHub repository. Additionally, I would also like to shout out to the first two outside contributors to the project, Andrew and Sander, thanks & cheers! :)

Chinese whispers

Although I am slowly preparing some learning material in The book of Iris, as well as there being online executable demos for both the Go and Erlang bindings in the Iris playground, I thought it appropriate to write up a small tutorial to go along with the release. It is not so formal as the book, neither so detailed as the demos, but it does make for a nice show-off. Hope you like it! ;)

If you are not familiar with it, Chinese whispers is a game where the first player whispers a sentence to the second, who passes it along to the third player, to the fourth, continuing until the messages reaches back to the originating player, usually convoluting into something incomprehensible. We could call it a communication ring, but hey, Chinese whispers sounds much cooler.


Image originates from Rob Pike's Go Concurrency Patterns talk

The game itself thus is pretty straightforward, we just need to convince a few gophers to pass messages round and round along a ring. The question is, how fancy will the program end up if the gophers are individual machines or VMs of a cloud? With classical networking or popular messaging systems, finding the next gopher in the line is a non-trivial task!

So how does Iris make our life simpler? By taking care of distributed network formation and service discovery, whilst at the same time providing semantic addressing and complete decentralization. Simply put, forget the networking, name the gophers and Iris will sort out the rest. But let's see code, not just talk!

Laying the groundwork

First up, we'll need to download the Iris node, which will do all the magical things for us. It was an explicit design decision to have a separate process opposed to pushing Iris into a client library. The main reason is language agnosticism, as it trivializes the addition of new languages.

Iris requires two things to assemble into a self-organized network: an arbitrary name for the network – allowing multiple Iris networks to coexist – and an RSA private key to authenticate peer nodes and ensure all communication is encrypted. An optional client port can also be specified if multiple nodes are run on the same machine.

> openssl genrsa -out key.rsa 2048
> iris -net "chinese whisperers" -rsa ./key.rsa
2014/03/31 08:00:00 main: booting iris overlay...
2014/03/31 08:00:00 scribe: booting with id 601780164183.
2014/03/31 08:00:10 main: iris overlay converged with 0 remote connections.
2014/03/31 08:00:10 main: booting relay service...
2014/03/31 08:00:10 main: iris successfully booted, listening on port 55555.

Since above we started only a single node, after a period of unsuccessful discovery (10 seconds currently), Iris will report converged. If on the other hand we start multiple concurrent nodes, they will successfully find each other.

> iris -net "chinese whisperers" -rsa ./key.rsa -port 3333
...
08:03:02 main: iris overlay converged with 1 remote connections
08:03:02 main: iris successfully booted, listening on port 3333
> iris -net "chinese whisperers" -rsa ./key.rsa -port 4444
...
80:03:02 main: iris overlay converged with 1 remote connections
80:03:02 main: iris successfully booted, listening on port 4444

Of course, you can run arbitrarily many of these on top of cloud VMs, just make sure the IP address space is not too large or convergence time will suffer. The takeaway is, you just assembled a whole network of communicating processes without even thinking about networking. Great job! :D Time to call the whisperers.

Bring in the gophers

As already mentioned, Iris uses semantic addressing. All participants are assigned textual names rather than low level network addresses. Although normally multiple clients would register with the same name (forming load balanced clusters), we'll assign a separate name for each gopher to be able to send messages to individuals. We'll assign gopher #0 to the first player – who will begin and end the whispering – and gopher #1, gopher #2... to subsequent players who will just pass the message on.

Each gopher will need to connect (Go|Erlang) to a locally running Iris node (by default listening on port 55555) and register with its name and a callback handler which will receive the inbound messages. The connection code is the same for all gophers (change the port according to your setup).

name := fmt.Sprintf("gopher #%d", index)
conn, err := iris.Connect(55555, name, handler)
if err != nil {
    log.Fatalf("failed to connect to Iris: %v.", err)
}

The handlers on the other hand are a bit different for the initiator gopher and the subsequent forwarder gophers. The first player will only display the looped messages, while other players will pass everything received to the next player in line. For our purposes, we'll use Iris broadcasts (Go|Erlang), which will send a message to all clients registered under a particular name (currently only one).

type initiator struct {}

func (i *initiator) HandleBroadcast(msg []byte) {        
    fmt.Println("Message received:", string(msg))
}
type forwarder struct {
    conn iris.Connection // Connection to send messages
    next string          // Name of the next hop
}

func (f *forwarder) HandleBroadcast(msg []byte) {
    if err := f.conn.Broadcast(f.next, msg); err != nil {
        log.Printf("failed to forward message: %v.", err)
    }
}

The forwarders are pretty much done, they will just sit idly and wait for messages to pass on. The initiator will need a bit more logic to start pushing out the messages.

for i := 0; i < 100; i++ {
    msg := fmt.Sprintf("whisper #%d", i)
    if err := conn.Broadcast(next, []byte(msg)); err != nil {
        log.Printf("failed to initiate message: %v.", err)
    }
    time.Sleep(time.Second)
}

In essence, the above code snippets are enough to create an arbitrarily large, completely self-organized and decentralized process ring passing messages around. Actual code requires a bit more boilerplate to glue everything together. You can find the full listing for both the initiator as well as the forwarder. And of course, running a ring of size 2 (one initiator and one forwarder) yields the following output.

> iris -net "chinese whisperers" -rsa ./key.rsa -port 3333 &
> ./initiator -relay 3333 -nodes 2
(message lost, no forwarder)
(message lost, no forwarder)
(message lost, no forwarder)
Message received: whisper #3
Message received: whisper #4
Message received: whisper #5
(message lost, no forwarder)
^C (terminate initiator)
> iris -net "chinese whisperers" -rsa ./key.rsa -port 4444 &
(do nothing)
(do nothing)
(do nothing)
> ./forwarder -relay=4444 -nodes=2 -index=1
(do nothing)
(do nothing)
^C (terminate forwarder)

Pushing to the limit

As a conclusion to this tutorial, I was curious what the performance of the Chinese whisperers was in a distributed environment. But since implementing a sophisticated benchmark was beyond the scope of this release, I figured I'd modify the initiator node to push messages as fast as capable into the system and see what happens.

The tests were ran on the Atlasz HPC cluster of the Eötvös Loránd University, each node consisting of 2x Intel Xeon E5520 processors, 12GB memory and a 1Gbit Ethernet connection. The reported global throughput of the system was measured as the broadcasts that actually looped through the whole ring (e.g. a 256 byte message passed through 16 nodes counts as 4KB, but the same message lost at the 15th node is considered 0). The tests were repeated 5 times, ran for 20 seconds, first 2s warm-up and last 1s tear-down discarded, results averaged out.


Chinese whispers global message throughput (Google Charts)

First of all, a ring benchmark is a very inappropriate way to measure a decentralized backend framework, since many of its strengths, such as load balancing, application clusters and optimized distribution pathways are thrown out the window when each instance is addressed directly. Nonetheless, the experience was an enlightening one, showing both strengths and current weaknesses of the system.

The strength of Iris is that it actually does deliver on the promise of high throughput, completely decentralized, fully secured yet extremely simple way of programming backend services. For a thorough analysis a more appropriate benchmark is called for, but the noticed weak points were mostly related to overloading:

  • The benchmark was pushing messages into the system as fast as that would allow, filling up both network links as well as consuming available CPU resources. This sometimes lead to internal heartbeats timing out and the ring temporarily breaking up. Further investigation is needed as to exactly where and why the heartbeat messages got stalled, but the most probable issue is goroutine starvation.
  • Currently there is no throttling mechanism in place client side, nor memory caps Iris side, which – with the appropriate message speeds and sizes – can actually fill up message buffers till all available memory is consumed, crashing the process. Fixing this is not too involved, but I would also like to add some memory optimizations while at it (sync.Pool, reusable messages, etc), so it got delayed till the next release.
  • Looking at the chart above, you can see that messages below 1024 bytes perform significantly poorer than their bigger counterparts. The reason is the processing- and bandwidth overhead (encryption and added headers) associated with each message. This will also be addressed in a future release via message grouping.

Closing remarks

Plenty to see, plenty to do. Although Iris is coming along nicely, there are still rough edges need sorting out (see above) as well as a few essential features needed adding: leveled logging, statistics reporting and federation.

Although I would not recommend using Iris in a live production system – unless you're sure of yourself, did your homework and took care of the overloading issues – it was successfully applied in RegionRank, running non-stop for 61 days now (small number in the service world, but large enough to make a point).

Finally, as to what the future holds, consider Iris an enormous amount of work, developed by a single PhD student, unsupported by his own university for the better half of a year. I am putting a lot of faith into this project, but without an active supporter, the time will come when I can no longer afford to do so. If not otherwise, help spread the word and get the project some coverage.

Try it, code it, share it! :)