Message Dissemination with Epidemic Algorithms in Onyx

Abstract

The Onyx Platform is a data processing framework, that utilizes a masterless coordination design and a centralized log through the ZooKeeper system. At large cluster sizes, the centralized log experiences performance issues due to the large amount of read and write requests. This thesis utilizes epidemic techniques for sharing log events in order to reduce read requests to the primary log nodes. An implementation of these techniques will be presented, together with an analysis of the results. Problems with actually applying the received log events to the local state made realistic performance testing impossible, but the actual epidemic message dissemination show some promising results with a high degree of connectivity and small average shortest path between nodes.