The DADI decentralized network consists of three main layers:

• Stargates: authoritative DNS and security layer
• Gateways: application gateway layer
• Hosts: application host layer

The services that make up these layers need to be able to discover each other, make decisions based on service metadata, and collectively manage the state of the network as a whole.

ZooKeeper

The DADI network started life with Apache ZooKeeper at its heart, making use of its hierarchical tree based key value storage. ZooKeeper is a centralized key-value storage service trusted by the likes of Rackspace, Yahoo and eBay. Services would create ephemeral "ZNodes", or key value pairs, in directories relating to their state. For example, a Gateway that is waiting for more Hosts would sit in gateways/awaiting_hosts and Hosts looking for a Gateway would iterate over this directory, attempting to connect to an available Gateway.

If a service disconnected, its associated key value pairs would disappear. If the service's state changed, the previous key value pair would be removed (hosts/awaiting_gateway/<uuid>), and a new key value pair would be created (hosts/live/<uuid>).

In addition to the location of the key value pairs representing state (<service>/<state>/<uuid>), each key value pair would also hold service metadata, consisting of information such as geolocation, user applications, and public IP address etc.

This worked well at a low request level, but while ZooKeeper is known for its fast read operations, it's not so pacy when it comes to write operations. When scaling out to thousands of services where those services need to change their states, update their metadata and discover other services on a regular basis, the performance needed just couldn't be guaranteed.

It's worth noting that we also encountered a number of glitches in ZooKeeper which were extremely hard to investigate owing to the lack of management tools available.