The DADI decentralized network consists of three main layers:
- Stargates: authoritative DNS and security layer
- Gateways: application gateway layer
- Hosts: application host layer
The services that make up these layers need to be able to discover each other, make decisions based on service metadata, and collectively manage the state of the network as a whole.
🔗ZooKeeper
The DADI network started life with Apache ZooKeeper at its heart, making use of its hierarchical tree based key value storage. ZooKeeper is a centralized key-value storage service trusted by the likes of Rackspace, Yahoo and eBay. Services would create ephemeral “ZNodes”, or key value pairs, in directories relating to their state. For example, a Gateway that is waiting for more Hosts would sit in gateways/awaiting_hosts and Hosts looking for a Gateway would iterate over this directory, attempting to connect to an available Gateway.
If a service disconnected, its associated key value pairs would disappear. If the service’s state changed, the previous key value pair would be removed (hosts/awaiting_gateway/<uuid>), and a new key value pair would be created (hosts/live/<uuid>).
In addition to the location of the key value pairs representing state (<service>/<state>/<uuid>), each key value pair would also hold service metadata, consisting of information such as geolocation, user applications, and public IP address etc.
This worked well at a low request level, but while ZooKeeper is known for its fast read operations, it’s not so pacy when it comes to write operations. When scaling out to thousands of services where those services need to change their states, update their metadata and discover other services on a regular basis, the performance needed just couldn’t be guaranteed.
It’s worth noting that we also encountered a number of glitches in ZooKeeper which were extremely hard to investigate owing to the lack of management tools available.
🔗Consul
We worked with a number of alternatives to ZooKeeper, the best of which were etcd from CoreOS and Consul from Hashicorp. While etcd offers a lot of the features required for the DADI network, Consul was the stand out choice.
🔗Service discovery
Service discovery is conspicuous by its absence in ZooKeeper. Picture this for a moment: you have Stargates running across every continent on Earth, Gateways running in every region, Hosts that number in the hundreds of thousands dotting the globe like streetlights, and among this living network, innumerable websites & web services running in harmony.
You type www.cat.blog into your browser and press return. Your machine attempts to resolve this domain name to an IP, and receives a list of IPs for the authoritative DNS servers – our Stargate layer – which it then queries for the DNS record for www.cat.blog.
At this point, the Stargate service needs to return a single IP address for a Gateway. It needs to be able to quickly and efficiently determine which Gateways are running the cat-blog application, and retrieve enough information to make an intelligent decision as to which IP address to return.
Using ZooKeeper would require either an iterative approach to reading Gateway key value pairs or a more intelligent data structure layered on top of this. DADI Stargate would need to iterate through each Gateway entry, checking application status and geographic metadata amongst other things. This isn’t a solution that will scale well.
🔗Service discovery as a first class citizen
Consul has service discovery built in. Each service, whether it is Stargate, Gateway or Host, will register itself with Consul. The metadata will be stored in a key value pair (gateways/<uuid>) and its state will be associated with the service registration in the form of a tag, which can be used later when querying the service registry.
So let’s see how Stargate would find the correct Gateway. We’ve typed www.cat.blog into our browser and pressed return. Our machine has queried a Stargate for an IP. The Stargate now queries Consul for all Gateway services that are running the cat-blog application. Consul returns a number of Gateways, all of which have Hosts connected that are running the cat-blog application.
DADI Stargate then intelligently determines the optimal Gateway and returns its IP. Our machine now connects to that Gateway and asks for www.cat.blog, a request which is then picked up by a Host connected to that Gateway, processed, verified and passed back to our machine.
🔗Better performance
As part of the due dilligence on the technologies reviewed for use at this level in the DADI network, we ran a high number of performance tests designed to simulate heavy network activity. Primarily we tested key value read/write operations: a key value pair would be written to, read and written to again, all the way from 0 to 100, ensuring the value had written before it was iterated.
We tested ZooKeeper running a single instance on a small machine based in Paris. We ran Consul both as a standalone server, as well as a group of a server and a client on a LAN network, again on small 2 core amd64 machines. To simulate real world network conditions, test applications ran over a WAN connection from the UK.
🔗Benchmarking
The first set of tests were on an empty server. Average atomic write/read transactions took ~77ms for Consul, and ~150ms for ZooKeeper.
🔗Adding traffic
Next, we added network activity. 100 test services reading and writing key value data for ZooKeeper; changing ZNode locations to simulate state change for Consul; registration of services with differing states; and finally, random disconnections.
An average atomic write/read transaction took ~97ms for Consul under these conditions, and ~201ms for ZooKeeper. Interestingly, even when we tested through Consul clients rather than direct connections to Consul servers, the transactions were still more performant than ZooKeeper.
🔗More traffic
We added more network activity: 250 very noisy test services. Consul completed atomic write/read transactions in ~115ms. ZooKeeper in ~309ms.
Finally, we added 10 very intensive services, completing back to back read/write operations and endless updates, all running concurrently. Consul completed write/read transactions in ~129ms. ZooKeeper in ~396ms.
Throughout the tests, Consul remained responsive to queries and its host machine relatively untaxed. With ZooKeeper we saw intensive CPU usage and the occassional spike. With Consul we also saw less network traffic overall, both with direct and indirect tests.
Read/write operations were consistently twice as fast under Consul, with service discovery lookups seemingly unaffected by the amount of network activity.
🔗Conclusion
We’re working on switching out ZooKeeper for Consul. We know that Consul provides better performance for what the DADI network needs, and critically it has built in service discovery. We’ve already built a proof of concept with Consul, and we’ll be implementing this for real over the coming days & weeks.
As part of this work, we are changing how we handle states and are introducing finite state machines to better help manage these. We’ll be writing more about that soon.
Do you have any questions about how we’re using Consul? Why not pop along to the first of our new fortnightly AMAs on Telegram, Discord and Reddit.
