An Introduction to Telemetry

Behind the firewall

The Edge Network is comprised of three device types: Stargates, Gateways, and Hosts. The latter devices—Hosts—operate on such a variety of networks, with such a variety of performance and restrictions, that we have to be able to receive device measurements from them as and when the devices are online and able to report, regardless of limitations such as firewalls or patchy connectivity.

All Hosts run a background process called the Telemetry Client which sidesteps these restrictions by broadcasting to the network's telemetry receivers over HTTPS. Both HTTP on port 80 and HTTPS on port 443 are generally open in firewall egress ruletables, which is the basis on which the network runs it's gRPC connections between Hosts and Gateways.

Collecting the metrics

The Telemetry Client collects basic system measurements such as load averages, memory usage, disk space utilisation and network activity. It also collects some other less interesting statistics such as system fork count, active process count, and established TCP connection counts.

This data is then packaged up and sent securely to the network's telemetry receiver servers, which sit outside of the main network infrastructure — critical to remaining accessible during possible outages — before being verified and stored alongside the metrics of all other devices.