Machine Learning at the Edge
AI workloads are moving out of central data centres and closer to users. Here's what edge ML means, why latency is the driving force, and how to deploy models on distributed infrastructure.
What is machine learning at the edge?
Machine learning at the edge means running ML models — most often inference, the stage where a trained model makes predictions — on infrastructure located close to where the data is generated and the results are consumed, rather than in a distant central cloud region.
Training usually stays in large clusters, because it needs sustained access to big datasets and lots of parallel compute. Inference is different: it happens on every user request, its output is needed immediately, and it benefits enormously from being nearby. That split — train centrally, infer at the edge — is the pattern behind most edge ML deployments, and it mirrors how CDNs and edge computing divide work more generally.
Why latency matters for AI workloads
An inference request's response time has two parts: the time the model spends computing, and the time the request and response spend travelling over the network. You can buy faster hardware to attack the first part, but the second is governed by distance. A user in Singapore calling a model hosted in Virginia pays a transcontinental round trip on every single request, before the model does any work at all.
For interactive AI — chat interfaces, voice assistants, live translation, recommendations, fraud checks in a payment flow — that network tax is the difference between an experience that feels instant and one that feels sluggish. Streaming LLM responses makes it worse, not better: every token round trip inherits the same latency floor.
Moving inference to nodes near the user removes most of the travel time. It also reduces how much raw data has to cross the internet: a computer-vision system that analyses video at the edge sends back results, not footage.
GPU compute at the edge
Modern models — particularly LLMs and vision models — run far faster on GPUs than CPUs, so edge ML at scale means putting GPU capacity into the distributed network itself. This is a natural fit for a decentralised platform: the same model that distributes nodes for content delivery can distribute GPU nodes for inference.
Edge is bringing GPU instances to its network, with support for multiple vendors including NVIDIA, AMD, and Intel, billed hourly with no long-term commitments. In the meantime, Edge Compute VMs run CPU-based inference today, and smaller quantised models are genuinely practical on CPU.
The economics matter as much as the hardware. Model serving moves a lot of data, and on traditional clouds that means egress charges on every response. Edge charges zero egress fees, so serving predictions — or shipping model weights between environments — carries no bandwidth cost. See AI & Machine Learning on Edge for the full picture.
Deploying models: Ollama, vLLM, and llama.cpp
You don't need a bespoke ML platform to serve models on edge infrastructure. The open source ecosystem has settled on a few excellent serving tools, each of which deploys cleanly to an Edge VM:
Ollama
The simplest way to run open models like Llama and Mistral on your own VM, with a clean local API.
vLLM
High-throughput serving for production LLM workloads, with efficient batching and an OpenAI-compatible API.
llama.cpp
Lightweight, CPU-friendly inference for quantised models — useful where GPU capacity is unnecessary or unavailable.
Why run ML at the edge?
The gains compound: faster responses, lower bills, and less data in transit.
Lower Latency
Inference happens close to the user, cutting the round trip that dominates response time for real-time AI features.
Lower Cost
Moving less data to and from a central cloud means less bandwidth spend — and on Edge, egress is free entirely.
Better Privacy
Data can be processed near where it is generated rather than shipped to a distant region, reducing exposure in transit.
Resilience
Distributed inference keeps working when any single location fails — there is no central endpoint to go down.
Run your models on Edge
Deploy inference on Edge Compute today, and join the waitlist for GPU instances.