Build on-device AI features: when local inference makes sense, model size and memory constraints, battery and thermal limits, hybrid architectures, and privacy gains.
On-device AI development runs models on the user's phone, laptop, or embedded hardware rather than in a data centre. The appeal is privacy, latency, offline capability, and zero marginal inference cost. The constraints are memory, battery, thermal limits, and device fragmentation.
Most successful implementations are hybrid, running some work locally and some remotely.
When local inference is the right choice
Privacy-sensitive processing. Data that should not leave the device at all, such as personal message content, health data, or camera input. This is the strongest argument and often the deciding one.
Latency-critical interaction. Features that must respond within tens of milliseconds, such as live transcription, keyboard prediction, or camera effects, where a network round trip is disqualifying.
Offline requirement. Applications used in poor connectivity, including field work, transport, and rural deployments.
High-volume, low-complexity inference, where per-request cloud cost would dominate the economics.
The real constraints
Memory. Model weights must fit alongside your application in a memory budget the operating system may reclaim. Mobile platforms terminate apps that exceed limits, so a model that loads on a flagship device may crash on a mid-range one.
Battery and thermal. Sustained inference drains battery noticeably and generates heat, after which the device throttles and your performance measurements no longer hold. Test sustained use, not a single inference.
Fragmentation. Accelerator availability, supported operations, and quantization support vary widely across devices. A feature working on recent hardware may be unusable on the devices much of your user base actually owns.
Distribution size. Bundling model weights inflates app size, which measurably reduces install rates. Downloading after install adds a first-run delay and a failure path to handle.
Architecture patterns
Local-first with cloud fallback. Run locally when the device supports it, fall back to a server when it does not or when the task exceeds local capability. This handles fragmentation without excluding users.
Local pre-processing. Run extraction, filtering, or redaction on device, then send only what is necessary to a server. This preserves much of the privacy benefit while allowing a larger model to do the heavy work.
Local for interactive, cloud for batch. Immediate feedback comes from the device, while heavier processing runs remotely when connectivity and power allow.
Progressive capability. Detect device capability at runtime and enable the appropriate tier, communicating honestly rather than silently degrading.
Model selection and optimization
Choose the smallest model meeting your quality threshold, measured on your own task rather than on benchmarks.
Quantization is essential for practical deployment and involves a quality trade-off that varies by task. Evaluate the quantized model rather than assuming parity with full precision.
Use the platform's inference frameworks and accelerators rather than generic runtimes, since the difference in performance and power draw is substantial.
Measure on the devices your users actually have, weighted by your install base. Testing on current flagship hardware produces numbers that do not represent your users.
Update and lifecycle
Model updates ship separately from app updates in most designs, which requires versioning, integrity verification, and a rollback path.
Plan storage carefully. Users notice large model files, and platforms may evict cached data under pressure, so handle absence gracefully.
Keep the server path maintained even when most inference is local, since it is your fallback for unsupported devices and failed downloads.
Frequently asked questions
Which tasks run acceptably on device today?
Transcription, translation, classification, extraction, image processing, and short-form generation are all practical. Long-context reasoning generally is not.
Does on-device inference remove privacy obligations?
It reduces transmission risk substantially. Obligations around storage, consent, and processing on the device still apply.
How much does it save?
Marginal inference cost approaches zero, offset by development complexity and support burden across device variation.
Should we start local or cloud?
Usually cloud first to validate the feature, then move suitable parts on device once requirements are proven.
Explore our software and web capabilities or discuss your product constraints.
Written by
Moueen Togarvi
Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.