best 8 small on-device llms

all three sizes handle text, images and audio, including speech recognition and translation. nothing else in this ranking is multimodal at all, and gemma does it in a footprint that fits a phone at e2b and a modest laptop at e4b. that combination is the entire case and it's a strong one.

the licence change is real and underreported. gemma 1 through 3 shipped under google's custom gemma terms of use, with a prohibited-use policy and redistribution obligations — widely called open, never actually open source. gemma 4's own terms page carves itself out and points at plain apache 2.0. google still hosts a prohibited-use page on the same site, but it is not incorporated into the gemma 4 grant.

the honest limits: the 12b dense variant wants real gpu or unified memory to feel quick, and the family trades reasoning depth for footprint against its larger siblings. context is confirmed at 128k on the e4b card, with google's materials claiming up to 256k across the family that we could not verify per size.

262,144 tokens of native context in a model that needs about three gigabytes at four-bit is the standout number in this category. most rivals sit at 128k or below, and the ones that match it are larger. alibaba reports a nine-point gain on its own intelligence index over qwen3-4b at identical size and licence — its figure, not an independent one.

the licence is standard apache 2.0 with no appendix, consistent across the qwen3 and qwen3.5 dense lines, which makes it one of the cleanest commercial stories here.

two cautions. the reasoning gain comes with heavier token usage per answer, so on constrained hardware complex questions get slower and not just better. and the versioning has moved fast — qwen3.5 landed in march 2026 and qwen3.6 in april, so the qwen3-4b that most guides still recommend is two steps behind, even though it runs fine under the same terms.

21b total parameters with 3.6b active, shipped post-trained in mxfp4 so the native format is already roughly four-bit, and openai states it fits within 16gb. that is the top of the consumer envelope rather than the middle, but it buys noticeably more capability than the 3-4b models around it.

configurable reasoning effort — low, medium, high — is the feature worth having. most small models make you choose a model per latency budget; this one lets you choose per request, which matters when the same application does both quick lookups and hard problems. tool-calling in openai's harmony format is strong.

two caveats. hitting the 16gb figure depends on mxfp4 kernel and hardware support; loading naively in bf16 needs substantially more. and openai publishes a separate gpt-oss usage policy that isn't part of the apache licence file — we could not confirm whether it is contractually binding or advisory like google's, so check it before assuming the apache grant is the whole story.

ibm signs the weights cryptographically, which nothing else in this ranking does. for anyone who has to answer where a model came from and whether it was tampered with in transit, that is a category of assurance the licence alone doesn't provide.

granite 4.1 also marks a real architecture change rather than a version bump: ibm moved from the hybrid state-space and moe designs of granite 4.0 nano to a pure dense line, and says the recipe lets its 8b sibling match the old 32b moe model. the 3b inherits it at about 2gb, running on a laptop cpu or a low-end gpu.

the cost of being current is being new. released 29 april 2026, it has far less third-party benchmarking and community tooling than qwen or gemma at the same size, and no named leaderboard entry we could verify. note also that 4.1 doesn't go below 3b — if you need something smaller from ibm, granite 4.0 h 350m is still the answer.

the licence is plain mit with no field-of-use limits, revenue cap or naming mandate, which is the simplest commercial story on this page. at 3.8b and roughly 2.5gb quantised it runs comfortably on any modern laptop, with 128k context and an unusually large 200k-token vocabulary that helps on multilingual input.

what it does well is follow instructions and call functions reliably at a size where that usually degrades. for pipelines that need structured output rather than conversation, that reliability is worth more than a benchmark point.

two practical notes. the flash-attention path assumes datacentre cards — a100 or h100 — so consumer gpus fall back to slower eager attention, which blunts the speed advantage its size implies. and it dates from february 2025; blogs referencing a 'phi-5' family exist but we could not confirm one on microsoft's own hugging face organisation or azure blog, so treat that as rumour.

76.7% on ifeval against qwen2.5-3b's 65.6% is a wide margin for instruction following at this size, and the dual think and no-think modes let you spend reasoning tokens only when a task warrants it. at roughly 2gb quantised it runs on anything, cpu included.

apache 2.0 with no conditions, from hugging face, which has a stronger institutional commitment to publishing training details than any commercial vendor on this page.

the number to know before relying on it: long-context recall trails its class, at 67.85% on ruler-64k against llama 3.2's 72.93%. the 128k figure comes from yarn extension over a 64k-native training window, so treat the far end of that context as softer than the spec implies. no successor has appeared since july 2025.

the architecture is the appeal: 8.3b total parameters with only 1.5b active per token, so it generates at roughly the speed of a 1.5b model while carrying far more capacity. it runs across llama.cpp, mlx, vllm and sglang, so apple silicon gets a native path.

the memory maths is the thing people get wrong about moe, and it's worth stating plainly: all 8.3b parameters must be resident, about 5gb at four-bit. mixture-of-experts shrinks compute, not footprint. this is not a 1.5b model's memory profile.

the licence is the real catch. the lfm open license v1.0 grants free use 'for any purpose — including commercial products and services' only to companies under $10m in annual revenue; above that you need to buy a commercial licence. that reads like apache-style permissiveness and isn't, and it is the wrong direction of surprise for anything you're building a business on. liquid's headline 18,500 tokens per second is also measured on an h100, not on your laptop.

the technical story is genuinely interesting. it's a hybrid mamba-2 design with just four attention layers, compressed from the 9b v2 model through nvidia's nemotron elastic framework, and explicitly built for cpu-only inference. 262k of context at around 2.5gb without needing a graphics card is a combination nothing else here offers.

the licence has teeth that apache 2.0 doesn't. commercial use and redistribution are permitted and nvidia disclaims ownership of outputs — but your rights terminate if you sue over the model, they terminate if you 'bypass, disable, reduce the efficacy of, or circumvent' any safety guardrail in it, and nvidia reserves the right to update the licence unilaterally to meet legal or regulatory requirements, leaving you to accept the new terms or stop using and distributing it.

the anti-circumvention clause is a behavioural restriction, not paperwork — it constrains what you may do with weights on your own hardware. combined with a training cutoff of september 2024, noticeably staler than everything else in this ranking, it's a specialised choice rather than a default one.