small on-device llms

ranked on what they need in memory at 4-bit, what the licence permits, and what they're genuinely good at — not on benchmark places they were never going to win.

5 listed · 5 researched by us · featured first, then most upvoted

Gemma 4 (E2B / E4B / 12B)

we checked thisfree (apache 2.0)

text, images and audio on a phone, under an apache licence google took three generations to arrive at.

#anything-that-needs-to-s#on-hardware-you-already-

Qwen3.5-4B-Instruct

we checked thisfree (apache 2.0)

262k of context in about three gigabytes, apache 2.0, and two generations newer than the qwen everyone links to.

#long-document-work-on-a-#where-the-context-window

gpt-oss-20b

we checked thisfree (apache 2.0)

openai's small one — 21b of capacity in 16gb, with reasoning effort you can dial down.

#agentic-tool-calling-on-#where-you-want-to-trade-

Granite 4.1-3B

we checked thisfree (apache 2.0)

apache 2.0 with cryptographically signed weights — the enterprise answer at edge scale.

#regulated-environments-t#not-just-a-permissive-li

Phi-4-mini-instruct

we checked thisfree (mit)

plain mit, strong function calling, and the oldest model in this ranking by a year.

#structured-output#tool-calling-in-a-small-#where-reliability-beats-

this page is deliberately short: a handful of small on-device llms we looked at properly, rather than everything that exists. if you build one that belongs here, a free listing takes two minutes and $50 pins it to the top of this page.

add a tool →