LFM2.5-1.2B-Thinking — LiteRT-LM

LiquidAI/LFM2.5-1.2B-Thinking converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 / a recent AI Edge Gallery). Sibling of litert-community/LFM2.5-1.2B-Instruct.

Update (2026-08-04): the .litertlm files were updated in place to add the ExecutorMetadata section that litert-lm ≥ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with missing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files continue to run on litert-lm 0.14.

LFM2.5-1.2B-Thinking is the reasoning variant of Liquid AI's hybrid conv-attention flagship: it works problems inside <think>…</think> before answering. The bundle declares the thought channel in its metadata, so LiteRT-LM ≥ 0.14 streams the reasoning on a separate thought channel — your app can show or hide it natively, and past thinking is stripped from multi-turn context automatically.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-1.2B-Thinking_int8.litertlm	int8 dynamic (linears + embedding; convs float)	1.24 GB	77% (bf16 reference: 81%)
LFM2.5-1.2B-Thinking_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	736 MB	72%
LFM2.5-1.2B-Thinking_int4_gpu.litertlm	same int4 recipe, re-exported so it runs on the GPU	736 MB	= int4 (see notes)

Context (KV cache)	4096 max
Backend	CPU for every file; the _int4_gpu file also runs on the GPU with litert-lm ≥ 0.16.0 (Android OpenCL and macOS — verified by generation, not just by a benchmark table; iOS Metal still fails at engine creation, tracked upstream in LiteRT-LM#3129)
Template	bundled — full chat template + thought channel (<think>/</think>)
Base model	LiquidAI/LFM2.5-1.2B-Thinking (LFM Open License v1.0)

Accuracy

GSM8K (greedy, 0-shot CoT, max-tokens 2048 — a thinking model needs the budget, n=100, same harness for all rows): PyTorch bf16 81% · LiteRT int8 77% (−4pt) · int4 72% (−9pt). Both files pass an 8-question sanity gate (7/8, zero degenerate); the reasoning stream arrives on the thought channel and the final answer follows cleanly after </think>.

Usage

litert-lm run ./LFM2.5-1.2B-Thinking_int8.litertlm --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?"

Give it a generous token budget (≥2048) — a reasoning model truncated mid-thought produces no final answer. The reasoning arrives on the thought channel; the final answer arrives on the main text channel after </think>.

GPU — the _int4_gpu file

Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:

Backend	Prefill (263 tok)	Decode	TTFT
GPU (OpenCL)	189–192 tok/s	21.8–22.4 tok/s	1.41–1.44 s
CPU	31.3–31.7 tok/s	16.3–16.7 tok/s	8.4–8.5 s

On phone-class hardware the GPU's win is prefill and time-to-first-token (3–6× both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.

Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):

Backend	Prefill (256)	Decode	TTFT
GPU	3734 tok/s	317.9 tok/s	0.07 s
CPU	334.4 tok/s	79.2 tok/s	0.78 s

On a desktop GPU the gain is across the board — roughly 11× prefill and 4× decode against the same file on CPU.

Run on Android

Install a recent Google AI Edge Gallery, import this repo (or adb push a file and use local import: menu → Models → “+” → From local model file), select the CPU backend, set max tokens high (2048–4096), and chat. GPU acceleration needs the _int4_gpu file and litert-lm ≥ 0.16.0; the Gallery bundles its own runtime, which may lag that version — if its GPU toggle fails there, use CPU.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, CPU backend, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine. Decode on this family depends strongly on the KV budget, so both settings are listed:

Variant	--max-num-tokens	Prefill (256)	Decode	TTFT
int8	1024	1536 tok/s	98.8 tok/s	0.18 s
int4	1024	386 tok/s	118.9 tok/s	0.67 s
int8	4096	1121 tok/s	83.2 tok/s	0.24 s
int4	4096	342 tok/s	77.4 tok/s	0.76 s

Set --max-num-tokens to the smallest value your use case needs — 1024 is a good chat default, and the file allows up to 4096. At 1024 the int4 file decodes fastest; at 4096 the two variants converge.

Use the CPU backend — this bundle cannot create a GPU engine. It comes from the pre-0.9.2 ShortConv export generation, whose prefill graph still carries INT64 ADD/CAST inside Lfm2ShortConv, plus GATHER_ND and a GREATER_EQUAL with const inputs. The GPU delegate takes 536 of the 579 operations and leaves 43 on the CPU, and the runtime then refuses the partial split: Hint fully delegated to single delegate is set, but the graph is not fully delegated. Re-exporting from the post-0.9.2 lineage removes those INT64 ops and does run fully delegated on the macOS GPU; the remaining iOS Metal failure is tracked upstream in LiteRT-LM#3129.

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery) int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is about 1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices. Those Android figures are single ship-gate runs, not medians. First device load compiles the graph and can take about a minute; later loads are instant.

Known limitation — multi-turn conversations on litert-lm >= 0.15

On the current runtime a conversation's context retains previous turns' <think> reasoning (history is appended, not re-rendered), while the model was trained with past reasoning stripped from its context. Single-turn quality is unaffected, but across several turns in one conversation the model can drift — a later answer may repeat an earlier turn's reply instead of addressing the new question (observed at turn 3 of a 3-turn probe, litert-lm 0.15.0 CPU, both variants). Practical guidance: start a fresh conversation per task (conversation objects are cheap; the engine can stay loaded), and size --max-num-tokens so thinking turns complete — a reply truncated mid-<think> derails the turns after it.

Conversion notes

Converted with released litert-torch 0.9.1 with the same exporter fix as the Instruct sibling: the stock LFM2 short-conv block saves its conv state from padded prefill columns, corrupting the first generated token of nearly every reply; the fix derives the chunk's valid length from the attention mask in-graph and gathers the state from the last valid columns (verified token-identical to an exact per-token reference loop). Multi-length prefill signatures (1–1024). Quantization: export-time int8 including convs, or post-hoc int4 on linears only — post-hoc conv quantization breaks generation.

Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both — the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm ≥ 0.15 needs. It delegates fully on Android OpenCL — 501/501 and 519/519 nodes, zero rejected ops — and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity gate on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.