File size: 11,852 Bytes

---
license: apache-2.0
base_model: openbmb/MiniCPM5-2B
pipeline_tag: text-generation
library_name: coreai
tags:
- coreai
- core-ai
- coreml
- apple
- on-device
- iphone
- metal
base_model_relation: quantized
---

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->
This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
<!-- gen-cards:devicemark end -->

# MiniCPM5-2B — Core AI (int8 block-32, runs on iPhone)

Apple **Core AI** (`.aimodel`) conversion of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) —
OpenBMB's 2.5B on-device LLM (released 2026-09-06, the 1B's 42-layer sibling) with **hybrid
Think / No-Think reasoning**, native tool calling and **128K** context; OpenBMB reports it as
2B-class open-source SOTA (LiveCodeBench v6 69.1, AIME 2026 86.5, BFCL v4 66.6, SWE-bench Verified
46.4 on their card). Runs fully on-device on **iPhone** and Apple Silicon Macs (GPU, pipelined engine).

Part of the community Core AI model zoo: **https://github.com/john-rocky/coreai-model-zoo**

<!-- gen-cards:use-it begin id=minicpm5-2b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
## Use it

⚡ **One line** — run the kit's task op on this model
(`import CoreAIOps`; no session, no model plumbing, downloads on first use):

```swift
let tldr = try await CoreAI.summarize(text, options: .model("minicpm5-2b"))
```

Every op, one shape — [Cookbook](https://github.com/john-rocky/coreai-kit/blob/main/docs/COOKBOOK.md).

▶️ **Run it (source)** — the [ChatDemo runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/ChatDemo)
(GUI + CLI, one app for every chat model in the catalog):

```bash
git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "MiniCPM5 2B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model minicpm5-2b --prompt "What can you do, offline?"
```

💻 **Build with it** — complete; the glue is kit API, copy-paste runs:

```swift
import CoreAIKit

let chat = try await ChatSession(catalog: "minicpm5-2b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device
```

The take-home is [`Examples/ChatDemo/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/ChatDemo/Sources/QuickStart.swift)
— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
the GUI drives the same `ChatSession` across turns for its transcript.
Multi-turn? Hold the `ChatSession` and call `respond(to:)` per turn — it keeps the
conversation history; `streamResponse(to:)` yields tokens as they decode.

**Integration checklist**

- SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKit**
- Info.plist: none needed
- Entitlements: none on Mac; iPhone needs `com.apple.developer.kernel.increased-memory-limit` (the 2.7 GB cold specialization passes the default jetsam limit)
- First run downloads the model — 2.7 GB (Mac) / 2.7 GB (iPhone) — then it loads from the
  local cache (Application Support; progress via the `downloadProgress` callback)
- Measure in Release — Debug is ~3× slower on per-token host work
<!-- gen-cards:use-it end -->

## Neural Engine bundle (iOS static export, AOT h18p) — 2026-09-16

`ios-ane-h18p/` is Apple's stock `coreai.llm.export --platform iOS` static export of this checkpoint at
**6-bit k-means palettization, group 8** (the shape of Apple's own iOS preset for Qwen3-1.7B;
`conversion/minicpm5_pal6_g8.yaml` in the zoo; embeddings int8; static graphs `prompt_opt`/`extend` ×
contexts {256, 512, 1024, 2048, 4096} × query {8, 16, 64}), AOT-compiled with
`xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p`
(31/31 ANE regions, 2.5 GB). It loads through Apple's `EngineFactory` → `StaticShapeEngine` unchanged
(iPhone 17-class devices). `ios-static/` is the same export before AOT — the portable IR; compile it
for another chip yourself.

Why 6-bit: the 2026-09-15 bundle was Apple's iOS default (4-bit, group 32). It passed the three short
gate prompts but diverged from fp32 at the first token of a 109-token free-form answer (and got the
physics wrong). At 6-bit the same phone matches fp32 token-for-token on the three short prompts (24/24,
8/8 including the stop, 16/16) and for 33 tokens of the long answer, then diverges on a step where fp32
itself is nearly tied (top-2 margin 0.17); 5 of its 109 teacher-forced steps differ — transcript
`models/minicpm5-2b/gate-minicpm5-2b-ane-6bit-device.json` in the zoo. An 8-bit shape is exact in an
fp32 simulation of the recipe, but the 2B at 8 bits (2.16 GB of weights) loads on the ANE and never
returns its first token, and 4-bit group 8 does not compile for the ANE at all — zoo
`knowledge/ane-vs-gpu-iphone-2026-09.md` §9.

Speed on an iPhone 17 Pro (iOS 27.0), same day, engine released and idle until the thermal state was
back at `fair`, then 60 s of 128-token-prompt / 256-token trials: **38.5 tok/s decode** (flat over the
minute), prefill 1679 tok/s, footprint 2.6 GB. In the same run the `int8/` GPU bundle decoded 23.8 tok/s;
the previous 4-bit ANE bundle did ~55 (`models/minicpm5-2b/bench-iphone-ane-6bit-vs-gpu-2026-09-16.json`).
**The first launch on a phone builds the ANE programs: about 23 minutes** (0.2 s afterwards; the cache
lives in the app's container and is invalidated by an iOS update).

Task accuracy, GSM8K test (first 200 questions, 0-shot CoT, greedy, no-think, max 640 new tokens; same
prompt and scoring as the litertlm-convert evals) — `models/minicpm5-2b/gsm8k-200-2026-09-17.json` in the zoo:

| model | correct / 200 |
|---|---:|
| fp32 checkpoint (bf16 on a Mac) | 172 (86.0 %) |
| **`ios-ane-h18p/` 6-bit, on the iPhone 17 Pro** | **173 (86.5 %)** |
| the same 6-bit recipe applied to the fp32 weights (Mac) | 172 (86.0 %) |
| the `int8/` recipe applied to the fp32 weights (Mac) | 171 (85.5 %) |
| the replaced 4-bit bundle, on the iPhone 17 Pro | 131 (65.5 %) |

## Measured

| | decode | prefill | numerics | size |
|---|---:|---:|---|---:|
| **iPhone 17 Pro** (A19 Pro, `PipelinedBench`, Release) | **22.4 tok/s** | 27.3 tok/s | **24/24 + 24/24 token-exact** vs HF fp32 (nat + oracle, the margin-clean alphabet prompt); engine ready 28.9 s cold | **2.7 GB** |
| **M4 Max** (macOS 27, `llm-benchmark`) | **127.6 tok/s** | 2654 tok/s | **16/16 token-exact** vs the fp32 oracle (margin-aware gate, min margin 0.925) | |

Free-run check (4 prompts × 30 greedy tokens vs fp32 HF, `verify_minicpm5.py`): **3/4 exact**; the one miss is a name at fp32 probability 0.2126 vs 0.2065 (`Emma`/`Lily`, top-2 margin 0.006) — a tie any precision may flip. The fp16 control export scores 4/4, and the **per-channel** int8 sibling of this bundle scored 2/4 with a real 0.245-margin flip (`,`→` and`), which is why this repo ships per-**block-32** scales instead (same recipe, three YAML lines; see *Quantization*).

⚠️ **iPhone context cap: prompt + generated tokens must stay under 1024.** The bundle declares a 131072
dynamic KV, and the shipped `CoreAIPipelinedEngine` caps iOS growing-KV capacity at 1024 (its guard
against the iOS compiler miscompiling growing-KV specializations at seq ≥ 2048) — so a phone
conversation truncates at absolute position 1024. Chunk or trim the history on iOS; macOS has no cap.

Same recipe as the published [MiniCPM5-1B](https://huggingface.co/mlboydaisuke/MiniCPM5-1B-CoreAI)
(int8 66.8 tok/s on the same phone) with one YAML changed — **per-block-32 scales instead of
per-channel**. Measured on the Mac before picking it (`llm-benchmark`, 512p/1024g): int8
per-channel 25.6 tok/s, fp16 80.0, int8 per-block-32 **127.6**. Per-channel int8 lowers to a
slow dequant path on the Mac GPU; block-32 lands on the fast quantized-matmul path, 5× the
per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
(bandwidth-bound), so block-32 wins on both.

## Quantization

Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
clipping), applied as a torch pre-export pass via `coreai-opt`; SDPA / RoPE / RMSNorm stay full
precision. The 1B ships the per-channel version of the same config.

```bash
uv run coreai.llm.export openbmb/MiniCPM5-2B --experimental --compute-precision float16 \
  --compression-config minicpm5_int8sym_b32.yaml
# minicpm5_int8sym_b32.yaml: quantization_config → op_state_spec.weight = {dtype: int8,
#   qscheme: symmetric, granularity: {type: per_block, block_size: 32}}
```

## Conversion notes

- **`llama → mistral` remap.** MiniCPM5-2B's `model_type` is `llama` (a plain `LlamaForCausalLM`:
  42 layers × hidden 2048, GQA 16:2, `head_dim` 128, RoPE θ 5e6, untied 130560-vocab head); the
  stock exporter has no `llama` graph family, but Mistral's builder is architecturally identical for
  this config (GQA, no qkv bias, no qk-norm, explicit `head_dim` honored). One-line remap in the
  model registry — the same line that ships the 1B.
- **Chat EOS.** Base `eos_token` is `</s>`, but the chat template ends turns with `<|im_end|>`
  (id 130073). The bundle's tokenizer `eos_token` is set to `<|im_end|>` (as Qwen ships) so
  generation halts cleanly — checked through the engine with the chat template applied: the model
  thinks, answers, and stops (124-token reply, 131.6 tok/s short-context on the M4 Max, on the
  published bundle).
- **Dynamic-shape bundle** → the Core AI pipelined engine (the iPhone path); a static iOS export
  routes to the static-shape engine instead, which this FM-format bundle doesn't target. The 2.67 GB
  single-file bundle cold-specializes on the phone in 28.9 s (no AOT); that step needs the
  increased-memory entitlement and ~3 GB of free phone storage.
- **Thinking.** The model thinks by default (`<think>…</think>` before the answer); pass
  `enable_thinking=False` through the chat template for a direct answer. Give generation a
  generous budget (the kit caps at 4096) — the think trace alone can run several hundred tokens.

## Run

```swift
import FoundationModels
import CoreAILanguageModels
let model = try await CoreAILanguageModel(resourcesAt: int8BundleURL)   // …/int8
let session = LanguageModelSession(model: model)
print(try await session.respond(to: "Explain on-device AI in one sentence."))
```

Or in the zoo's **CoreAIChat** app / the kit's **ChatDemo** (Model → "MiniCPM5 2B").

## Reproduce

Exporter, gate, card and port notes live in the
[Core AI model zoo](https://github.com/john-rocky/coreai-model-zoo):
[`models/minicpm5-2b/`](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/minicpm5-2b),
[`conversion/export_minicpm5.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_minicpm5.py),
[`knowledge/minicpm5-1b.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/minicpm5-1b.md).

```bash
python3 conversion/zoo_convert.py show minicpm5-2b
python3 conversion/zoo_convert.py run  minicpm5-2b
```

## Credits

Model: **MiniCPM5-2B** by **OpenBMB** ([openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B), Apache-2.0).
Core AI conversion: the Core AI model zoo.