VibeThinker Routed Stage2

This repository contains the deployable artifacts and decision record for the local routed VibeThinker agent system built on July 4, 2026.

This is not a single merged GGUF. The working system is:

a local uncensored VibeThinker GGUF base,
a stage2b LoRA GGUF for ordinary tool calling,
a stage2f LoRA GGUF for inline <tool_response> interpretation,
a small router/harness that selects the right LoRA per request.

The router path is intentional. A single-adapter path was tested and rejected.

What We Learned

Stage2c CP-500 was the best checkpoint on the short math screen, but it failed the dense agent/tool gate:

Part A tool selection: 0/26
Part A argument correctness: 0/26
Part A no-tool discrimination: 19/19
Part B compute/tool-response handling: 0/32

That means the math-replay adapter suppressed or destroyed the tool-use behavior needed for agent work. Even if a BFCL subset shows non-zero behavior under a different prompt distribution, CP-500 is not mergeable as the primary agent model.

The useful system is a router:

Pure math/reasoning -> bare VibeThinker route.
Ordinary tool-call prompts -> stage2b checkpoint-250 route.
Inline/post tool-response prompts -> stage2f checkpoint-100 route.

This keeps the strongest observed behavior from each component instead of forcing a single LoRA to preserve incompatible skills.

Included Artifacts

Adapters:

adapters/vibe_stage2b_ck250_lora.gguf
adapters/vibe_stage2f_ck100_lora.gguf

Runtime scripts:

scripts/vibethinker_routed_harness.py
scripts/tool_call_shim.py
scripts/stage2_route_infer.py
scripts/stage2_routed_score.py

Integration notes:

integration/komodo-trade-desk.md

Local service examples:

systemd/vibethinker.service
systemd/vibethinker-tool-harness.service

Evaluation and decision record:

evaluation/stage2_routed_scorecard.json
evaluation/stage2_routed_scorecard.txt
evaluation/stage2_routed_verdict.md
evaluation/stage2_local_routed_vibethinker_status.md
evaluation/dense_cp500.log
evaluation/bfcl_cp500_100.log

Evaluation Summary

Composite routed scorecard:

Math route, bare VibeThinker: GSM8K 74/100 = 74%
General tool route, stage2b ck250: BFCL 405/500 = 81.0%
BFCL irrelevance: 95/100 = 95%
Domain routed holdout:
- Part A tool selection: 26/26
- Part A argument correctness: 22/26
- No-tool discrimination: 19/19
- Part B inline tool-response primary compute: 21/32
- Recursive tool calls in Part B: 0

Single-adapter warning:

stage2f ck100 alone: GSM8K 22/40 = 55%
stage2f ck100 alone: BFCL 386/500 = 77.2%
verdict: do not merge stage2f as the single model.
stage2c CP-500 BFCL formality: official-style re-score 401/500 = 80.2%, but dense suite remains 0/26 tool selection and 0/32 Part-B compute, so CP-500 is still rejected for agent work.

Local Deployment Shape

The tested laptop deployment used:

base GGUF: /home/hazypiff/models/vibethinker-ablated/VibeThinker-3B-Ablated.Q4_K_M.gguf
llama-server: http://127.0.0.1:11501
routed OpenAI-compatible endpoint: http://127.0.0.1:11502/v1/chat/completions
health endpoint: http://127.0.0.1:11502/health

The llama-server loads both LoRA adapters at global scale 0.0. The harness applies per-request LoRA scales:

{
  "bare": [
    {"id": 0, "scale": 0.0},
    {"id": 1, "scale": 0.0}
  ],
  "stage2b": [
    {"id": 0, "scale": 1.0},
    {"id": 1, "scale": 0.0}
  ],
  "stage2f": [
    {"id": 0, "scale": 0.0},
    {"id": 1, "scale": 1.0}
  ]
}

Example llama.cpp Server

Adjust paths for your machine:

llama-server \
  -m /path/to/VibeThinker-3B-Ablated.Q4_K_M.gguf \
  --alias vibethinker-routed-local \
  --lora-scaled /path/to/adapters/vibe_stage2b_ck250_lora.gguf:0.0,/path/to/adapters/vibe_stage2f_ck100_lora.gguf:0.0 \
  --lora-init-without-apply \
  -ngl 0 --host 127.0.0.1 --port 11501 \
  -c 32768 -t 6 -b 512 -ub 128 -np 1 -fa on \
  --repeat-last-n 256 --repeat-penalty 1.15 \
  --dry-multiplier 0.8 --dry-allowed-length 2 --dry-penalty-last-n 2048 \
  --cache-ram 0 --jinja --no-webui --metrics

Then run the routed harness:

export VT_HARNESS_UPSTREAM=http://127.0.0.1:11501/v1/chat/completions
export VT_HARNESS_PORT=11502
export VT_HARNESS_MODEL=vibethinker-routed-local
export VT_HARNESS_PROMPT_MODE=hermes
export VT_STAGE2B_LORA_ID=0
export VT_STAGE2F_LORA_ID=1
export VT_HARNESS_UPSTREAM_TIMEOUT=600
export VT_HARNESS_FALLBACK_TIMEOUT=600
export VT_HARNESS_STREAM_KEEPALIVE_SEC=15
export VT_HARNESS_PLAIN_MAX_TOKENS=1536
export VT_HARNESS_PLAIN_RESCUE_MAX_TOKENS=384
export VT_HARNESS_PLAIN_RESCUE_THINKING_BUDGET=0
python3 scripts/vibethinker_routed_harness.py

Routing Policy

The deployed harness uses this policy:

If the request is a post-tool turn or contains <tool_response> in a non-system message, use stage2f.
Else if tools are supplied or the prompt contains a <tools> block, use stage2b.
Else use bare VibeThinker.

For the bare route, the harness uses a single direct completion pass with a final answer commit instruction. This avoids the old two-round tool-call rescue loop for plain no-tool reasoning requests while preserving the stage2b/stage2f tool routes.

For local agents that call this endpoint with stream: true, use the current harness. It opens the SSE stream immediately with an assistant-role chunk and then sends no-op keepalive chunks while llama.cpp is still evaluating a long reasoning prompt. Without that, clients such as Hermes can report waiting for stream response and fail the provider call before the CPU backend produces its first final token.

The bare route intentionally allows the base model to think, but it also has a plain-answer rescue path. If a no-tool reasoning pass spends the whole budget inside <think> with no committed answer, the harness runs a compact final-answer rescue against the original user message instead of returning an empty/fallback assistant response.

Local Smoke Tests

Passed on the laptop before upload:

Bare route: What is 12 * 7? -> 84
Stage2b route: weather tool prompt -> structured get_weather({"city":"Boston"})
Stage2f route: inline close/fill <tool_response> -> direct answer, no recursive tool call
OpenAI-compatible streaming: first SSE chunk arrives immediately; keepalives protect long CPU prompt-eval/generation gaps.
Hermes profile smoke test: Reply with exactly OK. -> OK
Komodo /api/agent/chat, forced reasoning: 12 * 7 -> 84
Komodo /api/agent/chat, forced reasoning: desk state -> We are in paper mode with $808,550 equity, 13 robots (11 running, 5 open).

Compatibility Notes

The LoRAs were trained against WeiboAI/VibeThinker-3B and converted to llama.cpp LoRA GGUF. The local deployment tested them against an uncensored/ablated VibeThinker GGUF. That pairing smoke-tested successfully, but for maximum fidelity use a GGUF base with the same architecture and lineage as the training base.

The base model GGUF is not included in this repository. This repo contains the router system, LoRA adapters, service wiring, and evaluation record.