site-needle 20260910-204035-3618393

Needle 2 (Cactus Compute's 45M-parameter tool-calling model, 14 MB as a 2-bit .cact) fine-tuned with LoRA into an intent router for alexnodeland.com. Given a visitor's question it returns one of five typed site tools — lookup_role, lookup_project, check_skill, search_site, contact — or the empty call for anything off-topic, injected, negated, or conversational. It generates no prose: the site's chat retrieves and answers; this model decides what to look up.

Use

import json, needle

tools = json.load(open('tools.json'))          # the catalogue this model was tuned on
system = open('system.txt').read().strip()
agent = needle.Needle(tools=tools, system=system, weights='site-needle.cact')
print(agent.complete('what did alex do at musiio?')['function_calls'])
# [{'name': 'lookup_role', 'arguments': {'company': 'musiio'}}]

Training data

Derived from the site's content at sha256:05401f9493d9: 11 roles, 2 degrees, 22 projects, 14 posts. 2314 training examples (assistant questions templated over those entities, refusals, and the site's prose as extraction records) and 272 held-out test cases on unseen phrasings and unseen entities. Corpus sha256:a33cb2275939.

Training

LoRA rank 32 (alpha 64) on the five attention projections, lr 0.0002, batch 8, 3 epochs (783 steps at sequence length 192), quantisation-aware through the checkpoint's 2-bit scheme, seed 0. Final loss 0.0822, validation 0.0524; epoch 3 of 3 selected by highest dev objective. cactus-needle 2.0.13, jax 0.4.38 on METAL.

Evaluation

272 held-out cases through the native engine.

metric	base	tuned	Δ
objective (exact call)	0.173	0.581	+0.408
tool accuracy	0.320	0.754	+0.434
argument F1	0.215	0.564	+0.349
false refusal rate	0.149	0.041	-0.107
missed refusal rate	0.567	0.433	-0.133
critical pass rate	0.500	1.000	+0.500
latency p50 / p95 (ms)	56 / 125	34 / 60
decode tok/s	1126	1400
peak RAM (MB)	245	65

By slice

	n	base	tuned	Δ
natural	131	0.14	0.62	+0.481
novel_entity	30	0.23	0.50	+0.267
paraphrase	108	0.20	0.56	+0.352
site_prompts	3	0.00	0.67	+0.667

On 76 hand-written questions, never generated and never used for selection:

metric	base	tuned	Δ
objective (exact call)	0.210	0.526	+0.316
tool accuracy	0.342	0.697	+0.355
argument F1	0.196	0.488	+0.292
false refusal rate	0.140	0.088	-0.053
missed refusal rate	0.632	0.210	-0.421
critical pass rate	0.500	0.833	+0.333
latency p50 / p95 (ms)	60 / 123	35 / 55
decode tok/s	1127	1376
peak RAM (MB)	261	62

Limitations

• Fine-tuning does not update Needle's confidence head; confidence is None for this model. Gate on the call itself (an empty call is a refusal).
• English only, and only questions about this one site. Anything else is meant to come back as the empty call.
• Free-text arguments are copied from the question verbatim; resolve them case-insensitively against the site's data.

Provenance

• run 20260910-204035-3618393, git 361839340f7b42bb3ecd3019b85385929ad9faf4, 2026-09-10T20:40:35+00:00
• pipeline: apps/model in alexnodeland/alexnodeland
• rendered 2026-09-10T20:49:18+00:00